SlideShare a Scribd company logo

Turbo2018 workshop JIT as a Service

•
M
Mark Stoodley

The presentation discusses 'JIT as a Service', emphasizing the move towards remote compilation for lightweight runtimes, which can optimize performance without the constraints of local resources. It exemplifies how this client-server model can help maintain throughput while managing memory consumption efficiently, even in constrained environments. Future work includes closing performance gaps, improving collaborative compilations, and extending JIT services beyond current languages.

Software
1 of 27
Download to read offline
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
JIT as a Service Compiling for Runtimes in the Cloud Mark Stoodley Eclipse OMR and Eclipse OpenJ9 project lead
2 Important Disclaimers § THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY. § WHILST EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION CONTAINED IN THIS PRESENTATION, IT IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. § ALL PERFORMANCE DATA INCLUDED IN THIS PRESENTATION HAVE BEEN GATHERED IN A CONTROLLED ENVIRONMENT. YOUR OWN TEST RESULTS MAY VARY BASED ON HARDWARE, SOFTWARE OR INFRASTRUCTURE DIFFERENCES. § ALL DATA INCLUDED IN THIS PRESENTATION ARE MEANT TO BE USED ONLY AS A GUIDE. § IN ADDITION, THE INFORMATION CONTAINED IN THIS PRESENTATION IS BASED ON IBM’S CURRENT PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM, WITHOUT NOTICE. § IBM AND ITS AFFILIATED COMPANIES SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION. § NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, OR SHALL HAVE THE EFFECT OF: – CREATING ANY WARRANT OR REPRESENTATION FROM IBM, ITS AFFILIATED COMPANIES OR ITS OR THEIR SUPPLIERS AND/OR LICENSORS
Runtimes need to operate in small spaces 8 cores 8GB 1 core, 1GB 1 core, 1GB 1 core, 1GB 1 core, 1GB 1 core, 1GB 1 core, 1GB 1 core, 1GB 1 core, 1GB 1. Virtualization 2. Microservice Architecture Monolithic Application High Input Load Service A Service B Service C Service D Service E Service F Possibly lower load connections Many lighter-weight applications need to be allocated to many small VMs 3
Driving some interest in Ahead of Time (AOT) compilers even for ”dynamic” languages 4
5 AOT compiler JIT compiler Runtime CPU cycles none considerable early on Runtime memory none considerable early on Ability to prove things excellent more limited Performance ramp-up immediate takes time Target environment have to choose CPU its running on Profile based optimizations can use earlier run data but awkward collects and uses data from current run Speculative optimization can’t really afford yes, aggressively AOT vs. JIT compilers for dynamic languages
AOT compilation has some advantages but leaves significant performance on the table by itself 6
What if… • What if we could JIT compile out of process • On another machine or even on a cluster of machines • With independently deployed CPUs and memory • Able to serve multiple applications and sharing code • Optimizing with information collected from many runtime clients 7
Enter “JIT as a Service” 8
1. Move runtime costs to remote service …can afford JIT even for very lightweight runtimes 2. Still running alongside applications …so no loss in code performance 9
Yes! There will be latency! But maybe not as bad as you think 10
The Basic JITaaS Architecture • A client-server model • Bidirectional communication • Target method + metadata sent to server • Compiled code + metadata returned to client Client Server Compilation begins VM queries ×N Compilation ends Time 11
Two ongoing projects 1. Eclipse OMR • Record JitBuilder calls on client, replay on server 2. Eclipse OpenJ9 • Compile Java methods from bytecodes 12
IBM ExtremeBlue student project over the summer : 1. Record JitBuilder API calls on the client side 2. Send that record to a JIT server 3. If no code generated already for the provided record: 4. Replay the recorded calls to JitBuilder implementation in the JIT server 5. Store generated code using the record as key 6. Send the code back to be installed in client’s code cache 7. Client can call the native code as a C function pointer Created a client JIT for ultra simple (but Turing complete!) “BF” language https://en.wikipedia.org/wiki/Brainf*** JitBuilder as a Service
Demo 1 JitBuilder as a Service 14
Eclipse OpenJ9x Java JIT as a Service 15
8 Steps to a Remote Compiled Method 16 Client Server
Graphs & Numbers • Throughput • Memory Consumption • Constrained environments 17
Throughput 18 0 2000 4000 6000 8000 10000 12000 14000 0 50 100 150 200 250 300 Pages/sec Time (seconds) AcmeAir Throughput OpenJ9 JITaaS
Throughput Takeaways • JIT as a service doesn’t have to sacrifice peak throughput • Network latency (in aggregate) is tolerable • Many parallel compilation requests overlap much of the cost • Can cache many query results to avoid network round trips 19
Memory Consumption 20 0 50 100 150 200 250 300 350 0 50 100 150 200 250 300 Residentsetsize(MiB) Time (seconds) AcmeAir Footprint OpenJ9 JITaaS Client
Memory Consumption Takeaways • Dominant memory consumption is now application • Size your containers to your apps, forget about the JIT • Workload balancers will no longer “fooled” by spiky JIT activity 21
Demo 2 Java Jit as a Service in a Constrained Environment (256MB, ½ core) 22
Private
Future Work • Close remaining performance gap • Share compilations between clients • Optimizations enabled through connection to multiple clients • Build different kinds of remote JIT clients, not just for languages • Merge hardened prototypes to OMR/OpenJ9 master branches 24 Shameless self plug: Whittier room at 3:30 today by Mark Stoodley https://2018.splashcon.org/event/splash-2018-splash-i-oh-the-compilers-you-will-build-
All open and on the way to merged • Eclipse OMR • https://github.com/eclipse/omr/pull/3019 (JitBuilder record function) • https://github.com/eclipse/omr/pull/3056 (JitBuilder replay function) • https://eclipse-omr.slack.com • Eclipse OpenJ9 • https://github.com/eclipse/openj9/tree/jitaas • https://github.com/eclipse/openj9-omr/tree/jitaas • https://openj9.slack.com 25
Questions? 26
Throughput in a Constrained Environment 27 0 200 400 600 800 1000 1200 1400 1600 0 100 200 300 400 500 Pages/sec Time (seconds) AcmeAir Throughput in a Micro-Container (64 MiB, 0.5 CPU) HotSpot OpenJ9 JITaaS

More Related Content

Similar to Turbo2018 workshop JIT as a Service (20)

PDF
JITServerTalk Nebraska 2023.pdf
RichHagarty
 
PDF
JITServerTalk JCON World 2023.pdf
RichHagarty
 
PDF
DevNexus 2024: Just-In-Time Compilation as a Service for cloud-native Java mi...
RichHagarty
 
PDF
JITServerTalk-OSS-2023.pdf
RichHagarty
 
PDF
Understand the Trade-offs Using Compilers for Java Applications
C4Media
 
PDF
Jit builder status and directions 2018 03-28
Mark Stoodley
 
PPTX
Javaland_JITServerTalk.pptx
Grace Jansen
 
PDF
Clone Clone Make: a better way to build
DanHeidinga
 
PPTX
JPrime_JITServer.pptx
Grace Jansen
 
PDF
It's always sunny with OpenJ9
DanHeidinga
 
PDF
Under the Hood of the Testarossa JIT Compiler
Mark Stoodley
 
PPTX
Jfokus 2016 - A JVMs Journey into Polyglot Runtimes
Charlie Gracie
 
PDF
Everything You Wanted to Know About JIT Compilation but Were Afraid to Ask [J...
David Buck
 
PDF
BM Real-time Technologies for SUSE Linux Enterprise Real Time
Novell
 
PPTX
Clr jvm implementation differences
Jean-Philippe BEMPEL
 
PDF
Oh the compilers you'll build
Mark Stoodley
 
DOCX
just in time JIT compiler
Mohit kumar
 
PPTX
Cloud Native Compiler
Simon Ritter
 
PDF
JavaOne 2015 CON7547 "Beyond the Coffee Cup: Leveraging Java Runtime Technolo...
0xdaryl
 
PPTX
JavaLand_To InstantOn and Beyond.pptx
Grace Jansen
 
JITServerTalk Nebraska 2023.pdf
RichHagarty
 
JITServerTalk JCON World 2023.pdf
RichHagarty
 
DevNexus 2024: Just-In-Time Compilation as a Service for cloud-native Java mi...
RichHagarty
 
JITServerTalk-OSS-2023.pdf
RichHagarty
 
Understand the Trade-offs Using Compilers for Java Applications
C4Media
 
Jit builder status and directions 2018 03-28
Mark Stoodley
 
Javaland_JITServerTalk.pptx
Grace Jansen
 
Clone Clone Make: a better way to build
DanHeidinga
 
JPrime_JITServer.pptx
Grace Jansen
 
It's always sunny with OpenJ9
DanHeidinga
 
Under the Hood of the Testarossa JIT Compiler
Mark Stoodley
 
Jfokus 2016 - A JVMs Journey into Polyglot Runtimes
Charlie Gracie
 
Everything You Wanted to Know About JIT Compilation but Were Afraid to Ask [J...
David Buck
 
BM Real-time Technologies for SUSE Linux Enterprise Real Time
Novell
 
Clr jvm implementation differences
Jean-Philippe BEMPEL
 
Oh the compilers you'll build
Mark Stoodley
 
just in time JIT compiler
Mohit kumar
 
Cloud Native Compiler
Simon Ritter
 
JavaOne 2015 CON7547 "Beyond the Coffee Cup: Leveraging Java Runtime Technolo...
0xdaryl
 
JavaLand_To InstantOn and Beyond.pptx
Grace Jansen
 

Recently uploaded (20)

PDF
Infrastructure planning and resilience - Keith Hastings.pptx.pdf
Safe Software
 
PDF
System Center 2025 vs. 2022; What’s new, what’s next_PDF.pdf
Q-Advise
 
PDF
AWS_Agentic_AI_in_Indian_BFSI_A_Strategic_Blueprint_for_Customer.pdf
siddharthnetsavvies
 
PPTX
TexSender Pro 8.9.1 Crack Full Version Download
cracked shares
 
PDF
AI Software Engineering based on Multi-view Modeling and Engineering Patterns
Hironori Washizaki
 
PPTX
Contractor Management Platform and Software Solution for Compliance
SHEQ Network Limited
 
PDF
Using licensed Data Loss Prevention (DLP) as a strategic proactive data secur...
Q-Advise
 
PPTX
slidesgo-unlocking-the-code-the-dynamic-dance-of-variables-and-constants-2024...
kr2589474
 
PDF
Adobe Illustrator Crack Full Download (Latest Version 2025) Pre-Activated
imang66g
 
PDF
Salesforce Pricing Update 2025: Impact, Strategy & Smart Cost Optimization wi...
GetOnCRM Solutions
 
PDF
Supabase Meetup: Build in a weekend, scale to millions
Carlo Gilmar Padilla Santana
 
PDF
ChatPharo: an Open Architecture for Understanding How to Talk Live to LLMs
ESUG
 
PDF
Step-by-Step Guide to Install SAP HANA Studio | Complete Installation Tutoria...
SAP Vista, an A L T Z E N Company
 
PDF
AI Image Enhancer: Revolutionizing Visual Quality”
docmasoom
 
PPTX
Farrell__10e_ch04_PowerPoint.pptx Programming Logic and Design slides
bashnahara11
 
PDF
What companies do with Pharo (ESUG 2025)
ESUG
 
PPTX
TRAVEL APIs | WHITE LABEL TRAVEL API | TOP TRAVEL APIs
philipnathen82
 
PDF
Download iTop VPN Free 6.1.0.5882 Crack Full Activated Pre Latest 2025
imang66g
 
PDF
Enhancing Healthcare RPM Platforms with Contextual AI Integration
Cadabra Studio
 
PPTX
Explanation about Structures in C language.pptx
Veeral Rathod
 
Infrastructure planning and resilience - Keith Hastings.pptx.pdf
Safe Software
 
System Center 2025 vs. 2022; What’s new, what’s next_PDF.pdf
Q-Advise
 
AWS_Agentic_AI_in_Indian_BFSI_A_Strategic_Blueprint_for_Customer.pdf
siddharthnetsavvies
 
TexSender Pro 8.9.1 Crack Full Version Download
cracked shares
 
AI Software Engineering based on Multi-view Modeling and Engineering Patterns
Hironori Washizaki
 
Contractor Management Platform and Software Solution for Compliance
SHEQ Network Limited
 
Using licensed Data Loss Prevention (DLP) as a strategic proactive data secur...
Q-Advise
 
slidesgo-unlocking-the-code-the-dynamic-dance-of-variables-and-constants-2024...
kr2589474
 
Adobe Illustrator Crack Full Download (Latest Version 2025) Pre-Activated
imang66g
 
Salesforce Pricing Update 2025: Impact, Strategy & Smart Cost Optimization wi...
GetOnCRM Solutions
 
Supabase Meetup: Build in a weekend, scale to millions
Carlo Gilmar Padilla Santana
 
ChatPharo: an Open Architecture for Understanding How to Talk Live to LLMs
ESUG
 
Step-by-Step Guide to Install SAP HANA Studio | Complete Installation Tutoria...
SAP Vista, an A L T Z E N Company
 
AI Image Enhancer: Revolutionizing Visual Quality”
docmasoom
 
Farrell__10e_ch04_PowerPoint.pptx Programming Logic and Design slides
bashnahara11
 
What companies do with Pharo (ESUG 2025)
ESUG
 
TRAVEL APIs | WHITE LABEL TRAVEL API | TOP TRAVEL APIs
philipnathen82
 
Download iTop VPN Free 6.1.0.5882 Crack Full Activated Pre Latest 2025
imang66g
 
Enhancing Healthcare RPM Platforms with Contextual AI Integration
Cadabra Studio
 
Explanation about Structures in C language.pptx
Veeral Rathod
 
Ad

Turbo2018 workshop JIT as a Service

  • 1. JIT as a Service Compiling for Runtimes in the Cloud Mark Stoodley Eclipse OMR and Eclipse OpenJ9 project lead
  • 2. 2 Important Disclaimers § THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY. § WHILST EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION CONTAINED IN THIS PRESENTATION, IT IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. § ALL PERFORMANCE DATA INCLUDED IN THIS PRESENTATION HAVE BEEN GATHERED IN A CONTROLLED ENVIRONMENT. YOUR OWN TEST RESULTS MAY VARY BASED ON HARDWARE, SOFTWARE OR INFRASTRUCTURE DIFFERENCES. § ALL DATA INCLUDED IN THIS PRESENTATION ARE MEANT TO BE USED ONLY AS A GUIDE. § IN ADDITION, THE INFORMATION CONTAINED IN THIS PRESENTATION IS BASED ON IBM’S CURRENT PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM, WITHOUT NOTICE. § IBM AND ITS AFFILIATED COMPANIES SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION. § NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, OR SHALL HAVE THE EFFECT OF: – CREATING ANY WARRANT OR REPRESENTATION FROM IBM, ITS AFFILIATED COMPANIES OR ITS OR THEIR SUPPLIERS AND/OR LICENSORS
  • 3. Runtimes need to operate in small spaces 8 cores 8GB 1 core, 1GB 1 core, 1GB 1 core, 1GB 1 core, 1GB 1 core, 1GB 1 core, 1GB 1 core, 1GB 1 core, 1GB 1. Virtualization 2. Microservice Architecture Monolithic Application High Input Load Service A Service B Service C Service D Service E Service F Possibly lower load connections Many lighter-weight applications need to be allocated to many small VMs 3
  • 4. Driving some interest in Ahead of Time (AOT) compilers even for ”dynamic” languages 4
  • 5. 5 AOT compiler JIT compiler Runtime CPU cycles none considerable early on Runtime memory none considerable early on Ability to prove things excellent more limited Performance ramp-up immediate takes time Target environment have to choose CPU its running on Profile based optimizations can use earlier run data but awkward collects and uses data from current run Speculative optimization can’t really afford yes, aggressively AOT vs. JIT compilers for dynamic languages
  • 6. AOT compilation has some advantages but leaves significant performance on the table by itself 6
  • 7. What if… • What if we could JIT compile out of process • On another machine or even on a cluster of machines • With independently deployed CPUs and memory • Able to serve multiple applications and sharing code • Optimizing with information collected from many runtime clients 7
  • 8. Enter “JIT as a Service” 8
  • 9. 1. Move runtime costs to remote service …can afford JIT even for very lightweight runtimes 2. Still running alongside applications …so no loss in code performance 9
  • 10. Yes! There will be latency! But maybe not as bad as you think 10
  • 11. The Basic JITaaS Architecture • A client-server model • Bidirectional communication • Target method + metadata sent to server • Compiled code + metadata returned to client Client Server Compilation begins VM queries ×N Compilation ends Time 11
  • 12. Two ongoing projects 1. Eclipse OMR • Record JitBuilder calls on client, replay on server 2. Eclipse OpenJ9 • Compile Java methods from bytecodes 12
  • 13. IBM ExtremeBlue student project over the summer : 1. Record JitBuilder API calls on the client side 2. Send that record to a JIT server 3. If no code generated already for the provided record: 4. Replay the recorded calls to JitBuilder implementation in the JIT server 5. Store generated code using the record as key 6. Send the code back to be installed in client’s code cache 7. Client can call the native code as a C function pointer Created a client JIT for ultra simple (but Turing complete!) “BF” language https://en.wikipedia.org/wiki/Brainf*** JitBuilder as a Service
  • 14. Demo 1 JitBuilder as a Service 14
  • 15. Eclipse OpenJ9x Java JIT as a Service 15
  • 16. 8 Steps to a Remote Compiled Method 16 Client Server
  • 17. Graphs & Numbers • Throughput • Memory Consumption • Constrained environments 17
  • 18. Throughput 18 0 2000 4000 6000 8000 10000 12000 14000 0 50 100 150 200 250 300 Pages/sec Time (seconds) AcmeAir Throughput OpenJ9 JITaaS
  • 19. Throughput Takeaways • JIT as a service doesn’t have to sacrifice peak throughput • Network latency (in aggregate) is tolerable • Many parallel compilation requests overlap much of the cost • Can cache many query results to avoid network round trips 19
  • 20. Memory Consumption 20 0 50 100 150 200 250 300 350 0 50 100 150 200 250 300 Residentsetsize(MiB) Time (seconds) AcmeAir Footprint OpenJ9 JITaaS Client
  • 21. Memory Consumption Takeaways • Dominant memory consumption is now application • Size your containers to your apps, forget about the JIT • Workload balancers will no longer “fooled” by spiky JIT activity 21
  • 22. Demo 2 Java Jit as a Service in a Constrained Environment (256MB, ½ core) 22
  • 24. Future Work • Close remaining performance gap • Share compilations between clients • Optimizations enabled through connection to multiple clients • Build different kinds of remote JIT clients, not just for languages • Merge hardened prototypes to OMR/OpenJ9 master branches 24 Shameless self plug: Whittier room at 3:30 today by Mark Stoodley https://2018.splashcon.org/event/splash-2018-splash-i-oh-the-compilers-you-will-build-
  • 25. All open and on the way to merged • Eclipse OMR • https://github.com/eclipse/omr/pull/3019 (JitBuilder record function) • https://github.com/eclipse/omr/pull/3056 (JitBuilder replay function) • https://eclipse-omr.slack.com • Eclipse OpenJ9 • https://github.com/eclipse/openj9/tree/jitaas • https://github.com/eclipse/openj9-omr/tree/jitaas • https://openj9.slack.com 25
  • 27. Throughput in a Constrained Environment 27 0 200 400 600 800 1000 1200 1400 1600 0 100 200 300 400 500 Pages/sec Time (seconds) AcmeAir Throughput in a Micro-Container (64 MiB, 0.5 CPU) HotSpot OpenJ9 JITaaS