Finally! Real Java for low latency and low jitter

Size: px

Start display at page:

Download "Finally! Real Java for low latency and low jitter"

Mary Atkinson
5 years ago
Views:

1 Finally! Real Java for low latency and low jitter Gil Tene, CTO & co-founder, Azul Systems

2 High level agenda Java in a low latency application world Why Stop-The-World is a problem (Duh?) Java vs. actual, real Java Some Garbage Collection terminology Classifying current commercially available collectors The C4 collector: What a solution to STW looks like...

About me: Gil Tene co-founder, CTO @Azul Systems Have been working on a think

(Tene, Wolf) A Long history building Virtual & Physical Machines, Operating

3 About me: Gil Tene co-founder, Systems Have been working on a think different GC approaches since 2002 Created Pauseless & C4 core GC algorithms (Tene, Wolf) A Long history building Virtual & Physical Machines, Operating Systems, Enterprise apps, etc... * working on real-world trash compaction issues, circa 2004

4 About Azul We make scalable Virtual Machines Vega Have built whatever it takes to get job done since generations of custom SMP Multi-core HW (Vega) Now Pure software for commodity x86 (Zing) C4 Industry firsts in Garbage collection, elastic memory, Java virtualization, memory scale

5 Java in the low latency world

6 Java in a low latency world Why do people use Java for low latency apps? Are they crazy? No. There are good, easy to articulate reasons Mostly around productivity and long term cost Developer productivity. Leverage. Time-to-product, Time-to-market,... Extras: leverage, ecosystem, ability to hire

7 E.g. Customer answer to: Why do you use Java in Algo Trading? Strategies have a shelf life We have to keep developing and deploying new ones Only one out of N is actually productive Profitability therefore depends on ability to successfully deploy new strategies, and on the cost of doing so Our developers seem to be able to produce 2x-3x as much when using a Java environment as they would with C++...

8 So what is the problem? Is Java Slow? No A good programmer will get roughly the same speed from both Java and C++ A bad programmer won t get you fast code on either The 50% ile and 90% ile are typically excellent... It s those pesky occasional stutters and stammers that are the problem... Ever hear of Garbage Collection?

9 Java s achilles heel

10 Garbage Collection How bad is it? Let s ignore the bad multi-second pauses for now... Low latency applications regularly experience small, minor GC events that range in the 10s of msec Frequency directly related to allocation rate In turn, directly related to throughput So we have great 50%, 90%. Maybe even 99% But 99.9%, 99.99%, Max, all suck So bad that it affects risk, profitability, service expectations, etc.

11 What do low latency developers do about it? They use Java instead of Java They write in the Java syntax They avoid allocation as much as possible E.g. They build their own object pools for everything They write all the code they use (no 3rd party libraries) They train developers for their local discipline In short: They revert to many of the practices that hurt productivity. They loose out on much of Java.

12 Some GC Terminology

13 A Basic Terminology example: What is a concurrent collector? A Concurrent Collector performs garbage collection work concurrently with the application s own execution A Parallel Collector uses multiple CPUs to perform garbage collection

14 Classifying a collector s operation A Concurrent Collector performs garbage collection work concurrently with the application s own execution A Parallel Collector uses multiple CPUs to perform garbage collection A Stop-the-World collector performs garbage collection while the application is completely stopped An Incremental collector performs a garbage collection operation or phase as a series of smaller discrete operations with (potentially long) gaps in between Mostly means sometimes it isn t (usually means a different fall back mechanism exists)

15 What s common to all Java GC mechanisms? Identify the live objects in the memory heap Reclaim resources held by dead objects Periodically relocate live objects Examples: Mark/Sweep/Compact (common for Old Generations) Copying collector (common for Young Generations)

16 Mark (aka Trace ) Start from roots (thread stacks, statics, etc.) Paint anything you can reach as live At the end of a mark pass: all reachable objects will be marked live all non-reachable objects will be marked dead (aka non-live ). Note: work is generally linear to live set

17 Sweep Scan through the heap, identify dead objects and track them somehow (usually in some form of free list) Note: work is generally linear to heap size

18 Compact Over time, heap will get swiss cheesed : contiguous dead space between objects may not be large enough to fit new objects (aka fragmentation ) Compaction moves live objects together to reclaim contiguous empty space (aka relocate ) Compaction has to correct all object references to point to new object locations (aka remap ) Remap scan must cover all references that could possibly point to relocated objects Note: work is generally linear to live set

19 Copy A copying collector moves all lives objects from a from space to a to space & reclaims from space At start of copy, all objects are in from space and all references point to from space. Start from root references, copy any reachable object to to space, correcting references as we go At end of copy, all objects are in to space, and all references point to to space Note: work generally linear to live set

20 Generational Collection Weak Generational Hypothesis; most objects die young Focus collection efforts on young generation: Use a moving collector: work is linear to the live set The live set in the young generation is a small % of the space Promote objects that live long enough to older generations Only collect older generations as they fill up Generational filter reduces rate of allocation into older generations Tends to be (order of magnitude) more efficient Great way to keep up with high allocation rate Practical necessity for keeping up with processor throughput

21 The typical combos in commercial server JVMS Young generation usually uses a copying collector Young generation is usually monolithic, stop-the-world Old generation usually uses Mark/Sweep/Compact Old generation may be STW, or Concurrent, or mostly-concurrent, or Incremental-STW, or mostly- Incremental-STW

22 Empty memory and CPU/throughput

23 CPU% Heap size vs. 100% GC CPU % Heap size Live set

24 Empty memory needs (empty memory == CPU power) The amount of empty memory in the heap is the dominant factor controlling the amount of GC work For both Copy and Mark/Compact collectors, the amount of work per cycle is linear to live set The amount of memory recovered per cycle is equal to the amount of unused memory (heap size - live set) The collector has to perform a GC cycle when the empty memory runs out A Copy or Mark/Compact collector s efficiency doubles with every doubling of the empty memory

25 What empty memory controls Empty memory controls efficiency (amount of collector work needed per amount of application work performed) Empty memory controls the frequency of pauses (if the collector performs any Stop-the-world operations) Empty memory DOES NOT control pause times (only their frequency) In Mark/Sweep/Compact collectors that pause for sweeping, more empty memory means less frequent but LARGER pauses

26 Delaying the inevitable Some form of copying/compaction is inevitable in practice And compacting anything requires scanning/fixing all references to it Delay tactics focus on getting easy empty space first This is the focus for the vast majority of GC tuning Most objects die young [Generational] So collect young objects only, as much as possible. Hope for short STW. But eventually, some old dead objects must be reclaimed Most old dead space can be reclaimed without moving it [e.g. CMS] track dead space in lists, and reuse it in place But eventually, space gets fragmented, and needs to be moved Low Latency Enterprise Apps Much of the heap is not popular [e.g. G1, Balanced ] A non popular region will only be pointed to from a small % of the heap So compact non-popular regions in short stop-the-world pauses But eventually, popular objects and regions need to be compacted

27 Classifying common collectors

28 The typical combos in commercial server JVMs Young generation usually uses a copying collector Young generation is usually monolithic, stop-the-world Old generation usually uses a Mark/Sweep/Compact collector Old generation may be STW, or Concurrent, or mostly-concurrent, or Incremental-STW, or mostly-incremental-stw

29 HotSpot ParallelGC Collector mechanism classification Monolithic Stop-the-world copying NewGen Monolithic Stop-the-world Mark/Sweep/Compact OldGen

30 HotSpot ConcMarkSweepGC (aka CMS) Collector mechanism classification Monolithic Stop-the-world copying NewGen (ParNew) Mostly Concurrent, non-compacting OldGen (CMS) Mostly Concurrent marking Mark concurrently while mutator is running Track mutations in card marks Revisit mutated cards (repeat as needed) Stop-the-world to catch up on mutations, ref processing, etc. Concurrent Sweeping Does not Compact (maintains free list, does not move objects) Fallback to Full Collection (Monolithic Stop the world). Used for Compaction, etc.

31 HotSpot G1GC (aka Garbage First ) Collector mechanism classification Monolithic Stop-the-world copying NewGen Mostly Concurrent, OldGen marker Mostly Concurrent marking Stop-the-world to catch up on mutations, ref processing, etc. Tracks inter-region relationships in remembered sets Stop-the-world mostly incremental compacting old gen Objective: Avoid, as much as possible, having a Full GC Compact sets of regions that can be scanned in limited time Delay compaction of popular objects, popular regions Fallback to Full Collection (Monolithic Stop the world). Used for compacting popular objects, popular regions, etc.

32 Monolithic-STW GC Problems

33 One way to deal with Monolithic-STW GC

34 Common ways people deal with hiccups Averages and Standard Deviation

35 Max"per"Interv H Hiccup&Dura*on&(msec)& 18000" 16000" 14000" 12000" 10000" 8000" 6000" 4000" 2000" 0" 0" 2000"

36 Hiccups&by&Time&Interval& Max"per"Interval" 99%" 99.90%" 99.99%" Max" Hiccup&Dura*on&(msec)& 18000" 16000" 14000" 12000" 10000" 8000" 6000" 4000" 2000" 0" 0" 2000" 4000" 6000" 8000" 10000" &Elapsed&Time&(sec)& Hiccup&Dura*on&(msec)& Hiccups&by&Percen*le&Distribu*on& 18000" 16000" 14000" Max= & 12000" 10000" 8000" 6000" 4000" 2000" 0" 0%" 90%" 99%" & 99.9%" 99.99%" %" & %" Percen*le&

38 Another way to cope: Creative Language Guarantee a worst case of X msec, 99% of the time Mostly Concurrent, Mostly Incremental Translation: Will at times exhibit long monolithic stopthe-world pauses Fairly Consistent Translation: Will sometimes show results well outside this range Typical pauses in the tens of milliseconds Translation: Some pauses are much longer than tens of milliseconds 2012 Azul Systems, Inc.

39 Actually measuring things (e.g. jhiccup) 2012 Azul Systems, Inc.

40 Incontinuities in Java platform execution Hiccups"by"Time"Interval" Max"per"Interval" 99%" 99.90%" 99.99%" Max" Hiccup&Dura*on&(msec)& 1800" 1600" 1400" 1200" 1000" 800" 600" 400" 200" 0" 0" 200" 400" 600" 800" 1000" 1200" 1400" 1600" 1800" &Elapsed&Time&(sec)& Hiccup&Dura*on&(msec)& 1800" 1600" 1400" 1200" 1000" 800" 600" 400" Max= & 200" 0" 0%" 90%" 99%" 99.9%" & & 99.99%" %" Percen*le& 2012 Azul Systems, Inc.

41 FiServ Pricing Application Hiccups"by"Time"Interval" Max"per"Interval" 99%" 99.90%" 99.99%" Max" 700" Hiccup&Dura*on&(msec)& 600" 500" 400" 300" 200" 100" 0" 0" 2000" 4000" 6000" 8000" 10000" 12000" 14000" 16000" &Elapsed&Time&(sec)& Hiccup&Dura*on&(msec)& 700" 600" 500" 400" 300" 200" 100" Max= & Hiccups"by"PercenCle"DistribuCon" 0" 0%" 90%" 99%" 99.9%" & 99.99%" %" %" & Percen*le&

42 Yet another FiServ application 160" Hiccups&by&Time&Interval& Max"per"Interval" 99%" 99.90%" 99.99%" Max" Hiccup&Dura*on&(msec)& 140" 120" 100" 80" 60" 40" 20" 0" 0" 500" 1000" 1500" 2000" 2500" 3000" 3500" 4000" 4500" &Elapsed&Time&(sec)& 160" Hiccups&by&Percen*le&Distribu*on& Hiccup&Dura*on&(msec)& 140" 120" 100" 80" 60" 40" 20" Max= & 0" 0%" 90%" 99%" 99.9%" & 99.99%" %" %" & Percen*le& 2012 Azul Systems, Inc.

43 A post-monolithic-stw world

44 We needed to solve the right problems Even short pauses are a problem Scale is artificially limited by responsiveness Responsiveness must be unlinked from scale: Heap size, Live Set size, Allocation rate, Mutation rate Transaction Rate, Concurrent users, Data set size, etc. Responsiveness must be continually sustainable Can t ignore rare events Eliminate all Stop-The-World Fallbacks Any STW fall back is a failure

45 The problems that needed solving (areas where the state of the art needed improvement) Robust Concurrent Marking In the presence of high mutation and allocation rates Cover modern runtime semantics (e.g. weak refs, lock deflation) Compaction that is not monolithic-stop-the-world E.g. stay responsive while compacting entire, modern sized heaps Must be robust: not just a tactic to delay STW compaction [current incremental STW attempts fall short on robustness] Young-Gen that is not monolithic-stop-the-world Stay responsive while promoting and copying data spikes Surprisingly little work done in this specific area

46 Azul s C4 Collector Continuously Concurrent Compacting Collector Concurrent guaranteed-single-pass marker Oblivious to mutation rate Concurrent ref (weak, soft, final) processing Concurrent Compactor Objects moved without stopping mutator References remapped without stopping mutator Can relocate entire generation (New, Old) in every GC cycle Concurrent, compacting old generation Concurrent, compacting new generation No stop-the-world fallback Always compacts, and always does so concurrently

47 Benefits

48 Sample responsiveness behavior SpecJBB + Slow churning 2GB LRU Cache Live set is ~2.5GB across all measurements Allocation rate is ~1.2GB/sec across all measurements

49 Sustainable Throughput: The throughput achieved while safely maintaining service levels Unsustainable Throughout

50 Instance capacity test: Fat Portal HotSpot CMS: Peaks at ~ 3GB / 45 concurrent users * LifeRay portal on 99.9% SLA of 5 second response times

51 Instance capacity test: Fat Portal C4: still 800 concurrent users 2012 Azul Systems, Inc.

52 2012 Azul Systems, Inc. Fun with jhiccup

53 Zing 5, 1GB in an 8GB heap Oracle HotSpot CMS, 1GB in an 8GB heap Hiccups&by&Time&Interval& Hiccups&by&Time&Interval& Max"per"Interval" 99%" 99.90%" 99.99%" Max" Max"per"Interval" 99%" 99.90%" 99.99%" Max" 25" 14000" Hiccup&Dura*on&(msec)& 20" 15" 10" 5" Hiccup&Dura*on&(msec)& 12000" 10000" 8000" 6000" 4000" 2000" 0" 0" 500" 1000" 1500" 2000" 2500" 3000" 3500" &Elapsed&Time&(sec)& 0" 0" 500" 1000" 1500" 2000" 2500" 3000" 3500" &Elapsed&Time&(sec)& Hiccups&by&Percen*le&Distribu*on& Hiccups&by&Percen*le&Distribu*on& 25" 14000" Hiccup&Dura*on&(msec)& 20" 15" 10" 5" Max=20.384& Hiccup&Dura*on&(msec)& 12000" 10000" 8000" 6000" 4000" 2000" Max= & 0" 0%" 90%" 99%" 99.9%" & & 99.99%" %" %" Percen*le& 0" 0%" 90%" 99%" & 99.9%" & 99.99%" %" Percen*le& 2012 Azul Systems, Inc.

54 Zing 5, 1GB in an 8GB heap Oracle HotSpot CMS, 1GB in an 8GB heap Hiccups&by&Time&Interval& Hiccups&by&Time&Interval& Max"per"Interval" 99%" 99.90%" 99.99%" Max" Max"per"Interval" 99%" 99.90%" 99.99%" Max" 14000" 14000" Hiccup&Dura*on&(msec)& 12000" 10000" 8000" 6000" 4000" 2000" Hiccup&Dura*on&(msec)& 12000" 10000" 8000" 6000" 4000" 2000" 0" 0" 500" 1000" 1500" 2000" 2500" 3000" 3500" &Elapsed&Time&(sec)& 0" 0" 500" 1000" 1500" 2000" 2500" 3000" 3500" &Elapsed&Time&(sec)& Hiccups&by&Percen*le&Distribu*on& Hiccups&by&Percen*le&Distribu*on& 14000" 14000" Hiccup&Dura*on&(msec)& 12000" 10000" 8000" 6000" 4000" 2000" Hiccup&Dura*on&(msec)& 12000" 10000" 8000" 6000" 4000" 2000" Max= & 0" 0%" Max=20.384& 90%" 99%" 99.9%" & 99.99%" %" %" & Percen*le& 0" 0%" 90%" 99%" & 99.9%" & 99.99%" %" Percen*le& 2012 Azul Systems, Inc.

55 GC Tuning

56 Java GC tuning is hard Examples of actual command line GC tuning parameters: Java -Xmx12g -XX:MaxPermSize=64M -XX:PermSize=32M -XX:MaxNewSize=2g -XX:NewSize=1g -XX:SurvivorRatio=128 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:MaxTenuringThreshold=0 -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSParallelRemarkEnabled -XX:+UseCMSInitiatingOccupancyOnly -XX:ParallelGCThreads=12 -XX:LargePageSizeInBytes=256m Java Xms8g Xmx8g Xmn2g -XX:PermSize=64M -XX:MaxPermSize=256M -XX:-OmitStackTraceInFastThrow -XX:SurvivorRatio=2 -XX:-UseAdaptiveSizePolicy -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSParallelRemarkEnabled -XX:+CMSParallelSurvivorRemarkEnabled -XX:CMSMaxAbortablePrecleanTime= XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=63 -XX:+UseParNewGC Xnoclassgc

57 The complete guide to Zing GC tuning java -Xmx40g

58 What you can expect (from Zing) in the low latency world Assuming individual transaction work is short (on the order of 1 msec), and assuming you don t have 100s of runnable threads competing for 10 cores... Easily get your application to < 10 msec worst case With some tuning, 2-3 msec worst case Can go to below 1 msec worst case... May require heavy tuning/tweaking Mileage WILL vary 2012 Azul Systems, Inc.

59 An example of out-of-the-box behavior

60 Oracle HotSpot (pure newgen) Zing Hiccups&by&Time&Interval& Hiccups&by&Time&Interval& Max"per"Interval" 99%" 99.90%" 99.99%" Max" Max"per"Interval" 99%" 99.90%" 99.99%" Max" 25" 1.8" Hiccup&Dura*on&(msec)& 20" 15" 10" 5" Hiccup&Dura*on&(msec)& 1.6" 1.4" 1.2" 1" 0.8" 0.6" 0.4" 0.2" 0" 0" 100" 200" 300" 400" 500" 600" &Elapsed&Time&(sec)& 0" 0" 100" 200" 300" 400" 500" 600" &Elapsed&Time&(sec)& Hiccup&Dura*on&(msec)& Hiccups&by&Percen*le&Distribu*on& 25" 20" Max=22.656& 15" 10" 5" 0" 0%" 90%" & 99%" 99.9%" 99.99%" & %" Percen*le& Hiccup&Dura*on&(msec)& Hiccups&by&Percen*le&Distribu*on& 1.8" 1.6" 1.4" Max=1.568& 1.2" 1" 0.8" 0.6" 0.4" 0.2" 0" 0%" 90%" & 99%" 99.9%" 99.99%" & %" Percen*le& Low latency trading application 2012 Azul Systems, Inc.

61 Oracle HotSpot (pure newgen) Zing Hiccups&by&Time&Interval& Hiccups&by&Time&Interval& Max"per"Interval" 99%" 99.90%" 99.99%" Max" Max"per"Interval" 99%" 99.90%" 99.99%" Max" 25" 25" Hiccup&Dura*on&(msec)& 20" 15" 10" 5" Hiccup&Dura*on&(msec)& 20" 15" 10" 5" 0" 0" 100" 200" 300" 400" 500" 600" &Elapsed&Time&(sec)& 0" 0" 100" 200" 300" 400" 500" 600" &Elapsed&Time&(sec)& Hiccups&by&Percen*le&Distribu*on& Hiccups&by&Percen*le&Distribu*on& 25" 25" Hiccup&Dura*on&(msec)& 20" 15" 10" 5" Max=22.656& Hiccup&Dura*on&(msec)& 20" 15" 10" 5" Max=1.568& 0" 0%" 90%" & 99%" 99.9%" 99.99%" & %" Percen*le& 0" 0%" 90%" & 99%" 99.9%" 99.99%" & %" Percen*le& Low latency - Drawn to scale 2012 Azul Systems, Inc.

62 Takeaway: Real Java is finally viable for low latency applications GC is no longer a dominant issue, even for outliers 2-3msec worst case case is easy tuning < 1 msec worst case is very doable No need to code in special ways any more You can finally use real Java for everything You can finally 3rd party libraries without worries You can finally use as much memory as you want You can finally use regular (good) programmers 2012 Azul Systems, Inc.

63 Q & A

Enabling Java in Latency Sensitive Environments

Enabling Java in Latency Sensitive Environments Gil Tene, CTO & co-founder, Azul Systems 2011 Azul Systems, Inc. High level agenda Intro, jitter vs. JITTER Java in a low latency application world The (historical)