It s Good to Have (JVM) Options Chris Hansen / Sr Engineering Manager / @cxhansen http://bit.ly/2g74cnh Tori Wieldt / Technical Evangelist / @ToriWieldt JavaOne 2017
Safe Harbor This presentation and the information herein (including any information that may be incorporated by reference) is provided for informational purposes only and should not be construed as an offer, commitment, promise or obligation on behalf of New Relic, Inc. ( New Relic ) to sell securities or deliver any product, material, code, functionality, or other feature. Any information provided hereby is proprietary to New Relic and may not be replicated or disclosed without New Relic s express written permission. Such information may contain forward-looking statements within the meaning of federal securities laws. Any statement that is not a historical fact or refers to expectations, projections, future plans, objectives, estimates, goals, or other characterizations of future events is a forward-looking statement. These forward-looking statements can often be identified as such because the context of the statement will include words such as believes, anticipates, expects or words of similar import. Actual results may differ materially from those expressed in these forward-looking statements, which speak only as of the date hereof, and are subject to change at any time without notice. Existing and prospective investors, customers and other third parties transacting business with New Relic are cautioned not to place undue reliance on this forward-looking information. The achievement or success of the matters covered by such forward-looking statements are based on New Relic s current assumptions, expectations, and beliefs and are subject to substantial risks, uncertainties, assumptions, and changes in circumstances that may cause the actual results, performance, or achievements to differ materially from those expressed or implied in any forward-looking statement. Further information on factors that could affect such forward-looking statements is included in the filings New Relic makes with the SEC from time to time. Copies of these documents may be obtained by visiting New Relic s Investor Relations website at ir.newrelic.com or the SEC s website at www.sec.gov. New Relic assumes no obligation and does not intend to update these forward-looking statements, except as required by law. New Relic makes no warranties, expressed or implied, in this presentation or otherwise, with respect to the information provided. 2
Too Many Options.x10 Source: http://docs.oracle.com/javase/8/docs/technotes/tools/unix/java.html Source: https://docs.oracle.com/javase/8/docs/technotes/tools/windows/java.html 3
4
Focus 5
Visualization software courtesy wordle.net 6
Whence Data? New Relic monitors 500,000 JVMs running apps and services We report JVM options to aid in troubleshooting and discovery The JVM option and other data here is fully anonymized Filtered for Oracle Hotspot JVM only 7
What do we count as the use of an option? The key... -Xmx512M -Xloggc:/var/log/jvmgc.log -XX:MaxPermSize=64M -XX:+PrintGCDetails Value, sort of 8
Most Common JVM Options 9
Categories of Options Memory sizing Diagnostics GC tuning Other tuning 10
Basics of Java Heap and GC Source: Jörge Prante http://jprante.github.io/2012/11/28/elasticsearch-java-virtual-machine-settings-explained.html 11
Object Tenure req sesh Eden sesh S0 sesh S1 sesh Tenured (i.e. Old) 12
Source: https://publicdomainvectors.org/en/free-clipart/snellen-eye-test-chart-image/33940.html 13
Resource Limits (Object Heap) -Xmx* : Heap max (87%) -Xms* : Heap initial (64%) -Xss* : Per-thread stack size (11%) -XX:MaxHeapFreeRatio=* : When to shrink heap (6%) -XX:MinHeapFreeRatio=* : When to expand heap (5%) -Xmn* : Size of young generation (6%) -XX:SurvivorRatio=* : Eden : Survivor (6%) 14
Resource Limits (PermGen/Metaspace) -XMaxPermSize=* : PermGen max (29%) -XPermSize=* : PermGen threshold for GC (12%) -XX:MaxMetaspaceSize=* : Class metadata max [unlimited] (13%) -XX:MetaspaceSize=* : Class metadata threshold for GC (7%) 15
Garbage Collector Diagnostics -Xloggc:* : Log GC events to a file, which sets... (23%) -XX:+PrintGC, -verbose:gc : Log at a basic level (18%) -XX:+PrintGCDetails : Log at a detailed level (19%) -XX:+PrintGCTimeStamps : Include relative timestamp (18%) -XX:+PrintGCDateStamps : Include absolute timestamp (16%) -XX:+PrintHeapAtGC : Include more heap detail (10%) -XX:+PrintTenuringDistribution : Include post-gc tenure summary (8%) -XX:+PrintGCApplicationStoppedTime : Log safepoint time (6%) 16
-XX:+PrintGC (-Xloggc:*, -verbose:gc) Java HotSpot(TM) 64-Bit Server VM (25.144-b01) for bsd-amd64 JRE (1.8.0_144-b01), built on Jul 21 2017 22:07:42 by "java_re" with gcc 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.11.00) Memory: 4k page, physical 16777216k(780800k free) /proc/meminfo: CommandLine flags: -XX:MaxHeapSize=1610612736 -XX:MaxNewSize=268435456 -XX:+PrintGC -XX:+PrintGCTimeStamps -XX:SurvivorRatio=1 -XX:TargetSurvivorRatio=2 -XX:+UseParallelGC 1.087: [GC (Allocation Failure) 58880K->8704K(466432K), 0.0064928 secs] 1.515: [GC (Allocation Failure) 67584K->12147K(466432K), 0.0090999 secs] 2.197: [GC (Allocation Failure) 71027K->19407K(466432K), 0.0110946 secs] 2.644: [GC (Allocation Failure) 78287K->31108K(525312K), 0.0199316 secs] 4.221: [GC (Metadata GC Threshold) 136969K->18311K(525312K), 0.0111167 secs] 4.232: [Full GC (Metadata GC Threshold) 18311K->17905K(525312K), 0.0390435 secs] 17
-XX:+PrintTenuringDistribution 1.112: [GC (Allocation Failure) Desired survivor size 59244544 bytes, new threshold 7 (max 15) 58880K->8719K(466432K), 0.0059594 secs] 1.627: [GC (Allocation Failure) Desired survivor size 59244544 bytes, new threshold 7 (max 15) 67599K->12121K(466432K), 0.0112993 secs] 2.368: [GC (Allocation Failure) Desired survivor size 59244544 bytes, new threshold 7 (max 15) 71001K->19519K(466432K), 0.0162481 secs] 2.808: [GC (Allocation Failure) Desired survivor size 59244544 bytes, new threshold 7 (max 15) 78399K->31713K(525312K), 0.0232932 secs] 4.377: [GC (Metadata GC Threshold) Desired survivor size 32505856 bytes, new threshold 6 (max 15) 139103K->18620K(525312K), 0.0111938 secs] 4.388: [Full GC (Metadata GC Threshold) 18620K->18060K(525312K), 0.0386663 secs] 18
-XX:+PrintGCDetails 1.076: [GC (Allocation Failure) [ PSYoungGen: 58880K->8689K(116736K)] 58880K->8689K(466432K), 0.0067968 sec s] [Times: user=0.01 sys=0.00, real=0.01 secs] 1.601: [GC (Allocation Failure) [PSYoungGen: 67569K->12219K(116736K)] 67569K->12219K(466432K), 0.0109878 s ecs] [Times: user=0.02 sys=0.00, real=0.01 secs] 2.248: [GC (Allocation Failure) [PSYoungGen: 71099K->19431K(116736K)] 71099K->19431K(466432K), 0.0104851 secs] [Times: user=0.03 sys=0.01, real=0.01 secs] 2.667: [GC (Allocation Failure) [PSYoungGen: 78311K->31397K(175616K)] 78311K->31397K(525312K), 0.0212616 secs] [Times: user=0.04 sys=0.01, real=0.03 secs] 4.247: [GC (Metadata GC Threshold) [PSYoungGen: 140097K->18470K(175616K)] 140097K->18470K(525312K), 0.0114535 secs] [Times: user=0.02 sys=0.00, real=0.01 secs] 4.259: [Full GC (Metadata GC Threshold) [PSYoungGen: 18470K->0K(175616K)] [ParOldGen: 0K->17922K(349696K)] 18470K->17922K(525312K), [Metaspace: 20851K->20847K(1069056K)], 0.0394262 secs] [Times: user=0.11 sys=0.01, real=0.04 secs] 19
-XX:+PrintHeapAtGC {Heap before GC invocations=2 (full 0): PSYoungGen total 116736K, used 67610K [0x00000007b0000000, 0x00000007baa80000, 0x00000007c0000000) eden space 58880K, 100% used [0x00000007b0000000,0x00000007b3980000,0x00000007b3980000) from space 57856K, 15% used [0x00000007b3980000,0x00000007b42069b0,0x00000007b7200000) to space 57856K, 0% used [0x00000007b7200000,0x00000007b7200000,0x00000007baa80000) ParOldGen total 349696K, used 0K [0x0000000760000000, 0x0000000775580000, 0x00000007b0000000) object space 349696K, 0% used [0x0000000760000000,0x0000000760000000,0x0000000775580000) Metaspace used 16469K, capacity 16654K, committed 16768K, reserved 1064960K class space used 2078K, capacity 2139K, committed 2176K, reserved 1048576K 1.553: [GC (Allocation Failure) 67610K->11771K(466432K), 0.0087631 secs] Heap after GC invocations=2 (full 0): PSYoungGen total 116736K, used 11771K [0x00000007b0000000, 0x00000007be400000, 0x00000007c0000000) eden space 58880K, 0% used [0x00000007b0000000,0x00000007b0000000,0x00000007b3980000) from space 57856K, 20% used [0x00000007b7200000,0x00000007b7d7ed80,0x00000007baa80000) to space 57856K, 0% used [0x00000007b3980000,0x00000007b3980000,0x00000007b7200000) ParOldGen total 349696K, used 0K [0x0000000760000000, 0x0000000775580000, 0x00000007b0000000) object space 349696K, 0% used [0x0000000760000000,0x0000000760000000,0x0000000775580000) Metaspace used 16469K, capacity 16654K, committed 16768K, reserved 1064960K class space used 2078K, capacity 2139K, committed 2176K, reserved 1048576K } 20
Garbage Collector Diagnostics (Files) -XX:+UseGCLogFileRotation (8%) -XX:NumberOfGCLogFiles=* (9%) -XX:GCLogFileSize=* (9%) 21
Heap Dump -XX:+HeapDumpOnOutOfMemoryError : hprof heap dump (20%) -XX:HeapDumpPath=* : Path and file name for.hprof file (20%) Analyze hprof file with VisualVM, jhat, or your favorite profiler 22
Choosing an Alternate Garbage Collector ParallelGC: Optimized for throughput (60% default) CMS (Concurrent Mark Sweep): Optimized for low pause (28%) G1: Low pause and high throughput (12%) Serial GC: Good for batch processes 23
Other Optimizations -XX:+UseCompressedOops : Compress 64-bit ordinary object pointers (12%) -XX:+DisableExplicitGC : Ignore System.gc() (11%) -XX:+AggressiveOpts : Use experimental runtime optimizations (5%) -XX:-OmitStackTraceInFastThrow [5] : Java 5 JIT optimization (7%) -XX:-TieredCompilation [7-] : Use simple JIT compilation (5%) 24
What about the least used options? -ea, -enableassertions (<1%) Watch for typos! Quoting/spacing Use proper key-value delimiter: :, =,, etc. Case sensitive JVM won t always warn that these are ignored Watch for multiple with the same key! (Last one wins) 25
Test and Monitor the Effects! -XX:SurvivorRatio=1 [v 8] -XX:TargetSurvivorRatio=2 [v 50] -XX:MaxNewSize=256M [v ergo] 26
Conclusion People want diagnostics, especially for object allocation and collection Know your garbage collector. Go to CON4479 this afternoon. Consider this MOOC. Defaults are good! Test and monitor your option changes. 27
Bonus Content! Crunching the Docker Numbers Full details https://blog.newrelic.com/2016/06/16/docker-container-infographic/ 28
Glamour Stats New Relic has been using and monitoring containers since 2014 Monitored 418 million containers in the last two years Monitor 1.5 million containers average a day Average customer has more than 800 containers running at any given moment Our top ten customers (by Docker container usage) spin up more than 92,000 containers a day 29
Java and Docker Docker is used on all kinds of applications: PHP (33%) Java (31%) Ruby (29%) Node (28%) Python (11%).NET (8%) 30
Docker Growth Average Number of Containers 2015 2016 96 280 Average # of containers per company for the year Greatest number of containers in a single day for a single company Greatest number of containers in a year for a single company 28,000 1,596 135,000 4,800,000 31
Pets to Cattle to Bacteria * Average life of containers: 13 hours (2015) to 9.25 hours (2016) * Average life of containers running less than an hour: 10 minutes (2015) to 8 minutes (2016). * 20% of all containers run for less than one minute. 32
Conclusions Container usage continues to grow From VM replacement to ephemeral Paradigms are shifting 33
fin This page is intentionally left blank 34