Disclaimer This presentation may contain product features that are currently under development. This overview of new technology represents no commitme

Transcription

1 VIRT1068BE Virtualizing and Tuning In-Memory Databases VMworld 2017 Content: Not for publication Emad Benjamin, Chief Technologist for Application Platforms, VMware, Inc. #VMworld

2 Disclaimer This presentation may contain product features that are currently under development. This overview of new technology represents no commitment from VMware to deliver these features in any generally available product. Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind. Technical feasibility and market demand will affect final delivery. Pricing and packaging for any new technologies or features discussed or presented have not been determined. VMworld 2017 Content: Not for publication #VIRT1068BE CONFIDENTIAL 2

3 Agenda 1 Overview 2 Sizing the Cache Servers 3 Network Placement of Clients and Cache Servers 4 Sizing the Clients 5 High Availability 6 State of the Industry VMworld 2017 Content: Not for publication 7 Summary and Misc. 8 Q and A #VIRT1068BE CONFIDENTIAL 3

4 Emad Benjamin Graduated with BE, Published undergraduate thesis Independent consultant On C++ and Java, Open source contributions VMware IT - virtualized all Java systems Joined VMware 2005 Blog: vmjava.com Tech lead for vfabric Reference Architecture VMworld In 2013 became Principal Architect Travelled 600k miles Tech Trainer (LiVefire), VMworld, UberConf, Spring1, PEX, ArchConf Presenter, Distributed Systems Specialist and Cloud Native Evangelist 2017 Chief Tech of App Platforms Content: Not for publication #VIRT1068BE CONFIDENTIAL 4

5 Common In Memory DBs Grids (IMDGs) Java based IMDGs VMworld 2017 Content: Not for publication #VIRT1068BE CONFIDENTIAL 5

6 IMDGs Use Distributed Partitioned Data Vertical Scalability DB Sharding Split the big monolithic DB into smaller DBs by functional key, e.g. geographyid or accountsregionid VMworld 2017 Content: Not for IMDG auto sharding/auto-partitioning A distributed data fabric that automatically partitions data across the hardware. In memory data with ability to write to disk or RDBMS publication IMDG Data Fabric #VIRT1068BE CONFIDENTIAL 6

7 The Main Application Platform Topology: Clients-talking to Distributed Cache Servers Clients Using the Data For example, enterprise portals or other applications Read/Write Data Servers Distributed data cache across multiple servers acting as one data cluster IMDG Data Fabric VMworld 2017 Content: Not for publication 100s of clients Dozen Cache Servers or less #VIRT1068BE CONFIDENTIAL 7

8 Common Questions 1. How do you size the cache servers? 2. Where should the clients be on the network in relation to the distributed cache servers? 3. How do you size the clients? Distributed Computing still works the same when being virtualized VMworld 2017 Content: Not for publication #VIRT1068BE CONFIDENTIAL 8

9 How do you size the cache servers? Ask for the total size of the IMDG. Is it 10GB or 1TB...?" VMworld 2017 Content: Not for publication 9

10 Ask your Developers these Questions Avoid letting them dictate how many servers are needed to house the data, unless there is a functional reason why they are splitting into so many scale-out copies For example, if the answer is we have 320GB of data and they ask for 10 nodes of 32GB each, then ask why not 5 nodes of 64GB? Option 1 Option 2 320GB modelled with 10 servers - 32GB servers VMworld 2017 Content: Not for publication 320GB modelled with 5 servers - 64GB servers Option 1 or Option 2? #VIRT1068BE CONFIDENTIAL 10

11 Rule #1 Understand When to Scale Up vs. Scale Out Scale Out Can be Costly VMworld 2017 Content: Not for publication

12 Comparing Test-Scenario-1 of 4 JVMs vs. Test-Scenario-2 of 2 JVMs Scenario-1 4 JVMs of 1GB Heap, Average R/T 166ms Scenario-2 2 JVMs of 2GB Heap, Average R/T 123ms VMworld 2017 Test-Scenario-2 has 26% better response time Content: Not for publication scenario-1 RT scenario-2 RT Scenario 2 doesn't just have better response time -- it has more predictable response time as well! #VIRT1068BE CONFIDENTIAL 12

13 Comparing 4 JVMs vs. 2 JVMs % CPU Scenario-1 (4JVMs, 1GB Heap Each) VMworld 2017 Content: Not for Scenario-2 (2 JVMs, 2GB heap each) has 60% less CPU utilization than scenario-1 publication Time #VIRT1068BE CONFIDENTIAL 13

14 Ask your Developers these Questions How much data will be saved into the IMDG, are we dealing with 100GB or 1TB? Avoid letting them dictate how many servers are needed to house the data, unless there is a functional reason why they are splitting into so many scale-out copies For example, if the answer is we have 320GB of data and they ask for 10 nodes of 32GB each, then ask why not 5 nodes of 64GB? Option 1 Option 2 320GB modelled with 10 servers - 32GB servers VMworld 2017 Content: Not for publication 320GB modelled with 5 servers - 64GB servers Option 1 or Option 2? #VIRT1068BE CONFIDENTIAL 14

15 In fact, even on vrops Internal IMDG The 2 node configuration beats the 8 node configuration by 33% less CPU utilization Option 1 Option 2 VMworld 2017 Content: Not for publication #VIRT1068BE CONFIDENTIAL 15

16 Rule #2 NUMA is your friend Big VMs and Big JVMs are Real Don t Ignore NUMA VMworld 2017 Content: Not for publication For Category 2 Platforms (In-memory DBs) The Golden Rule is 1 JVM per VM per NUMA node

17 ESX Scheduler Each NUMA Node has (256*0.94)/2 120GB VMworld vcpu VMs less than 120GB RAM on each VM If VM is sized greater than 120GB or 8 CPUs, then NUMA interleaving occurs and can cause 30% drop in memory throughput performance Content: Not for publication 256 GB RAM on Server 2 socket server 8 cores per socket #VIRT1068BE CONFIDENTIAL 17

18 NUMA Local Memory with Overhead Adjustment vsphere NUMA Local Memory Optimized VM Size = [Total RAM on Host {(Total RAM On Host*nVMs * 0.01) + 4GB }] /Number of Sockets Physical RAM On vsphere host Physical RAM On vsphere host Total vsphere Overhead Number of VMs On vsphere host 1% RAM overhead Number of Sockets On vsphere host vsphere RAM overhead VMworld 2017 Content: Not for publication #VIRT1068BE CONFIDENTIAL 18

19 NUMA Local Memory with Overhead Adjustment Rough Approximation Irrespective of Number of VMs and vsphere Versions Assumes 15% overhead high approximation but always works Intel NUMA Local Memory Optimized VM Size = (0.85 * Total RAM on Host) /Number of Sockets Because of Stacked NUMA nodes, it is divided 2x compared to the Intel Chips AMD NUMA Local Memory Optimized VM Size = (0.85 * Total RAM on Host) / 2 * Number of Sockets VMworld 2017 Content: Not for publication #VIRT1068BE CONFIDENTIAL 19

20 Max of Max NUMA range 1 VM Per NUMA Node and 1 JVM per VM, take all the NUMA memory and leave no CPU cores behind 16 vcpu VMs less than 120GB RAM on each VM with numa.vcpu.preferht = TRUE Minimum of Max NUMA range 8 vcpu VMs less than 120GB RAM on each VM VMworld 2017 Most Conservative VM and Heap Size 1 VM per NUMA node, 8vCPUs, VM Mem = 108GB = ((256*0.85))/2 Tuned for Extreme Performance 1 JVM per VM, JVM will have 120*0.75 GB - 1 GB for OS => 90GB Heap A must-config for in memory DBs, can work for Web Apps, but would be expensive/challenging to fill up such a big heap space for Web Apps/Clients Content: Not for publication Cache Mem Size IMDG JVM Heap per Node = (108GB -1GB for OS) * 0.8 = 86GB 20

21 Most Conservative VM and Heap Size 1 VM per NUMA node, 8vCPUs, VM Mem = 108GB = ((256*0.85))/2 Cache Mem Size IMDG JVM Heap per Node = (108GB -1GB for OS) * 0.8 = 86GB Option 1 320GB modelled with 10 servers - 32GB servers (VMs with 39GB RAM) Option 3- Best option for the type of hardware/esxi server: 2 socket 8 core per socket, 256GB RAM Option / 86 = 3.72 VMs => 4 IMDGs/Cache Nodes => => 4 VMs Option 2 320GB modelled with 5 servers - 64GB servers (VMs with 78GB RAM) VMworld 2017 Content: Not for publication 320GB modelled with 4 VMs, 108GB RAM, 8vCPU, and 86GB Heap on each JVM. Total of 2 ESXi hosts. #VIRT1068BE CONFIDENTIAL 21

22 Haswell Chips Have a feature called CoD which is Cluster On Die If CoD is enabled at BIOS level, ESXi6 onwards supports this, ESXi5.5u1 and u2 did not support it, but supported in ESXi5.5 u3 If it is enabled then split into 2 NUMA nodes Inter CoD memory migrations less costly than interleaving to remote sockets VMworld 2017 Content: Not for Interleaving between CoDs from remote sockets, cost 10% in 2 socket systems and 30%> in 4 sockets systems publication #VIRT1068BE CONFIDENTIAL 22

23 Haswell Chip - Largest NUMA Node VM Size Intel NUMA Local Memory Optimized VM Size = (0.85 * Total RAM on Host) / 2 * Number of Sockets VMworld 2017 Content: Not for publication #VIRT1068BE CONFIDENTIAL 23

24 Where should the clients be on the network in relation to the distributed cache servers? No, sorry Mr. Developer, the Network is not flat latency increase and throughput drop are real due to the network VMworld 2017 Content: Not for publication 24

25 Option-1 Adequate Scale-out, max Scale-UP, NUMA Bound 8 cache servers, 16 clients ESXi Host Gen9 with 2 socket servers of 12 cores per socket, 1TB RAM 8 Cache Server VMs with 192GB, set Mem reservation Each VM to run 1 GemFire JVM with 156GB heap space Each VM to have 12 vcpus configured with all cores allocated on 1 socket Enable High Latency Sensitivity on VM 2 Client VMs of 4vCPU each, and 32GB on each ESXi Host Apply below JVM config VMworld 2017 Servers 8 Cache Servers, 16 clients 1.248TB IMDGs heap Content: Not for publication -J-Xms156g -J Xmx156g -J-Xmn46g -J-XX:+UseConcMarkSweepGC J-Xss2048k -J-XX:+UseParNewGC -J XX:CMSInitiatingOccupancyFraction=75 -J XX:+UseCMSInitiatingOccupancyOnly -J-XX:+ScavengeBeforeFullGC -J-XX:TargetSurvivorRatio=80 -J-XX:SurvivorRatio=8 -J-XX:+DisableExplicitGC -J-XX:MaxTenuringThreshold=15 -J-XX:ParallelGCThreads=6 -J-XX:+AlwaysPreTouch -J-XX:+OptimizeStringConcat -J-XX:+UseCompressedStrings -J-XX:+UseStringCache Clients #VIRT1068BE CONFIDENTIAL 25

26 Option-1 Adequate Scale-out, max Scale-UP, NUMA Bound, 8 cache servers, 16 clients additional notes Since we are running the GemFire clients and Cache Servers on the same host, the increased chatter between the GemFire client and The GemFire Cache server may cause the scheduling to put the client and the server on the same NUMA node, and then a resource imbalance may ensue To avoid this set /Numa/LocalityWeightActionAffinity as 0 earch.do?language=en_us&cmd=displaykc& externalid= VMworld 2017 Content: Not for publication #VIRT1068BE CONFIDENTIAL 26

27 Option-2 Large Scale-out, Medium Scale-UP, NUMA Bound 16 cache servers, 16 clients ESXi Host Gen9 with 2 socket servers of 12 cores per socket, 1TB RAM 16 Cache Server VMs with128gb, set Mem reservation Each VM to run 1 GemFire JVM with 78GB heap space Each VM to have 8 vcpus configured with all cores allocated on 1 socket Enable High Latency Sensitivity on VM 2 Client VMs of 4vCPU each, and 32GB on each ESXi host, total 16clients NOTE: start cache server VMs first before starting clients Apply below JVM config VMworld J-Xms78g -J Xmx78g -J-Xmn23g -J-XX:+UseConcMarkSweepGC J-Xss2048k -J-XX:+UseParNewGC -J XX:CMSInitiatingOccupancyFraction=75 -J XX:+UseCMSInitiatingOccupancyOnly -J-XX:+ScavengeBeforeFullGC -J-XX:TargetSurvivorRatio=80 -J-XX:SurvivorRatio=8 -J-XX:+DisableExplicitGC -J-XX:MaxTenuringThreshold=15 -J-XX:ParallelGCThreads=4 -J-XX:+AlwaysPreTouch -J-XX:+OptimizeStringConcat -J-XX:+UseStringCache NOTE: Since we are running the GemFire clients and Cache Servers on the same host, the increased chatter between the GemFire client and The GemFire Cache server may cause the scheduling to put the client and the server on the same numa node, and then a resource imbalance may ensue To avoid this set /Numa/LocalityWeightActionAffinity as 0 16 Cache Servers, 16 clients 1.248TB IMDGs heap Content: Not for publication Clients #VIRT1068BE CONFIDENTIAL 27

28 Sizing IMDG Clients VM Memory (98+12=110GB) Leaving space for future growth Rounding up to 128GB JVM Memory (98G) JVM Max Heap -Xmx (78G) Guest OS Memory Java Stack PermGen Initial Heap set mem Reservation to 128GB, 8vCPU VM 12GB used by OS amd other processes -Xss per thread (256k*1000) Other mem (=256M) VMworld 2017 Content: Not for publication -Xms (78G) #VIRT1068BE CONFIDENTIAL 28

29 Option-3 Separate Clients from Cache Servers 16 clients, 8 cache servers ESXi Host Gen9 with 2 socket servers of 12 cores per socket, 1TB RAM 8 Cache Server VMs with 192GB, set Mem reservation Each VM to run 1 GemFire JVM with 156GB heap space Each VM to have 12 vcpus configured with all cores allocated on 1 socket Enable High Latency Sensitivity on VM 2 Client VMs of 4vCPU each, total 16 clients Apply below JVM config VMworld 2017 Servers 8 Cache Servers, 16 clients 1.248TB IMDGs heap Content: Not for publication Clients -J-Xms156g -J Xmx156g -J-Xmn46g -J-XX:+UseConcMarkSweepGC J-Xss2048k -J-XX:+UseParNewGC -J XX:CMSInitiatingOccupancyFraction=75 -J XX:+UseCMSInitiatingOccupancyOnly -J-XX:+ScavengeBeforeFullGC -J-XX:TargetSurvivorRatio=80 -J-XX:SurvivorRatio=8 -J-XX:+DisableExplicitGC -J-XX:MaxTenuringThreshold=15 -J-XX:ParallelGCThreads=6 -J-XX:+AlwaysPreTouch -J-XX:+OptimizeStringConcat -J-XX:+UseStringCache #VIRT1068BE CONFIDENTIAL 29

30 Option-3a Separate Clients from Cache Servers 10 clients, 8 cache servers ESXi Host Gen9 2 socket servers of 12 cores per socket, 1TB RAM 8 Cache Server VMs of 192GB, set Mem reservation Each VM to run 1 GemFire JVM with 156GB heap space Each VM to have 12 vcpus configured with all cores allocated on 1 socket Enable High Latency Sensitive on VM 2 Client VMs of 4vCPU each, total 10 clients Apply below JVM config VMworld 2017 Servers 8 Cache Servers, 10 clients 1.248TB IMDGs heap Spare ESXi host Content: Not for publication Clients -J-Xms156g -J Xmx156g -J-Xmn46g -J-XX:+UseConcMarkSweepGC J-Xss2048k -J-XX:+UseParNewGC -J XX:CMSInitiatingOccupancyFraction=75 -J XX:+UseCMSInitiatingOccupancyOnly -J-XX:+ScavengeBeforeFullGC -J-XX:TargetSurvivorRatio=80 -J-XX:SurvivorRatio=8 -J-XX:+DisableExplicitGC -J-XX:MaxTenuringThreshold=15 -J-XX:ParallelGCThreads=6 -J-XX:+AlwaysPreTouch -J-XX:+OptimizeStringConcat -J-XX:+UseStringCache #VIRT1068BE CONFIDENTIAL 30

31 High Level Pros and Cons of the Various Options Option Number of Cache Servers Cache Server Heap Size (GB) Number of Clients Cache Server VM Size (GB, vcpu) Client VM Size (GB, vcpu) Cluster Totals Heap, RAM, Total vcpu for Cache Servers Latency Comments performance, scalability, redundancy , 12 32, TB,1.5TB, 96 Low High perf, excellent scalability (optimal vertical and horizontal mix, good redundancy) , 8 32, TB, 2TB, 128 Lowest (if no CPU skewing due to client load) Medium Perf, uses more compute than Option-1, may impeded response time since client VMs are sharing the NUMA node with the cache servers. Cache servers also have 30% less vcpus than Option ,12 32, TB, 1.5TB, 96 medium This is option is a variant of Option-1, however the clients have been totally separated from the cache servers, this will have the best scalability, since it optimizes for independent scalability of client from the server, it should outperform both Option-1 and 2 3a ,12 32, TB, 1.5TB, 96 medium This is basically Option-3, but it has reduced clients from 16 to 12, this is designed to reduce the hardware cost, and leave enough spares in a cluster for maintenance tasks. VMworld 2017 Content: Not for publication Most Customers chose Option-3 (or 3a) as good balance between performance, redundancy, and scalability, but those not happy with Option-3 can choose Option-1. Option-2, requires lots of hand holding, not consistent as it depends on behavior of client and guaranteeing that it doesn t impact the server, it also uses more CPU and has more than double the cache servers. #VIRT1068BE CONFIDENTIAL 31

32 How do you size the clients? VMworld 2017 Content: Not for publication 32

33 Sizing IMDG Clients VM Memory (5088m) JVM Memory (4588m) JVM Max Heap -Xmx (4096m + 256m for permgen) Guest OS Memory Java Stack PermGen Initial Heap set mem Reservation to 5088m 500m used by OS -Xss per thread (256k*100) Other mem (=217m) VMworld 2017 Content: Not for publication -Xms (4096m) #VIRT1068BE CONFIDENTIAL 33

34 Sizing IMDG Clients - Understanding Memory Sizing VM Memory = Guest OS Memory + JVM Memory JVM Memory = JVM Max Heap (-Xmx value) + JVM Perm Size (-XX:MaxPermSize) + NumberOfConcurrentThreads * (-Xss) + other Mem If JVM Heap (-Xmx) is N then JVM Memory is (1.1 to 1.25) * N For example, if heap is 4GB then JVM Memory = (1.1 to 1.25) *4= 4.5 to 4.8GB Always use memory based sizing, so if a system needs 400GB made from 100 JVMs of 4GB each, then this is the capacity you are working to. CPU consumption will be secondary VMworld 2017 Content: Not for publication #VIRT1068BE CONFIDENTIAL 34

35 IMDGs Level HA vs Infrastructure HA and DRS VMworld 2017 Content: Not for publication

36 GemFire Redundancy Zones vs DRS Affinity/Anti-Affinity Rules GemFire Cluster VMworld 2017 Content: Not for publication #VIRT1068BE CONFIDENTIAL 36

37 GemFire Redundancy Zones vs DRS Affinity/Anti-Affinity Rules vsphere DRS Cluster GemFire Redundancy Zone 1 Gem1 Gem2 Gem1R Gem2R GemFire Cluster VMworld 2017 Content: Not for publication or distribution GemFire sees 8 physical nodes, it doesn t know they are VMs GemFire Redundancy Zone 2 37

38 GemFire Redundancy Zones vs DRS Affinity/Anti-Affinity Rules vsphere DRS Cluster VM1 Anti Affinity Rules: VM1-VM5 VM2-VM6 VM2 GemFire Redundancy Zone 1 VM3 VM4 VM5 VM6 VM7 Gem1 VM8 Gem2 Gem1R Gem2R GemFire Cluster VMworld 2017 Content: Not for publication GemFire Redundancy Zone 2 38

39 Skillset Needed and State of the Industry From Infrastructure Engineer to SRE VMworld 2017 Content: Not for publication 39

40 How We Got Here? What is Really Broken? Information Silos between Business, Application Teams, and Infrastructure Teams leads to excessive over provisioning of hardware in order to meet SLAs Business Comms Gap Apps/Data Comms Gap Infrastructure 44% of new applications failed to meet performance SLA (1) 90% of companies allocate at least 2X more 44% fail App Perf cloud capacity than needed to ensure performance (2) 2.X Hardware VMworld 2017 Content: Not for publication (1) 2014 Riverbed Tecnologies - The Transformers; (2) 2014 ElasticHosts CIO Study #VIRT1068BE CONFIDENTIAL 40

41 Platform Engineering Most misunderstood discipline Developers size/wrongly-size this, but Ops own it. A battle is brewing over who seeks control Business Sponsors CIO VP Apps VP Ops Enterprise Architect Business Process IT Services Apps Data Ops/Infrastructure Governance, Architecture, Program and Project Mgmt. Platform Engineering An intersection of three disciplines most commonly known as Site Reliability Engineering (SRE) Achieve Contracted SLA to Business Sponsor VMworld 2017 Content: Not for publication Platform Engineering Focus 41

42 Platform Engineering/ SRE = Apps Runtime + Ops/Infra Refer to my blog: vmjava.com VMworld 2017 Content: Not for publication In Memory DB Messaging Virtualization Pivotal tcserver Application Container Spring Framework #VIRT1068BE CONFIDENTIAL 42

43 Summary and Misc. Understand the total memory size of the IMDG Understand the NUMA architecture and use the calculation approach we covered, NUMA building block size becomes the divisor for all calculations vmotion works, but you do have to test for it, 10GigE, use vmxnet3 drivers If some vendors say they won t support vmotion just reach out to me ebenjamin@vmware.com In some cases you should use the NOOP scheduler at the OS level Set memory reservation SRE type of skills are needed to manage IMDGs NOTE: Java has +UseNUMA flag that is not compatible with Concurrent Mark and Sweep GC (CMS) GC used below No need to use large pages with in-memory DB and the CMS GC Use the GC tuning configurations examples discussed here. Of course adjust heap up or down based on your use case, and parallelgcthreads to 50% of available CPU to begin with, and then adjust upwards by 1 as needed. The below uses a combination of GC ParNewGC and CMS, it is very adequate for 90% of cases. If not, can consider G1GC. VMworld 2017 Content: Not for publication -J-Xms156g -J Xmx156g -J-Xmn46g -J-XX:+UseConcMarkSweepGC J-Xss2048k -J-XX:+UseParNewGC -J XX:CMSInitiatingOccupancyFraction=75 -J XX:+UseCMSInitiatingOccupancyOnly -J-XX:+ScavengeBeforeFullGC -J-XX:TargetSurvivorRatio=80 -J-XX:SurvivorRatio=8 -J-XX:+DisableExplicitGC -J-XX:MaxTenuringThreshold=15 -J-XX:ParallelGCThreads=6 -J-XX:+AlwaysPreTouch -J-XX:+OptimizeStringConcat -J-XX:+UseCompressedStrings -J-XX:+UseStringCache #VIRT1068BE CONFIDENTIAL 43

44 Types Of Questions to Ask about the IMDG Cluster 1. How many VMs in the Cassandra cluster? 2. What is the size of the VMs from vcpu and memory perspective? 3. How many ESXi hosts are used/dedicated to these VMs? 4. What is the hardware architecture of the ESXi servers, how many sockets, how many cores per socket, how much RAM on the host? 5. Is there only one Cassandra node per VM, or more? 6. Can you provide the Cassandra node configuration that shows the JVM options, how much heap is being allocated? 7. How much data are they writing to disk, what kind of storage? 8. NOTE: When they say there is latency, this is typically a client (smaller app) trying to access the Cassandra cluster. VMworld 2017 Content: Not for publication Question to ask: Where on the network is this client in relation to the Cassandra cluster? 9. How did you measure latency on the physical system vs the virtual? 10.Have you set reservation on the VMs? #VIRT1068BE CONFIDENTIAL 44

45 VMworld 2017 Content: Not for publication

46 VMworld 2017 Content: Not for publication