Virtualization Big Applications. Charu Chaubal Senior Architect, Technical Marketing 6 November, 2008

Transcription

1 Virtualization Big Applications Charu Chaubal Senior Architect, Technical Marketing 6 November, 2008

2 Overview Myths and Realities Case Studies CPU Intensive Workloads I/O and Network Intensive Workloads Applications SQL Server, Oracle, SAP, MS Exchange Futures

3 Myths & Perceptions Virtualization is only for smaller, non-critical systems ESX s s resource Near-linear Large ESX can CPU, drive Scalability Memory over management Databases, controls 100,000 allows and I/O many IOPS footprint and virtual 500 allow Services, resources and High to be applications machines disks per host, to have be End allocated Web Workloads to create enough consolidated been the for focus 85 without average for are isolation all virtualizable between additional ESX databases since overhead 3.0 critical applicatons Large applications have high overhead I/O Intensive Applications have too much IO to be virtualized Performance management will become complex I can t run too many performance critical VMs at the same time

4 Why Virtualize Larger or Mission Critical Apps? For Consolidation Leverage resource sharing: CPU, Memory, I/O Connectivity Consolidate several peaky workloads to increase utilization Need to leverage multi-core platforms most efficiently For Availability Minimize downtime via Virtual Infrastructure HA/VMotion/DRS For Operational Efficiency Want to leverage virtualization infrastructure already in place for other apps Take advantage of increased administrative flexibility

5 Evolution of VI Performance for Large Apps 100% Mission Critical Apps ESX 2.x VI 3.0 VI 3.5 Future General Population Of Apps 30-60% Overhead 2-way SMP 3.6 GB VM RAM 16 CPU support 4 CPU scaling 64GB Phys RAM <10,000 IOPS 380mbits max 20-30% Overhead 2-way SMP Scaling 16 GB VM RAM 16 core support 64GB Phys RAM 800mbits Gen-1 HW Virt 10-20% Overhead 4-way VSMP Scaling 64-bit OS Support 64GB VM RAM 64 core hosts 256GB Phys RAM 100,000 IOPS 9 Gigabits Gen-2 HW Virt 2-10% Overhead 8-way VSMP Scaling 256GB VM RAM 128 core scaling 512GBPhys RAM 200,000 IOPS 40 Gigabits 512 VMs Ability to satisfy Performance Demands

6 Enterprise-Class Performance (ESX 3.5) CPU Virtual SMP with 4 processor scalability 90%-95% performance of native server IO Extreme Disk IO supports 100,000+ DB IO per second Wire speed Network IO supports 9 Gbps Memory 64 GB per VM 256 GB per host Advanced memory management

7 Consolidation is a way to use all those cores VMWare ESX Scaling: Keeping up with core counts ESX 3.5 ESX Future Virtualization provides a means to exploit the hardware s increasing parallelism!" Most applications don t scale beyond 4/8 way

8 Example: Hardware Out-scales Web Servers

9 Consolidation & Sizing CPU Utilization Distribution Consolidation targets are often <30% Utilized Windows average utilization: 5-8% Linux/Unix average: 10-35% Number of Systems % CPU Utilization

10 Sizing and Requirements Virtual Machine sizing is different to Physical Don t just take the #cpus in the physical system as the vcpu requirement Many Physical systems are sized for the peak utilization for with ample headroom for future growth As a result, utilization is often very low in physical systems With virtual machines, it s not necessary to build headroom For example, many databases running on 4-cpu systems can easily run in a 2-vcpu guest Easy migration to future headroom: e.g. 8vcpu support

11 ! "#$%$##&

12 Common Questions for Big Apps Is VMware ESX able to scale up to the required number of CPUs/cores? Are the I/O overheads of virtualization too high? Resource MS Exchange Requirement (Exchange 2007) SMP Scaling Large node: 4 cores 4 vcpu VMware Capability Hardware Scaling Multiple VMs$ 32 pcpu today Allows 8x4vcpu VMs Memory 4 cores 64 GB per VM Network 2000 users 980 Mbits on GBE >5Gbits on multiple GBE 9.8 Gbits on 10GBE I/O 1000 users >100,000 IOPS on large disk configurations

13 ' "#$%$##&

14 CPU - Memory - I/O Ratios for Applications Assumption is that the core is 80% busy Lighter VMs will need proportionally less resource Application Memory (per core) Disk IOPS( (per core) Exchange 6GB 500 Oracle 2GB 320 SQLserver 2GB 240 DB2 2GB 160 Desktop 6GB 100 Web Servers 2GB 320

15 ) ##*+," "#$%$##&

16 Databases

17 Databases: Why Use VMs Rather than DB Virtualization? Virtualization at hypervisor level provides the best abstraction Each DBA has their own hardened, isolated, managed sandbox Strong Isolation Security Performance/Resources Configuration Fault Isolation Scalable Performance Low-overhead virtual Database performance Efficiently Stack Databases per-host

18 Measuring the Performance of DB Virtualization

19 Consolidating Microsoft SQL Server on ESX C PU utiliz ation & Throughput opm opm % cpu average %CPU 0 1-vm 2-vm 3-vm 4-vm 0 # # $% $$ &'% (' $# ' (&$ # $ $' $$ $&& %& $(&

20 Oracle Performance (Response time)

21 Databases: Tuning Recommendations Use 64-bit Database Add enough memory to cache DB, reduce I/O Use Direct-IO high performance uncached path in the Guest Operating System Use Asynchronous I/O to reduce system calls Use Large MMU Pages Optimize Storage Layout, # of Disk Spindles

22 Databases: Storage Configuration Virtualization-specific Considerations VMFS or RDM Fibre Channel, NFS or iscsi Partition Alignment Multiple storage paths Standard Best Practices OS/App, Data, Transaction Log and TempDB on separate physical spindles RAID 10 or RAID5 for Data, RAID 1 for logs Queue depth and Controller Cache Settings

23 SAP, Exchange

24 SAP Virtualization on ESX 3.5 CI DB 1 x vcpu 4 Gb CI DB 2 x vcpu 8 Gb DIA x 3 CI DB 4 x vcpu 16 Gb # of SD Users % Native Virtual # of vcpu

25 Scaling Exchange (Natively) On a Single Server Is storage the limit? No. Exchange 2007 on Windows Server makes excellent use of correctly configured storage. Is CPU the limit? Possibly. The Mailbox Server s recommended maximum is eight cores. Is memory the limit? Possibly. The maximum recommended memory allocation is 32G per server Maximum mailboxes: 8,000 Eight cores at recommended 1,000 mailboxes per core This results in providing 3.75MB/user at the server; this is in the middle of the 2-5MB/user recommendation

26 Multi-VM Scaling of Exchange on VI3 Mailboxes 16K 14K 12K 10K 8K 6K 4K 2K Building blocks stay within Microsoft recommendations: 1,000 mailboxes/core 5MB/mailbox Maximum performance! 5 MB/mailbox perf. threshold 1,000 mailboxes/core Eight core limit recommended by Microsoft CPUs (or cores) Three building blocks breaks through previrtual memory boundaries Five building blocks shatters pre-virtual CPU limitations Eight building blocks enables 16,000 users VI3 provides native-matching performance and complete resource utilization

27 Record-setting Server Performance Building Block Performance Mailboxes 450 Mailboxes SendMail Latency Latency (ms) VM 2 VMs 3 VMs 4 VMs 5 VMs 6 VMs 7 VMs 8 VMs 0 Virtual Machines

28 Performance Futures

29 Future: the Virtual Datacenter OS

30 Future (2009) vcompute Paravirtualized storage device enables over 200,000 i/o operations per second VMDirectPath technology for direct device access to a VM enables the lowest possible overhead Take Further Advantage of Hardware Assist: Instruction level, Memory Management and Device Driver Scalability Virtual machines increase in size from 4 way to 8 way SMP, from 64 GB to 256GB of RAM, enabling even the largest, most resource intensive applications to run on VMware. Hot add of virtual CPU, memory and network devices enables applications to scale seamlessly without disruption or downtime

31 Resources Performance community Performance web site VROOM! performance blog

32 Thank You Charu Chaubal Senior Architect, Technical Marketing 6 November, 2008

33 vapp New Model for Describing and Deploying Applications

34 Recommendations on Virtualizing Big Apps Candidate Selection: Which apps can I virtualize? Size VMs Correctly Setup Performance Monitoring Architecture Use the right SW: e.g. 64-bit Implementations Configure VMware ESX Appropriately Configure the Virtual I/O Infrastructure Accordingly Configure the Application Optimally

35 Can I Virtualize CPU Intensive Applications? Most CPU intensive applications have very low overhead VMware ESX 3.x compared to Native SPECcpu results covered by O.Agesen and K.Adams Paper Websphere results published jointly by IBM/VMware SPECjbb results from recent internal measurements

36 Can I Virtualize High Networking I/O Applications? Overall response time is lower when CPU utilization is less than 100% due to multi-core offload

37 Can I Virtualize High I/O Applications? 15k 30k 60k 100k 200,000 IOPs! - Good Exceeds Around Mailboxes for the I/O 120,000 intensive load of many VMs Microsoft databases Exchange mailboxes 1.3% Latency Overhead at Full load ESX layer is effectively transparent to high-i/o Apps

38 Mainframe class I/O in ESX 3.5 VMware ESX 16 cores (Intel Tigerton) 16 VMs (Windows 2003 Server) IO-intensive workload 8k block size 100% random Mixed read/write Storage Fabric EMC CLARiiON CX Disks 200,000 Mailboxes 77Terabytes of Data Enough I/O for 80 large databases Enough to hold the entire printed US Library of Congress

39 Backup: DB Configuration

40 Oracle File System Sync vs DIO

41 Oracle DIO vs. RAW

42 Comparison of DBcache vs Guest FScache Cache Rows/sec KernelCPU UserCPU FScache DBcache A 46GB table was populated, indexed, and then queried by 100 processes each requesting a range of data. A single row was retrieved at a time to simulate what would happen in an OLTP environment. The data was cached so that no IO occurred during any of the runs. Ref: Glenn Fawcett, PAE, Sun

43 Direct I/O Guest-OS Level Option for Bypassing the guest cache Uncached access avoids multiple copies of data in memory Avoid read/modify/write module file system block size Bypasses many file-system level locks Enabling Direct I/O for Oracle and MySQL on Linux # vi init.ora filesystemio_options= setall Check: # iostat 3 (Check for I/O size matching the DB block size ) # vi my.cnf innodb_flush_method to O_DIRECT Check: # iostat 3 (Check for I/O size matching the DB block size )

44 Direct I/O Guest-OS Level Option for Bypassing the guest cache Uncached access avoids multiple copies of data in memory Avoid read/modify/write module file system block size Bypasses many file-system level locks Enabling Direct I/O for MySQL on Linux # vi my.cnf innodb_flush_method to O_DIRECT Check: # iostat 3 (Check for I/O size matching the DB block size )

45 Asynchronous I/O An API for single-threaded process to launch multiple outstanding I/Os Multi-threaded programs could just just multiple threads Oracle databases uses this extensively See aio_read(), aio_write() etc... Enabling AIO on Linux # rpm -Uvh aio.rpm # vi init.ora filesystemio_options= setall Check: # ps aef grep dbwr # strace p <pid> io_submit() <- Check for io_submit in syscall trace

46 Use Large Pages Guest-OS Level Option to use Large MMU Pages Maps the large SGA region with fewer TLB entries Reduces MMU overheads ESX 3.5 Uniquely Supports Large Pages! Enabling Large Pages on Linux & SQLserver # vi /etc/sysctl.conf (add the following lines:) vm/nr_hugepages=2048 vm/hugetlb_shm_group=55 # cat /proc/vminfo grep Huge HugePages_Total: 1024 HugePages_Free: 940 Hugepagesize: 2048 kb To set a trace flag in SQL Server 2005 you need to follow the steps below: 1. Open SQL Server Configuration Manager 2. Select SQL Server 2005 Services 3. Right click the SQL Server service and select Properties 4. Select the Advanced tab 5. Edit the Startup Parameters property to set T834 The thing to note is that you need to make sure that you separate the Trace Flag from other startup parameters with a semicolon (;) and don't leave a space. ie.... mastlog.ldf;-t834

47 Linux Versions Some older Linux versions have a 1Khz timer to optimize desktop-style applications There is no reason to use such a high timer rate on server-class applications The timer rate on 4vcpu Linux guests is over 70,000 per second! Use RHEL5.1 Install kernel or later Put divider=10 on the end of the kernel line in grub.conf and reboot All the RHEL clones (CentOS, Oracle EL, etc.) work the same way

48 Backup: Futures