Certified Reference Design for VMware Cloud Providers

Transcription

1 VMware vclud Architecture Tlkit fr Service Prviders Certified Reference Design fr VMware Clud Prviders Versin 1.1 December 2017

2 2017 VMware, Inc. All rights reserved. This prduct is prtected by U.S. and internatinal cpyright and intellectual prperty laws. This prduct is cvered by ne r mre patents listed at VMware is a registered trademark r trademark f VMware, Inc. in the United States and/r ther jurisdictins. All ther marks and names mentined herein may be trademarks f their respective cmpanies. VMware, Inc Hillview Ave Pal Alt, CA VMware vclud Architecture Tlkit fr Service Prviders

3 Cntents Intrductin Audience Scpe... 5 Interperability Stack (Bill f Materials) Supprt Perfrmance Characteristics... 8 Sizing Guideline Tenant Resurce Capacity Clusters Management and Resurce Cmpnent Sizing Appendix A Test Envirnment and Benchmarking Methds Appendix B FAQ List f Tables Table 1. Bill f Materials... 7 Table 2. Scale Prfile A... 8 Table 3. Perfrmance and Thrughput... 9 Table 4. API Operatins Latency... 9 Table 5. OVF Uplad and Dwnlad Times... 9 Table 6. Management and Resurce Cmpnent Sizing Table 7. vclud Directr Operatins (Part 1) Table 8. vclud Directr Operatins (Part 2) List f Figures Figure 1. Management Cmpnent Deplyment Figure 2. Management Cluster Netwrking Figure 3. Resurce Cluster Setup Figure 4. Resurce Cluster Netwrking VMware vclud Architecture Tlkit fr Service Prviders

4 4 VMware vclud Architecture Tlkit fr Service Prviders

5 Intrductin The Certified Reference Design (CRD) fr VMware Clud Prviders is a pre-validated set f sftware cmpnents that simplify the deplyment f a VMware vclud Directr based multitenant clud in a predictable and efficient manner. The intent f the CRD initiative is t reduce the cmplexity f deplying, upgrading, and managing dependencies between the VMware cmpnents required fr a vclud Directr based service. While this initiative des nt yet invlve sftware autmatin fr sftware upgrades, it aims t clearly present what cmpnents are needed, which versins shuld be used, and what kind f scale and perfrmance VMware Clud Prviders can expect frm a CRD-cmpliant clud. The CRD gives VMware Clud Prviders clarity and predictability abut which versin f each sftware cmpnent f the stack is recmmended at a given time. The CRD als cmes with a predictable supprt time frame fr all underlying cmpnents, typically mnths frm the launch f the crrespnding CRD release. This reduces the expense and time invlved in determining what cmpnents t upgrade when and t which versin s that the entire sftware stack stays in supprt and incmpatible cmbinatins are avided. VMware Clud Prviders als benefit frm clear guidelines fr sizing hardware and sftware cmpnents t match their expected tenant lad. While we d nt cver every clud cnfiguratin and size, we strive t prvide a sizing recmmendatin fr a typical clud a clud size representative f a brad set f VMware Clud Prviders. Future versins f the CRD may address larger and less cmmn envirnment cnfiguratins as well as mre specialized use cases. It is nt the current CRD s intent t push vclud Directr t its abslute limits. Fr cnfiguratin maximums and limits, cnsult the Cnfiguratin Maximums dcument fr vclud Directr. Finally, we are dcumenting expected perfrmance as bserved by tenant users and VMware Clud Prvider administratrs interacting with the vclud Directr user interface and API. CRD-cmpliant slutins can be prperly sized by fllwing the sizing guidelines fr hardware and scale based n anticipated tenant demand. 1.1 Audience This dcument is intended fr VMware Clud Prvider architects and tech leads respnsible fr planning and executing the deplyment and upgrades f a VMware based clud envirnment. 1.2 Scpe This dcument addresses the fllwing aspects f Certified Reference Design: Interp stack Prvides a list f certified versins f all f the cmpnent sftware cmprising the sftware stack. Using the recmmended versin guarantees knwn supprt life f the stack as well as perfrmance characteristics. Sizing guidelines and sftware requirements Perfrmance characteristics f the slutin The Certified Reference Design-based slutin prvides knwn perfrmance and scale characteristics and cmes with recmmendatins and guidelines fr hardware and scale based n anticipated tenant demand. 5 VMware vclud Architecture Tlkit fr Service Prviders

6 Cnsult cmplimentary dcuments that are part f the VMware vclud Architecture Tlkit fr Service Prviders: Architecting a VMware vclud Directr Slutin fr VMware Clud Prviders Architecting Tenant Netwrking with VMware NSX in VMware vclud Directr Develping a Hyper-Cnverged Strage Strategy fr VMware vclud Directr with VMware vsan Cnfiguratin Maximums fr vclud Directr The VMware Prduct Interperability Matrix is the authritative resurce fr interperability between VMware sftware cmpnents. A CRD-cmpliant stack must cmply with all relevant security guidelines utlined in prduct-specific dcumentatin as well as security recmmendatins in the VMware vclud Architecture Tlkit fr Service Prviders dcument. 6 VMware vclud Architecture Tlkit fr Service Prviders

7 Interperability Stack (Bill f Materials) The fllwing table lists the pre-validated set f sftware cmpnents fr Clud Prviders at the time f the CRD 1.1 launch. While VMware Clud Prviders are free t chse and pick ther versins r different cmbinatins f VMware Clud Prvider Prgram sftware prducts, the specified stack guarantees a knwn predictable supprt time and specific perfrmance and scaling characteristics. Perfrmance and scaling infrmatin is prvided later this dcument. Prducts marked Cre are required t fficially achieve CRD cmpliance. Table 1. Bill f Materials Cmpnent Versin and Build Cre/ Optinal Ntes VMware vcenter Server 6.5 Cre See Table 6 fr patch level tested. VMware ESXi 6.5 Cre See Table 6 fr patch level tested. VMware NSX Cre VMware vsan Cre See Nte 2 vclud Directr 8.20 Cre 1. Database: SQL Server We recmmend the latest available patch f At the time f this dcument VMware vrealize Lg Insight VMware vclud Usage Meter VMware vrealize Operatins 4.5 Optinal 3.6 Cre We recmmend the latest available patch f r 6.6 Optinal We recmmend the latest available 6.6 patch release. Nte 1. These are the recmmended set f prducts, but this is nt a full interperability matrix. Fr example, vclud Directr 8.20 is supprted with multiple versins f NSX but we are recmmending a specific NSX versin here. Cnsult the VMware Prduct Interperability Matrix fr full vclud Directr interperability infrmatin. 2. vsan based strage must be deplyed in at least ne r mre clusters (either management r capacity). 7 VMware vclud Architecture Tlkit fr Service Prviders

8 2.1 Supprt Each cmpnent f the CRD stack is supprted accrding t its supprt lifecycle. A clud deplyment cmpliant with the Bill f Materials will stay in supprt fr at least 12 mnths after the CRD release date. The fllwing prfile represents a cmmn envirnment similar t the envirnments f apprximately 60 percent f all VMware Clud Prviders. While vclud Directr is capable f a larger scale, the fllwing prfile is what we validated and benchmarked in the current CRD. Table 2. Scale Prfile A Parameter Value Number f tenants (Organizatins in vclud Directr) 200 Number f tenant wrklads (mix f pwered-n and ff VMs) 10,000 Number f pwered-n tenant VMs 7000 Number f data centers 1 Number f vclud Directr cells 4 Number f vcenter Server instances managed by vclud Directr Number f hsts and clusters Max netwrk latency frm vclud Directr t VMware vcenter Server, VMware NSX Manager, and ESXi hsts 1 vcenter Server fr management cluster 3 vcenter Server instances fr resurce capacity 3 resurce capacity clusters 1 management cluster Netwrk RTT latency up t 100 ms 2.2 Perfrmance Characteristics Envirnment Setup The multitenant clud envirnment is set up based n Scale Prfile A. Latency frm vclud Directr cells t vcenter Server and NSX Manager cmpnents was set t 40 ms and 100 ms Perfrmance and Thrughput We measured the test thrughput as the number f peratins executed ver 30 minutes. The test was run with different test cncurrency (32, 64, and 128) and netwrk latency (0.3 ms, 40 ms, and 100 ms). We used a representative randm sample f peratins frm the List f Operatins during this test. 8 VMware vclud Architecture Tlkit fr Service Prviders

9 Table 3. Perfrmance and Thrughput Cncurrency (Number f cncurrent users) Thrughput at RTT = 0.3 ms (Successfully cmpleted peratins per minute) Thrughput at RTT = 40 ms Thrughput at RTT = 100 ms API Latency The fllwing table shws average user bserved latency (in secnds) fr a selectin f API peratins at RTT = 0.3 ms. See the List f Operatins fr the full list f peratins invked during this test. Table 4. API Operatins Latency Operatin Cncurrency Instantiate 150 MB vapp frm template 3 sec 4 sec 5 sec Create edge gateway 48 sec 65 sec 110 sec Create independent disk 8 sec 12 sec 22 sec Increasing netwrk RTT frm 0.3 ms t 100 ms affects these numbers with the size f the effect varying significantly depending n the peratin. The highest impact was bserved with deplying a vapp where an RTT f 100 ms caused the latency t increase by a factr f Uplad/Dwnlad Perfrmance The fllwing table shws OVF uplad and dwnlad times bserved in ur test envirnment under different netwrk latencies. Table 5. OVF Uplad and Dwnlad Times RTT = 0.3 ms RTT = 40 ms RTT= 100 ms OVF uplad time in secnds (4 GB) OVF dwnlad time (4 GB) VMware vclud Architecture Tlkit fr Service Prviders

10 Sizing Guideline This sectin includes guidelines t help yu size hardware and sftware cmpnents t match yur anticipated tenant lad. These guidelines apply t envirnments whse size is rughly similar t Scale Prfile A (that is, within a factr f 2 f prfile A). Sizing f the envirnment was dne in tw steps: 1. We determined the number f vcenter Server instances, ESXi hsts, and vclud Directr cells t use t handle at least 10,000 VMs with 7000 pwered-n VMs. 2. Fr each cmpnent in the stack, specific cnfiguratins f resurces (memry, CPU, disk) were made. 3.1 Tenant Resurce Capacity Clusters Number f vcenter Server instances The vcenter Server sizing guide suggests using a medium prfile f vcenter Server t supprt 4000 VMs. Number f vcenter Server instances = number f VMs/4000 = 10,000/4000 = 3 (runded) Number f ESXi hsts ESXi hst cunt was determined based n the number f pwered-n VMs, using the frmula belw. Please nte that the frmula prvides a rugh estimate and the actual number f hsts required depends n many parameters and the type f wrklads. Number f hsts = (number f pwered n VMs * Avg # f vcpus per VM) / (sckets*cres*hyper threading*vcpu-t-pcpu rati) = (7000*1)/(2*8*2*16) = 14 We used 15 hsts, with a set f 5 managed by each vcenter Server. Number f vclud Directr cells As with the vclud Directr design guide, the number f vclud Directr cells fr this setup is arrived at using fllwing frmula: Number f vclud Directr cells = (Number f VMs/4000) + 1 = (10,000/4000) + 1 = = Management and Resurce Cmpnent Sizing The fllwing table summarizes sizing chices made fr varius management cmpnents. 10 VMware vclud Architecture Tlkit fr Service Prviders

11 Table 6. Management and Resurce Cmpnent Sizing Cmpnent Versin Size Resurces Ntes Management vcenter Server (vcenter virtual appliance with embedded DB and VMware Platfrm Services Cntrller ) Resurce vcenter (vcenter virtual appliance with embedded DB and Platfrm Services Cntrller) ESXi vsan (deplyed in management cluster) 6.5 U1 Tiny RAM: 10 GB CPU: 2 Strage: 400 GB 6.5 U1 Medium RAM: 24 GB CPU: 8 Strage: 400 GB 6.5 U NSX RAM: 16 GB CPU: 4 Strage: 60 GB vclud Directr RAM: 16GB CPU: 4 Strage: 300GB + 500GB NFS vclud Directr DB MSSQL 2014 Enterprise RAM: 32GB CPU: 16 Strage: 1 TB vrealize Lg Insight deplyment 4.5 Medium RAM: 16 GB CPU: 8 Strage: 1 TB Fr a vrealize Lg Insight sizing calculatr, see ginsight/calculatr Cntent Pack fr NSX 3.6 Cntent Pack fr vsan 2 11 VMware vclud Architecture Tlkit fr Service Prviders

12 Cmpnent Versin Size Resurces Ntes Cntent Pack fr vclud Directr Cntent Pack fr vrealize Operatins Cntent Pack fr VMware vsphere vclud Usage Meter Standard RAM: 4 GB CPU: 2 Strage: 100 GB vrealize Operatins Large RAM: 48 GB CPU: 16 Strage: 400 GB The deplyment requirements fr vclud Usage Meter 3.6 are published here: prt/vclud-usagemeter/dc/vclud-usagemeter-36-interp.html vrealize Operatins 6.5 sizing guidelines are published in a VMware Knwledge Base article: Management Pack fr NSX fr vsphere Management Pack fr vsphere Management Pack fr vrealize Lg Insight Management Pack fr vclud Directr Management Pack fr Strage Devices Management Pack fr vsan VMware vclud Architecture Tlkit fr Service Prviders

13 Appendix A Test Envirnment and Benchmarking Methds Test Envirnment The test envirnment is bradly divided int three main setups: Management cluster Resurce cluster (30% f wrklads n vsan, 70% n iscsi strage) Test driver Management Cluster This is where all the management cmpnents were deplyed. Management cmpnents 1 x Management vcenter Server (Tiny) 4 x vclud Directr cells 1 x MSSQL 2014 Enterprise 3 x Resurce vcenter Server (Medium) 3 x NSX Manager 1 x Management NSX vclud Directr edge 1 x vrealize Lg Insight (Medium) 1 x vrealize Operatins (Large) 1 x vclud Usage Meter (Standard) Management Cluster Resurces 5 x UCS C240 M5 with 24x 2.5" slt (2x NVMe supprted) and 6x PCIe 16 CPUs 2.6 GHz 2 x 14 cre cpu 5120 (Skylake) 14c 2.2 GHz 2 x 10 Gb Intel X550 BaseT (n 1Gb) 192 GB (12x16GB DDR4) memry Cisc 12 Gbps SAS HBA 1 x 400 GB SAS SSD (in VSAN HCL) 1 x 1900 GB SATA SSD 1 x 240 GB M.2 t install and bt ESXi frm 10 TB iscsi shared LUN 13 VMware vclud Architecture Tlkit fr Service Prviders

14 Figure 1. Management Cmpnent Deplyment 14 VMware vclud Architecture Tlkit fr Service Prviders

15 Figure 2. Management Cluster Netwrking Resurce Cluster This is where Tenant Organizatins and wrklad VMs were created. Resurce cluster resurces 15 x UCS C240 M5 with 24x 2.5" slt (2x NVMe supprted) and 6x PCIe (5 hst fr each f the vcenter server instances) 2 x 14cre cpu 5120 (Skylake) 14c 2.2GHz 2 x 10Gb Intel X550 BaseT (n 1Gb) 192GB (12 x 16 GB DDR4) memry 15 VMware vclud Architecture Tlkit fr Service Prviders

16 Cisc 12 Gbps SAS HBA 1 x 400 GB SAS SSD (in VSAN HCL) 1x 1900 GB SATA SSD 1 x 240 GB M.2 t install and bt ESXi frm 10 TB iscsi shared LUN Figure 3. Resurce Cluster Setup 16 VMware vclud Architecture Tlkit fr Service Prviders

17 Figure 4. Resurce Cluster Netwrking Test Driver The test suite is executed frm this envirnment. 4 CPU, 8 GB memry, Cent OS 7.3 Benchmarking Methds The testing prcess fcused primarily n verifying and measuring envirnment behavir fr the fllwing: Scale Verify whether the envirnment meets the Scale Prfile A requirement f 10,000 VMs and 7000 pwered-n VMs. 17 VMware vclud Architecture Tlkit fr Service Prviders

18 Perfrmance Measure peratin latency and thrughput when the envirnment is running at scale (10,000 VMs with 7000 pwered-n VMs). Uptime Verify that the envirnment can perate at scale with reasnable perfrmance fr a lng duratin. The remainder f this sectin details the exact methds used fr test executin and measurement. Scale Test Scale was carried ut with a mix f manual peratins and JMeter test tl-based script peratins t using the fllwing steps: 1. Create 200 Tenant Organizatins in vclud Directr. 2. Create 10,000 VMs acrss these 200 Tenant Organizatins. 90% f the VMs were Dummy Small Tiny VMs with 4 MB disk, 4 MB memry 10% actual VMs 7% Linux VMs 3% Windws VMs 3. Pwer n 7000 VMs with similar distributin (90%:10%) f dummy t actual VMs. 4. Sme simple vclud Directr peratins were carried ut t verify that system behaves nrmally at this scale. Perfrmance Test Perfrmance tests were dne by executing a well-knwn distributin f vclud Directr peratins with the help f internal test tl. (A cmplete peratin list is included in the last sectin f this appendix.) The fllwing were the key steps in executin and measurement f the peratins: 1. Scaled up the envirnment as utlined in the previus sectin. 2. After the envirnment was at scale, executed cntinuus stream f peratins fr 30 minutes with fllwing distributin: 35-40% vapp peratins such as instantiate, deply, edit, clne, and delete. 25% strage-centric peratins such as create, attach, detach, and delete disk. 15% netwrking-related peratins, such as create/delete gateway, ruted netwrks and firewall cnfiguratins. 5% create/delete Orgs, users, catalg, and virtual data centers. 3. Operatins were executed using vclud Directr lcal users f different rles (vapp Authr, Org Admin, System Admin) with 10 percent admin rles and 90 percent user peratins. 5. Given that mst f the peratins are asynchrnus, the test tl mnitrs the task returned by vclud Directr t get cmpletin status and executin time details. 6. Steps 2 t 4 were repeated with 32, 64, and 128 cncurrent users t ascertain the ability f the system t deal with cncurrent peratin invcatin. 7. Step 5 was repeated fr fllwing latency (between vclud Directr and vcenter Server) values (achieved by artificial latency injectin with a tl): 0.3 ms (default) 40 ms 100 ms 18 VMware vclud Architecture Tlkit fr Service Prviders

19 Uptime Tests Uptime tests invlved executing peratins carried ut during perfrmance tests, with fllwing changes t exactin duratin and cncurrency: 1. Tests ran cntinuusly fr 7 days cncurrent users each invked an peratin at every 20 secnds. 3. N artificial latency injectin was dne. List f Operatins Fr perfrmance benchmarking, API test clients executed a predetermined distributin acrss different types f vclud Directr peratins as described in the fllwing tables. Table 7. vclud Directr Operatins (Part 1) vapp Operatins Netwrk Operatins Management Operatins Instantiate vapp Deply fenced vapp Create rg Deply (pwer n) Undeply fenced vapp Create user Edit vapp Create islated netwrk Create Org VDC Cmpse vapp Delete islated netwrk Create direct VDC netwrk Clne vapp Create gateway Create catalg Pwer ff vapp Create ruted Org netwrk Delete catalg Delete vapp Instantiate vapp in that netwrk Delete VDC netwrk Deply vapp Undeply vapp Delete vapp Delete Org VDC Delete user Delete Org Delete ruted Org netwrk Delete gateway Table 8. vclud Directr Operatins (Part 2) NSX Management Operatins Datastre Operatins OVF Operatins Cnvert edge t Advanced edge Create disk OVF uplad Edge ruting services Instantiate vapp OVF dwnlad Edge firewall services Attach disk t vapp 19 VMware vclud Architecture Tlkit fr Service Prviders

20 Edge NAT services Distributed firewall services Lad balancer services Detach disk frm vapp Delete disk Delete vapp 20 VMware vclud Architecture Tlkit fr Service Prviders

21 Appendix B FAQ Hw frequently will the CRD be updated? We expect t release an updated CRD with every majr vclud Directr release. Hw is this dcument related t the VMware interperability matrix? The recmmended CRD stack is a subset f the full interperability matrix and reflects the exact cmpnents we validated and benchmarked in this exercise Hw is the CRD related t VCF? VCF is nt cnsidered as part f this CRD Is CRD suitable fr greenfield envirnments r brwnfield envirnments? Any envirnment can be made CRD cmpliant by simply upgrading all f its cmpnents t versins listed in the CRD Bill f Materials. There is n ther qualificatin. Hw can we prvide input/recmmendatins fr future versins f this dc? Cntact vclud Directr team at vcd-feedback@vmware.cm r reach ut t yur VMware accunt team and pass yur feedback thrugh them. Is there CRD-specific supprt? N. Each cmpnent f the CRD is supprted thrugh its existing supprt arrangement. 21 VMware vclud Architecture Tlkit fr Service Prviders