VMware vcloud Architecture Toolkit Public VMware vcloud Implementation Example

Transcription

1 VMware vcloud Architecture Toolkit Version October 2011

2 This product is protected by U.S. and international copyright and intellectual property laws. This product is covered by one or more patents listed at VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies. VMware, Inc 3401 Hillview Ave Palo Alto, CA Page 2 of 59

3 Contents 1. Overview Business Requirements Use Cases Document Purpose and Assumptions vcloud Components Abstractions and VMware vcloud Constructs vsphere Design Architecture Overview Site Considerations Management Cluster Design Resource Group Design vcloud Design Provider Constructs Provider Virtual Datacenters External Networks Network Pools Users/Roles vcloud Design Consumer Constructs Organizations Organization Virtual Datacenters Organization Networks Catalogs Users/Roles vcloud Security Host Security Network Security vcenter Security VMware vcloud Director Security Additional Security Considerations Page 3 of 59

4 6. vcloud Management vsphere Host Setup Standardization vcloud Center of Excellence vcloud Logging vcloud Monitoring Extending vcloud Hybrid vcloud vcloud Connector vcloud API VMware vcenter Orchestrator vcloud Metering Cost Models Reporting Metering Internet Traffic Aggregator Reporting List of Figures Figure 1. VMware vcloud Director Abstraction Layer Figure 2. vsphere Logical Architecture Overview Figure 3. vcloud Physical Design Overview Figure 4. vsphere Logical Network Design Management Cluster Figure 5. vsphere Logical Network Design Figure 6. vcloud Logging Organization Figure 7. vcloud Connector Figure 8. vcloud API Logical Representation Figure 9. vcloud Orchestration Figure 10. One Time Router Cost Example (vcenter Chargeback UI) Page 4 of 59

5 List of Tables Table 1. Initial vcloud Capacity... 8 Table 2. Document Sections... 9 Table 3. vcloud Components Table 4. vcloud Components Table 5. vcenter Servers Table 6. Management Virtual Machines Table 7. Management Component Resiliency Table 8. vsphere Clusters Management Cluster Table 9: vsphere Management Cluster DRS Rules Table 10. Host Logical Design Specifications Management Cluster Table 11. Virtual Switch Configuration Management Cluster Table 12. Virtual Switch Configuration Settings Management Cluster Table 13. Shared Storage Logical Design Specifications Management Cluster Table 14. Resource Group Clusters Table 15. vsphere Cluster Configuration vcloud Resources Table 16. Host Logical Design Specifications Table 17. Virtual Switch Configuration vcloud Resources Table 18. dvresswitch01 Teaming and Failover Policies Table 19. dvresswitch01 Security Policies Table 20. dvresswitch01 General Policies Table 21. Storage Logical Design Specifications vcloud Compute Cluster Table 22. vsphere Clusters vcloud Compute Datastores Table 23. Datastore Size Estimation Factors res-pod1 Cluster Table 24. Provider Virtual Datacenter Specifications Table 25. Virtual Machine Sizing and Distribution Table 26. Provider External Network Specifications Table 27. System Administrator User and Group Table 28. NewCo Fixed-cost Cost Model Table 29. Virtual Switch Security Settings Table 30: vcloud Director Log Locations Table 31. VMware vcloud Director Monitoring Items Table 32.vCenter Orchestrator Monitored MBeans Table 33. vcloud Connector Components Page 5 of 59

6 Table 34. Allocation Units for vcloud Hierarchies Based on Allocation Model Table 35. Management Cluster Inventory Table 36. vcloud Resources Inventory Page 6 of 59

7 1. Overview This public VMware vcloud implementation example uses a fictitious corporation, New Company (NewCo), as a vehicle to provide a detailed implementation example for a public VMware vcloud. It is intended to provide architects and engineers who are interested in implementing a public vcloud with a reference implementation that conforms to VMware best practices, and describes the logical and physical design and implementation of the components of a VMware vcloud. Each document section elaborates on different aspects and key design decisions of this vcloud solution. This implementation example provides a baseline that is extensible for future usage patterns. 1.1 Business Requirements The NewCo vcloud implementation has the following characteristics and provides: Basic service offering as an instance-based Pay-As-You-Go resource consumption model, with each provisioned virtual machine charged separately and separate billing records produced for each virtual machine. Secure multitenancy controlling workload access as well as network isolation, permitting multiple organizations within an implementation to share public vcloud resources. A self-service portal where Infrastructure as a Service (IaaS) can be consumed from a catalog of predefined applications and services. Use of a public vcloud to rapidly provision complex multitier applications or entire environments to respond to the dynamic business requirements of an organization. Conform to vcloud Center of Excellence principles described in Operating a VMware vcloud. See the Public VMware vcloud Service Definition for additional details. Page 7 of 59

8 1.1.1 vcloud Capacity and Considerations The NewCo vcloud design is configured to initially support 600 organizations and 1500 virtual machines with the ability to scale to 5000 virtual machines under one resource group. Additional details regarding the initial targets are provided in Table 1. Table 1. Initial vcloud Capacity Initial Target 600 customers/organizations: o o o 30 virtual machines maximum. 1 public-routed network. 1 internal network virtual machines: o o o o o o 5% 16GB/4 vcpu. 10% 8GB/2 vcpu. 25% 4GB/1 vcpu. 20% 2GB/1 vcpu. 40% 1GB/1 vcpu. Average 40GB storage. Parallel Operations: o o SLA: 30 virtual machine provision/clone operations. 10 OVF uploads. o Infrastructure uptime 99.9%. o Portal/API uptime 99.9%. o Virtual machine provisioning < 5 minutes. Use of virtual hardware version 8 allows for larger virtual machines than are defined in this section. NewCo determined that the use cases for the vcloud do not warrant larger virtual machines at this time. 1.2 Use Cases The target use case for this vcloud environment includes, but is not limited to, transient workloads normally observed with: Software development. Quality Assurance and software testing. Page 8 of 59

9 1.3 Document Purpose and Assumptions VMware vcloud Architecture Toolkit This document is intended to serve as a reference for service providers and assumes that they have familiarity with VMware products, including VMware vsphere, VMware vcenter, VMware vcloud Director, VMware vshield, VMware vfabric, and VMware vcenter Chargeback. It covers both logical and physical design considerations for all VMware vcloud infrastructure components, with each section elaborating on different aspects and key design decisions of a public vcloud implementation. Public vcloud architecture topics are covered in the document sections listed in Table 2. Table 2. Document Sections Section Description 1. Overview Overview, business requirements, capacity considerations, and inventory of vcloud components. 2. vsphere Design Management cluster Management components that support the operation of the resource groups. Resource group Compute, storage, and network resources available for public consumption. 3. vcloud Design Provider VMware vcloud Director provider objects and Constructs configuration. Relationship of vcloud Director provider objects to vsphere objects. 4. vcloud Design Consumer VMware vcloud Director organization objects and Constructs configuration. Relationship of consumer objects to underlying provider objects. 5. vcloud Security Considerations as they apply to all management and resource components. 6. vcloud Management Considerations that apply to vcloud Director management components. 7. Extending vcloud Available options for increasing the functionality, automation, and orchestration of the vcloud. 8. vcloud Metering vcenter Chargeback design and configuration as well as vram metering of virtual machines. This document is not intended as a substitute for VMware product documentation. See the installation and administration guides as well as published best practices for the appropriate product for further information. Page 9 of 59

10 1.4 vcloud Components Table 3 lists the components that comprise the vcloud. Table 3. vcloud Components VMware vcloud Architecture Toolkit vcloud Component VMware vcloud Director VMware vsphere Description Abstracts and provides secure resource and workload isolation of underlying vsphere resources. Includes: VMware vcloud Director Server (two or more instances, each installed on a Linux virtual machine and referred to as a cell). VMware vcloud Director Database (one instance per clustered set of VMware vcloud Director cells). vsphere compute, network and storage resources. Foundation of underlying vcloud resources. Includes: VMware ESXi hosts (three or more instances for management cluster and three or more instances for resource group, also referred to as Compute Cell). VMware vcenter Server (one instance managing a management cluster of hosts, and one or more instances managing one or more clusters of hosts reserved for vcloud consumption). vcenter Server Database (one instance per vcenter Server). VMware vshield Provides network security services including Layer 2 isolation, NAT, firewall, DHCP, and VPN. Includes: vshield Manager (one instance per vcenter Server in the resource groups). vshield Edge (deployed automatically by VMware vcloud Director as virtual appliances on hosts within resource groups). VMware vcenter Chargeback Provides resource metering and cost models. Includes: vcenter Chargeback Server (one or more instances). vcenter Chargeback database (one or more instances). vcloud Director data collector (one or more instances). vshield Manager data collector (one per resource group). See Architecting a VMware vcloud for additional information about the vcloud components and options for planning, deployment, and configuration. Page 10 of 59

11 1.5 Abstractions and VMware vcloud Constructs Key features of the vcloud architecture are resource pooling, abstraction, and isolation. VMware vcloud Director further abstracts the virtualized resources presented by vsphere by providing the following logical constructs that map to vsphere logical resources: Organization A logical object that provides a security and policy boundary. Organizations are the main method of establishing multitenancy and typically represent a business unit, project, or customer in a private vcloud environment. Virtual datacenter Deployment environments in which virtual machines run. Organization virtual datacenter An organization s allocated portion of provider virtual datacenter resources, including CPU, RAM, and storage. Provider virtual datacenter vsphere resource groupings of compute, storage, and network that power organization virtual datacenters. Figure 1. VMware vcloud Director Abstraction Layer Page 11 of 59

12 2. vsphere Design 2.1 Architecture Overview vsphere resources are organized and separated into: A management cluster containing all core components and services needed to run the vcloud. One resource group that represents dedicated resources for vcloud consumption. Each compute cluster of ESXi hosts is managed by a vcenter Server, and is under the control of VMware vcloud Director. Multiple compute clusters can be managed by the same VMware vcloud Director instance as additional capacity or service offerings are added. Reasons for organizing and separating vsphere resources along these lines are: Facilitates quicker troubleshooting and problem resolution. Management components are strictly contained in a relatively small and manageable management cluster. Provides resource isolation between workloads running in the vcloud and the actual systems used to manage the vcloud. Separates the management components from the resources they are managing. Resources allocated for vcloud use have little reserved overhead. For example, vcloud resources would not host vcenter virtual machines. vcloud resources can be consistently and transparently managed, carved up, and scaled horizontally. Page 12 of 59

13 The components that comprise the vcloud are listed in Table 4. Table 4. vcloud Components vcloud Components vcenter Server version 5.0. vsphere ESXi version 5.0 hosts. vcenter Chargeback Server version vcenter Chargeback Collectors version vshield Manager version 5.0. Microsoft SQL Server 2008 R2 Standard 64-bit. vcenter Server Databases. vcloud Director Database. vcenter Update Manager Database. vcenter Chargeback Database. vcloud Director 1.5. vfabric Hyperic 4.5. Active Directory Syslog-ng. vcenter Update Manager 5.0. vcenter Configuration Manager. vcenter Orchestrator 5.0. These components map to the management cluster as noted in Section 2.3, Management Cluster Design. For a complete bill of materials, see Appendix A: Bill of Materials. Page 13 of 59

14 The high-level logical architecture is illustrated in Figure 2. Figure 2. vsphere Logical Architecture Overview Page 14 of 59

15 Figure 3 shows the physical design that corresponds to the logical architecture. Figure 3. vcloud Physical Design Overview 2.2 Site Considerations There is enough floor space, power, and cooling capacity for the management group and the resource groups to both reside within a single physical datacenter, and scale to support 5,000 virtual machines, as defined in the requirements. Table 5. vcenter Servers vcenter Datacenter Purpose mgmt-vc1 mgmt-newco Provides compute resource clusters for vcloud management components. res1-newco-vc1 Res1-newco Provides compute resource clusters for tenant workloads. Page 15 of 59

16 2.3 Management Cluster Design VMware vcloud Architecture Toolkit The vsphere management cluster design encompasses the ESXi hosts contained in the management group. The scope is limited to only the infrastructure components used to operate vcloud resource group workloads. The virtual machines that run in the management group are listed in Table 6. Table 6. Management Virtual Machines Virtual Machine mgmt-vcd1 mgmt-vcd2 res1-newco-vc1 mgmt-vc1 res-newco-vsm1 mgmt-newco-vsm1 mgmt-lb1 mgmt-lb2 mgmt-vsm1 mgmt-vco1 mgmt-vco2 mgmt-ad1 mgmt-ad2 mgmt-dns1 mgmt-dns2 mgmt-ipam mgmt-hyperic mgmt-syslog1 mgmt-syslog2 Purpose vcloud Director cell. vcloud Director cell. vcenter Server dedicated to vcloud Director and managing vcloud resources. vcenter Server dedicated to administering the management group. vshield Manager server paired with res1-newco-vc1. vshield Manager server paired with mgmt-vc1. Virtual load balancer appliance. Virtual load balancer appliance. vshield Manager server paired with the management vcenter Server. vcenter Orchestrator Server. vcenter Orchestrator Server. Active Directory 2008 Server. Active Directory 2008 Server. DNS, SMTP, and NTP node. DNS, SMTP, and NTP node. IPAM server for management cluster. VMware vfabric Hyperic Server. Syslog appliance for management and resource cluster. Syslog appliance for management and resource cluster. Page 16 of 59

17 mgmt-vma mgmt-cb1 mgmt-um1 vsphere Management Assistant. vcenter Chargeback Server. vcenter Update Manager. mgmt-mssql1 Microsoft MSSQL Server 2008 R2. mgmt-mssql2 Microsoft MSSQL Server 2008 R2. res-mssql1 res-mssql2 Microsoft MSSQL Server 2008 R2 attached to resource vcenter Servers. Microsoft MSSQL Server 2008 R2 attached to resource vcenter Servers Management Component Resiliency Considerations The following management components rely on HA, FT, and third-party clustering for redundancy. Table 7. Management Component Resiliency Management Component HA Enabled Virtual Machine Monitoring FT vcenter Heartbeat Clustered vcenter Server Yes Yes No Yes No VMware vcloud Director vcenter Chargeback Server Yes Yes No NA No Yes Yes No NA No vshield Manager Yes Yes Yes NA No SQL Server 2008 R2 Standard (x64) VMware vcenter Orchestrator Yes Yes No NA Yes Yes Yes No NA Yes Active Directory Yes Yes No NA No VMware Data Recovery Yes Yes No NA No Page 17 of 59

18 2.3.2 vcloud Cell Load Balancing VMware vcloud Director supports multiple cells in a management cluster. All of the cells act as front-ends for the same database, but adding cells and providing front-end load balancing can: Increase the number of concurrent operations. Increase the number of simultaneous consoles accessible via the console proxy service. Increase the number of vcenter Server operations carried out (assuming that the number of vcenter servers scales as well as the number of VCD cells). Allow for upgrade and maintenance of the vcloud Director cells without having to disable the public vcloud service vsphere Clusters The management cluster is comprised of the following vsphere HA/DRS details. Table 8. vsphere Clusters Management Cluster Attribute Cluster Name Specification Mgmt-NewCo1 Number of ESXi Hosts 3 VMware DRS Configuration Fully automated VMware DRS Migration Threshold 3 (of 5) VMware HA Enable Host Monitoring VMware HA Admission Control Policy Yes Enabled (percentage based) VMware HA Percentage 33% CPU 33% Memory N+1 for 3 host cluster VMware HA Admission Control Response VMware HA Default VM Restart Priority VMware HA Host Isolation Response VMware HA Enable VM Monitoring VMware HA VM Monitoring Sensitivity Disallow virtual machine power on operations that violate availability constraints. N/A Leave virtual machine Powered On Yes Medium Page 18 of 59

19 VMware DRS dynamically balances computing capacity across a collection of hardware resources aggregated into logical resource pools, continuously monitoring utilization across resource pools and intelligently allocating available resources among the virtual machines based on predefined rules that reflect business or application needs. Use of DRS rules allows for the separation of virtual machines so that there is limited impact in the event of a host failure or isolation event. DRS rules detailed here have been configured such that operation of management resources are minimally impacted by a host or chassis failure. Table 9: vsphere Management Cluster DRS Rules Rule Type Description Mgmt-LDAP Separate virtual machines Keep Active Directory 2008 servers separated. Resource vcenter Separate virtual machines Keep resource group vcenter Servers separated. Mgmt-DNS Separate virtual machines Keep DNS, SMTP, and NTP services from running on same host. Mgmt-SYSLOG Separate virtual machines Keep Syslog appliances separated. Mgmt-VCD Separate virtual machines Keep vcloud Director servers separated. Mgmt-Chargeback Keep virtual machines together Keep vcenter Chargeback Server and vcenter Chargeback database together. Resource-vCenterDB Keep virtual machines together Keep vcenter Server and vcenter Server database paired together. Mgmt-LB Separate virtual machines Separate virtual load balancing appliances. Mgmt-VCO Separate virtual machine Separate vcenter Orchestrator servers. Resource-VSM Keep virtual machines together Keep associated resource group vcenter Server and the paired vshield Manager appliance together. Page 19 of 59

20 2.3.4 Host Logical Design Each ESXi host in the management cluster has the following specifications. Table 10. Host Logical Design Specifications Management Cluster Attribute Specification Host Type and Version Blade, VMware ESXi 5 Processors Storage 2 x Intel Xeon x GHz (6 core, Westmere) NFS Storage Networking 802.1q Trunk Port Connectivity participating in the following VLANs: VLAN 600 management network (console) VLAN 610 VMKernel (vmotion non-routable) VLAN 620 VMKernel (iscsi) VLAN 630 VMKernel (NFS) VLAN 640 Fault Tolerance (non-routable) VLAN 650 management network (console) Memory 96GB Network Logical Design The network logical design defines how the vsphere virtual networking is configured. Following best practices, the network architecture must meet the following requirements: Separate networks for vsphere management, virtual machine connectivity, VMware vsphere vmotion traffic, IP storage, and VMware Fault Tolerance. Redundant dvswitch ports with at least two active physical NIC adapters each. Redundancy across different physical adapters to protect against NIC or PCI slot failure. Redundancy at the physical switch level. A mandatory dvswitch in the management cluster. Page 20 of 59

21 Table 11. Virtual Switch Configuration Management Cluster Switch Name Switch Type Function # of Physical NIC Ports dvmgmtswitch0 Distributed Management console VMKernel vmotion VMKernel NFS Fault Tolerance Virtual machine 2 x 10 GigE (teamed for failover) Figure 4 shows the virtual network infrastructure design for the vsphere management cluster. Figure 4. vsphere Logical Network Design Management Cluster Page 21 of 59

22 Table 12. Virtual Switch Configuration Settings Management Cluster Parameter Port Group Setting Load Balancing All Route based on originating port ID Failover Detection All Link status Notify Switches All Enabled Failback All No Failover Order Management vmotion Production virtual machines NFS Fault Tolerance Vmnic0 Active, Vmnic1 Standby Vmnic1 Active, Vmnic0 Standby Vmnic0 Active, Vmnic1 Standby Vmnic1 Active, Vmnic0 Standby Vmnic1 Active, Vmnic0 Standby Shared Storage Logical Design This shared storage logical design defines how vsphere storage is configured. Different volumes from the same storage system are used for both the management cluster and the vcloud resources. Following best practices, the shared storage architecture must meet the following requirements: Storage paths will be redundant at the host (connector), switch, and storage array levels. All hosts in the management cluster have access to the same volumes, but are isolated from datastores in the resource cluster. Table 13. Shared Storage Logical Design Specifications Management Cluster Attribute Number of Initial Volumes Volume Size Specification 5 dedicated 1TB Datastores per volume 1 Virtual Machines per Volume No greater than 15 while distributing redundant virtual machines. Page 22 of 59

23 2.3.7 vcloud Director Transfer Storage To provide temporary storage for uploads and downloads, a 500GB network share must be presented to all cells in the cluster. The transfer server storage volume must have write permissions for root. Each host must mount this storage at $VCLOUD_HOME/data/transfer (by default this is /opt/vmware/cloud-director/data/transfer). 2.4 Resource Group Design The resource group design represents the ESXi host clusters and infrastructure used to run the vapps that are provisioned and managed by vcloud Director. In this section the scope is further limited to only the infrastructure dedicated to the vcloud workloads Virtual Datacenters Each resource cluster is linked to a single virtual datacenter associated with a single vcenter instance. Table 14. Resource Group Clusters Datacenter res-newco1 Purpose Provides computer resource clusters for one of the NewCo resource groups vsphere Clusters All vsphere clusters will be configured similarly with the following specifications. Table 15. vsphere Cluster Configuration vcloud Resources Attribute Specification Number of ESXi Hosts 16 VMware DRS Configuration Fully automated. VMware DRS Migration Threshold 3 (of 5) VMware HA Enable Host Monitoring VMware HA Admission Control Policy Yes Enabled (percentage based). VMware HA Percentage 7% CPU 7% memory (N+1 for 16 host cluster) VMware HA Admission Control Response VMware HA Default VM Restart Priority VMware HA Host Isolation Response Prevent virtual machines from being powered on if they violate availability constraints. N/A Leave virtual machine Powered On. Page 23 of 59

24 2.4.3 Host Logical Design Each ESXi host in the vcloud resources will have the following specifications. Table 16. Host Logical Design Specifications Attribute Specification Host Type and Version Blade, VMware ESXi 5 Processors Storage Networking Memory 2 x Intel Xeon x GHz (6 core, Westmere) NFS 802.1q trunk port connectivity participating in the following VLANs: VLAN 700 management network (Console) VLAN 710 VMKernel (vmotion non-routable) VLAN 720 VMKernel (iscsi) VLAN 730 VMKernel (NFS) VLAN 750 management network (console) 96GB Network Logical Design The network logical design defines how vsphere virtual networking is configured. Following best practices, the network architecture must meet the following requirements: Separate networks for virtual machine connectivity and specific VMKernel port groups. Maintain isolation of organizations from other VLANs across physical and virtual networking infrastructure. dvswitch with, at minimum, two active physical adapter ports. Redundancy across different physical adapters to protect against NIC or PCI slot failure. Redundancy at the physical switch level. Table 17. Virtual Switch Configuration vcloud Resources Switch Name Switch Type Function # of NIC Ports vdresswitch01 Distributed Network pools External networks 2 x 10 GigE (teamed for failover) Page 24 of 59

25 When using the distributed virtual switch, dvuplink ports are the number of physical NIC ports on each host. The physical NIC ports are connected to redundant physical switches. Figure 5 depicts the vsphere virtual network infrastructure design. Figure 5. vsphere Logical Network Design Table 18. dvresswitch01 Teaming and Failover Policies Parameter Port Group Setting Load Balancing All Route based on NIC load (for vds) Failover Detection All Link status Notify Switches All Enabled Failback All No Failover Order Management vmotion External Network Virtual Machine NFS Fault Tolerance Vmnic0 Active, Vmnic1 Standby Vmnic1 Active, Vmnic0 Standby Vmnic0 Active, Vmnic1 Standby Vmnic0 Active, Vmnic1 Standby Vmnic1 Active, Vmnic0 Standby Vmnic1 Active, Vmnic0 Standby Page 25 of 59

26 Table 19. dvresswitch01 Security Policies Parameter Port Group Setting Promiscuous Mode All Reject MAC Address Change All Reject Forged Transmits All Reject Table 20. dvresswitch01 General Policies Parameter Port Group Setting Port binding Production external net Ephemeral no binding Port binding Development external net Ephemeral no binding Shared Storage Logical Design The shared storage design defines how the vsphere volumes will be configured. Following best practices, the shared storage architecture must meet the following requirements: Storage paths will be redundant at the host, switch, and storage array levels. All hosts in a cluster have access to the same volumes. Table 21. Storage Logical Design Specifications vcloud Compute Cluster Attribute Cluster Specification res-pod1 Number of Initial LUNs 100 LUN Size Virtual Machines per Volume 500GB 15 (simultaneous active virtual machines) Page 26 of 59

27 2.4.6 vcloud Resources Datastore Considerations When sizing datastores you have to determine the limit that should be implemented for the number of virtual machines per datastore. The reason for limiting this number is to minimize the potential for SCSI locking due to metadata updates and to spread the I/O across as many storage processors as possible. Most mainstream storage vendors will provide VMware-specific guidelines for this limit, and VMware recommends an upper limit of 15 virtual machines (active) per VMFS datastore, regardless of storage platform. Note that the number of virtual machines per LUN is also influenced by the size and I/O requirements of the virtual machine, but perhaps more importantly, by the selected storage solution and even disk types. When VMware vcloud Director provisions virtual machines it automatically places the virtual machines on datastores based on the free disk space of each of the associated datastores in an organization virtual datacenter. Due to this mechanism, we need to keep the size of the LUNs and the number of virtual machines per LUN relatively low to avoid possible I/O contention. When considering the number of virtual machines to place on a single datastore, some of the following factors should be considered in conjunction with any recommended VMs-per-LUN ratio: Average virtual machine workload/profile (in particular, the amount of I/O). Typical virtual machine size (including configuration files, logs, swap files, and snapshot files). VMFS metadata. Max requirement for IOPs and throughput per LUN, dependency on storage array and design. Max RTO, if a LUN is lost that is, your backup and restore design. If we approach this from an average I/O profile it would be tempting to create all LUNs the same, say as RAID 5, and let the law of averages take care of I/O distribution across all the LUNs and virtual machines on those LUNs. Another approach would be to create LUNs with different RAID profiles based on anticipated workloads to provide differentiated levels of service. These levels of service would be represented at the vsphere level by an HA/DRS cluster and its associated mapped storage and network objects. The vcloud logical design will map provider virtual datacenters to these clusters. To achieve the desired levels of service NewCo will start with one underlying vsphere vcloud compute cluster with dedicated storage. Additional vcloud compute clusters will be attached to their own dedicated storage as they are added. Table 22. vsphere Clusters vcloud Compute Datastores Cluster Name Datastores Quantity RAID Size res-pod1 res-pod1-xx GB Where xx = the LUN ID or volume for that device. As a starting point, VMware recommends RAID 5 storage profiles, and only creating storage tierspecific provider virtual datacenters as one-offs to address specific organization or business unit requirements. Page 27 of 59

28 2.4.7 Datastore Sizing Estimation An estimate of the typical datastore size can be approximated by considering the following factors. Table 23. Datastore Size Estimation Factors res-pod1 Cluster Variable Value Maximum Number of VMs per Volume 15 Average Size of Virtual Disk(s) per VM Average Memory Size per VM Safety Margin 40GB 2GB 20% (to avoid warning alerts) For example, ((15 * 40GB) + (15 * 2GB))+ 20% = (600GB + 30GB) * 1.2 = 636GB Storage VMotion Storage vmotion in ESXi 5.0 has been improved to support migration of linked clones, which is the technology used to implement fast provisioning in vcloud Director. In a vcloud Director environment, the migration of linked clones can only be invoked in the vcloud Director layer, through the REST API Relocate_VM method. In vcloud Director 1.5, the API call is the only method to migrate vapps provisioned through fast provisioning. It is not supported to invoke storage vmotion migration of linked clone virtual machines in the vsphere layer. When invoking the Relocate_VM API to migrate linked clones, make sure that the target organization virtual datacenter is part of the same provider virtual datacenter as the source organization virtual datacenter, or is backed by a provider virtual datacenter that has the same datastore where the source vapp resides. If the condition is not met, the API call will fail. Be aware of following when leveraging Storage vmotion in a vcloud environment: Source and destination volumes for Storage vmotion should both reside within the same provider virtual datacenter or vsphere cluster. For provider virtual datacenters that leverage fast provisioning, linked clones become full clones when virtual machines are migrated using Storage vmotion Storage I/O Control Storage I/O Control (SIOC) provides QoS for VMFS datastores and allows intelligent performance management across all nodes within an HA/DRS cluster. Enabling SIOC on all datastores in a cluster prevents virtual machines from monopolizing storage I/O, and provides a weighting mechanism that supplies adequate performance using relative shares, NFS datastores, or datastores with multiple extents. Page 28 of 59

29 Storage DRS Storage DRS provides initial placement and on-going balancing recommendations for datastores in a Storage DRS-enabled datastore cluster. Currently, datastore clusters cannot serve as a storage source to a provider virtual datacenter, and as a result Storage DRS is not used with resource groups, but is enabled for use with management group vsphere Storage APIs Array Integration vsphere Storage APIs Array Integration (VAAI) enables storage-based hardware acceleration by allowing vsphere to pass storage primitives to supported arrays, offloading functions such as full copy, block zeroing, and locking. VAAI improves storage task execution times, network traffic utilization, and CPU host utilization during heavy storage operations. VAAI is currently supported with NAS primitives and as a result is used with the vcloud resource groups. Page 29 of 59

30 3. vcloud Design Provider Constructs VMware vcloud Architecture Toolkit The vsphere management cluster design encompasses the vsphere hosts contained in the management cluster. In this section, the scope is limited to only the infrastructure that supports the vcloud management component workloads. 3.1 Provider Virtual Datacenters A vsphere cluster will scale to 32 hosts (typically 8-12 is a good starting point, allowing for future growth), allowing for up to 14 clusters per vcenter Server (the limit is bound by the maximum number of hosts per datacenter possible) and an upper limit of 10,000 virtual machines(this is a vcenter limit). The recommendation provided in Architecting a VMware vcloud is to start with 50% of the maximum cluster size and add additional hosts to the cluster as dictated by tenant consumption. When utilization of the total compute resources across the resource group for the cluster reaches 60%, VMware recommends that a new provider virtual datacenter be deployed. This provides for growth within the provider virtual datacenter for the existing organizations/business units without necessitating their migration as utilization nears maxing out a cluster s resources. As an example, a fully loaded resource group will contain 14 provider virtual datacenters and up to 350 ESXi hosts, giving an average virtual machine consolidation ratio of 26:1 assuming a 5:1 ratio of vcpu:pcpu. To increase this ratio, NewCo enterprise would need to increase the vcpu:pcpu ratio that they are willing to support. The risk associated with an increase in CPU over-commitment is mainly in degraded overall performance that can result in higher than acceptable vcpu ready times. The vcpu:pcpu ratio is based on the amount of CPU overcommitment for the available cores with which NewCo is comfortable. For virtual machines that are not busy this ratio can be increased without any undesirable effect on virtual machine performance. Monitoring of vcpu ready times helps identify if the ratio needs to be increased or decreased on a per cluster basis. A 5:1 ratio is a good starting point for a multi-core system. A provider virtual datacenter can map to only one vsphere cluster, but can map to multiple datastores and networks. Multiple provider virtual datacenters are used to map to different types/tiers of resources: Compute This is a function of the mapped vsphere clusters and the resources that back it. Storage This is a function of the underlying storage types of the mapped datastores. Networking This is a function of the mapped vsphere networking in terms of speed and connectivity. Multiple provider virtual datacenters are created for the following reasons: The vcloud requires more compute capacity than a single vsphere cluster (a vsphere resource pool cannot span vsphere clusters). Tiered storage is required; each provider virtual datacenter maps to datastores on storage with different characteristics. Requirement for workloads to run on physically separate infrastructure. Page 30 of 59

31 Table 24. Provider Virtual Datacenter Specifications Attribute Number of Provider Virtual Datacenters Specification 2 Number of Default External Networks 2 VMware recommends assessing workloads to assist in sizing. The following is a sample sizing table that can be used as a reference for future design activities. Virtual machine distribution is based on the percentages outlined in the service offering initial target with a maximum of 5000 virtual machines. Table 25. Virtual Machine Sizing and Distribution Virtual Machine Size Distribution Number of Virtual Machines 1 vcpu/1gb RAM 40% vcpu/2gb RAM 20% vcpu/4gb RAM 25% vcpu/8gb RAM 10% vcpu/16gb RAM 5% 250 Total 100% 5000 Page 31 of 59

32 3.1.1 Provider Virtual Datacenter Sizing Each NewCo provider virtual datacenter corresponds to one and only one vsphere HA/DRS cluster. Though a vsphere 5 cluster can scale to 32 hosts, each new cluster will start with 16 hosts, allowing for future growth until utilization reaches 60%. Through the concept of pooled and abstracted infrastructure, capacity can be added to the vcloud through this method allowing for expansion of provider virtual datacenters and the corresponding clusters without impacting running vapps. If expanding an existing cluster is not an option, VMware recommends that a new provider virtual datacenter and corresponding cluster be deployed. The design calls for two clusters initially sized at sixteen hosts. A single vcenter Server is limited to 1,000 ESXi hosts and 10,000 powered on virtual machines if spread across more than one VMware datacenter. In this configuration each vcenter hierarchy will act as a large resource pool which can scale up through the addition of hosts to existing vsphere clusters or by adding additional vsphere clusters and associated provider virtual datacenters. Multiple clusters can be managed by the same VMware vcloud Director and represent different levels of service. Based on analysis of the existing vsphere environment NewCo averages a 4:1 vcpu to physical CPU Core ratio for their virtual machines. Each one of the vsphere clusters provides approximately 192 usable cores based on the host hardware configuration and HA availability. Based on the estimated vcpu to pcpu ratio of 4:1, this should provide the ability to run 768 virtual machines of similar size and performance characteristics in the vcloud. As previously noted, the risk associated with an increase in CPU over-commitment is mainly degraded overall performance that can result in higher than acceptable vcpu ready times. The vcpu:pcpu ratio is based on the amount of CPU over-commitment for the available cores with which NewCo is comfortable. For virtual machines that are not busy this ratio can be increased without any undesirable effect on virtual machine performance. Monitoring of vcpu ready times helps identify if the ratio needs to be increased or decreased on a per cluster basis Provider Virtual Datacenter Expansion vcloud Director 1.5 introduced the concept of elastic virtual datacenters, allowing a provider virtual datacenter to recognize compute, network, and storage resources from multiple resource pools or vsphere clusters. In vcloud Director 1.5, only Pay-As-You-Go organization virtual datacenters can be backed by multiple resource pools or vsphere clusters. Organization virtual datacenter s that use the Reservation Pool or Allocation Pool allocation models, Committed and Dedicated service offerings respectively, cannot be backed by elastic virtual datacenters. To maintain consistency, all datastores that are mapped to the underlying vsphere clusters beneath an elastic provider virtual datacenter are added to the provider virtual datacenter Provider Virtual Datacenter Storage When creating the provider virtual datacenter, the vcloud administrator adds all of the shared storage LUNs available to the HA/DRS cluster to which the provider virtual datacenter is mapped. Storage LUNs should usually only be mapped to the hosts within an HA/DRS cluster to facilitate vsphere vmotion and DRS. vcloud Director 1.5 does not understand datastore clusters introduced in vsphere 5 so datastores should be added individually to provider virtual datacenters. For the Gold provider virtual datacenter, this means adding the 10 shared storage LUNs from the SAN with the naming standard Fc02-vcdgold01-xx, where xx is the LUN ID. For the Silver provider virtual datacenter, add the eight shared storage LUNs from the SAN with naming convention Fc02-vcdsilv01-xx, where xx is the LUN ID. Keep all of the LUNs within a provider virtual datacenter with the same performance and RAID characteristics to provide a consistent level of service to consumers. Only shared storage should be used to allow for vsphere vmotion and DRS to function. Page 32 of 59

33 3.2 External Networks VMware vcloud Architecture Toolkit A vcloud external network is a logical construct that maps directly to a vsphere port group that has multiple vmnic uplinks to a physical network. This construct represents an external connection for communication in and out of the vcloud. NewCo provides each organization with a guarantee of one routable organization external network, with the ability to request additional organization external networks as needed. Table 26. Provider External Network Specifications Attribute Specification Number of Default External Networks 1 Maximum # of Organization External Networks Default Network Pool Types Used 600 vcloud Director Network Isolation (VCD-NI) Routable Public IP Addresses 4096 or /20 More than one vcenter Server is required to manage 600 networks under a VCD-NI pool. Additional vcenter Servers will be added as VCD-NI network pools are exhausted. 3.3 Network Pools Network pools are a construct in vcloud Director and represent a preconfigured vcloud controlled pool of Layer 2 isolated networks that are automatically used to provide isolation between different organizations or even between vapps within an organization. Aside from the Layer 2 Isolation function, they also enable self-service by allowing the complicated underlying networking configuration to be abstracted from the application owner. NewCo will provide the following sets of network pools based on need: VMware vcloud Director Network Isolation-backed. VLAN-backed. For the VCD-NI pool VMware recommends the transport VLAN (VLAN ID: 1254) be a VLAN that is not in use within the infrastructure. This is for increased security and isolation. 3.4 Users/Roles For security purposes, the system administrators are a separate role and log into a different context than the vcloud consumers who exist within an organization. As a provider construct, the system administrator role has the ability to modify all organizations within the vcloud as well as create and configure objects that vcloud consumers cannot. The role of System Administrator should be reserved for a limited group of system administrators. Because this role has the ability to create and destroy objects as well as make configuration changes that can negatively impact multiple organization s users who possess this role should be knowledgeable about storage, networking, virtualization, and vcloud. The design calls for a single local account (cloudadmin) to be used as a backup for accessing VCD. The primary access method will be managed by adding members to the cloudadmins LDAP group. Page 33 of 59

34 Table 27. System Administrator User and Group Account User/Group Type Role cloudadmin User Local System Administrator NewCo\cloudadmins Group LDAP System Administrator Page 34 of 59

35 4. vcloud Design Consumer Constructs 4.1 Organizations VMware vcloud Architecture Toolkit Except for the service provider default organization whose primary function is publishing vapps and media for consumption by tenants, new organizations are created on demand and are not defined in this section. 4.2 Organization Virtual Datacenters An organization virtual datacenter is a subset of provider virtual datacenter that is backed by a pool of compute, memory, storage, and network resources. An organization virtual datacenter can be expanded by a vcloud system administrator to provide additional capacity by reserving compute, network, and storage up to the existing capacity of the underlying provider virtual datacenter. At NewCo, this expansion would need to be requested by an organization and the corresponding vcenter Chargeback costs would increase automatically through regular synchronization by the Chargeback vcloud data collector Thin Provisioning Thin provisioning will be leveraged by NewCo for templates and shadow virtual machine disk files to conserve storage for certain levels of service. This will be managed and configured from within vcloud Director at the organization virtual datacenter level and will be used only for certain levels of service. See Section 3.1.3, Provider Virtual Datacenter Storage, for details Fast Provisioning Fast provisioning is a feature in vcloud Director 1.5 that enables faster provisioning of vapps through the use of vsphere linked clones. A linked clone uses the same base disk as the original, with a chain of delta disks to keep track of the differences between the original and the clone. Fast provisioning is enabled by default when allocating storage to an organization virtual datacenter. If an organization administrator disables fast provisioning, all provisioning operations result in full clones. VMware recommends enabling or disabling fast provisioning on all organization virtual datacenters (and in turn, all datastores) allocated to a provider virtual datacenter for both manageability and chargeback purposes. For the same reasons, keep datastores separate for fast provisioning and full clone vapp workloads. All organization virtual datacenters created from the same dedicated provider virtual datacenter will have Enable Fast Provisioning selected. Placement of virtual machine disk files in a vcloud environment is based on available free capacity across datastores that are mapped to a provider virtual datacenter. In the case of organizational virtual datacenters that are leveraging fast provisioning, placement will first consider the location of the base or shadow virtual machines until the datastore reaches a preset Disk space threshold which is set for each datastore, and enforces the amount of free space in a datastore. After this threshold is reached the datastore will no longer be considered as a valid target for a clone operation regardless of where the new virtual machines base or shadow disk is located. Page 35 of 59

36 4.3 Organization Networks VMware vcloud Architecture Toolkit Organization networks are not defined in advance. Instead, they are created on demand during the formation of the organization virtual datacenter. During the creation of the organization and first organization virtual datacenter, the service provider will allocate four internal or vapp networks and one internet routable address backed by VDC-NI. 4.4 Catalogs The service provider catalog contains NewCo-specific templates that are made available to all organizations/business units. NewCo will make a set of catalog entries available to cover the classes of virtual machines, templates, and media, as specified in the Public VMware vcloud Service Definition. For the initial implementation, a single cost model will be created using the following fixed cost pricing and chargeback model. Table 28. NewCo Fixed-cost Cost Model Virtual Machine Configuration Price 1 vcpu and 512MB RAM $ vcpu and 1GB RAM $ vcpu and 2GB RAM $ vcpus and 2GB RAM $ vcpu and 3GB RAM $ vcpus and 3GB RAM $ vcpu and 4GB RAM $ vcpus and 4GB RAM $ vcpus and 4GB RAM $ vcpu and 8GB RAM $ vcpus and 8GB RAM $ vcpus and 8GB RAM $ vcpus and 16GB RAM $ Users/Roles By default, only one user is created during onboarding of an organization and that is the system administrator. All other roles, including additional system administrators, are managed by the primary system administrator by importing users into the public vcloud via LDAP synchronization. Page 36 of 59

37 5. vcloud Security VMware vcloud Architecture Toolkit Security is critical for any company. The following sections address host, network, vcenter, and vcloud Director security considerations. 5.1 Host Security ESXi will be configured with a strong root password stored following corporate password procedures. ESXi lockdown mode will be enabled to prevent root access to the hosts over the network, and appropriate security policies and procedures will be created and enforced to govern the systems. Because ESXi cannot be accessed over the network, sophisticated host-based firewall configurations are not required. 5.2 Network Security Virtual switch security settings will be set as follows. Table 29. Virtual Switch Security Settings Function Promiscuous Mode MAC Address Changes Forged Transmits Setting Management cluster Reject Resource groups Reject Management cluster Reject Resource groups Reject Management cluster Reject Resource groups Reject 5.3 vcenter Security vcenter Server is installed using a local administrator account. When vcenter Server is joined to a domain, this results in any domain administrator gaining administrative privileges to vcenter. VMware recommends that this potential security risk be removed by creating a new vcenter Administrators group in Active Directory and assigning it to the vcenter Server Administrator role, making it possible to remove the Local Administrators group from this role. By default, members of the vcloud System Administrator group are not associated with vcenter Administrators group. Page 37 of 59

38 5.4 VMware vcloud Director Security VMware vcloud Architecture Toolkit Standard Linux hardening guidelines need to be applied to the VMware vcloud Director virtual machine. There is no need for local users, and the root password will only be needed during install and upgrades to the VMware vcloud Director binaries. Additionally, certain network ports must be open for vcloud Director use. For additional information see the vcloud Director Administrator s Guide ( In vcloud Director version 1.5, VMware has implemented a new configurable account lockout feature where at the system level, accounts can be configured to lock out for a specified number of minutes after a specified number of failed login attempts. By default, the lockout feature is not enabled, but NewCo has chosen to enable it so that system administrators are locked out for 10 minutes after five failed login attempts. This feature is also available for organization accounts and can be requested during the organization onboarding. 5.5 Additional Security Considerations The following are examples of use cases that require special security considerations: End-to-end encryption from a guest virtual machine to its communication endpoint, including encrypted storage via encryption in the guest OS and/or storage infrastructure. Provisioning of user accounts and/or access control from a single console. Need to control access to each layer of a hosting environment (rules and role-based security requirements for an organization). vapp requirements for secure traffic and/or VPN tunneling from a vshield Edge device at any network layer. Page 38 of 59

39 6. vcloud Management 6.1 vsphere Host Setup Standardization Host profiles can be used to automatically configure network, storage, security and other features. This feature, along with automated installation of ESXi hosts, is used to standardize all host configurations. VM Monitoring is enabled on a cluster level within HA and uses the VMware Tools heartbeat to verify a virtual machine is alive. When a virtual machine fails, causing VMware Tools heartbeat to not be updated, VM Monitoring verifies if any storage or networking I/O has occurred over the last 120 seconds and if not, the virtual machine is restarted. VMware recommends enabling both VMware HA and VM monitoring on the management cluster and the resource groups. 6.2 vcloud Center of Excellence The vcloud Center of Excellence (vcoe) model is an extension of the VMware Center of Excellence model that has been used by many organizations of various sizes to facilitate the adoption of VMware technology and to reduce the complexity of managing a VMware virtual infrastructure. The vcloud Center of Excellence model defines cross-domain vcloud Infrastructure Management accountability and responsibility within team roles across an organization. These team roles enable an organization to consistently measure, account for, and improve the effectiveness of its vcloud infrastructure management even if its IT Service Management roles and responsibilities are distributed across multiple IT functional areas. See Operating a VMware vcloud for more information about the vcoe. 6.3 vcloud Logging Logging is one of key components in any infrastructure. It provides audit trails for user logins and logouts among other important functions. Logging records various events on servers, and helps diagnose problems, and detect unauthorized access. In some cases, regular log analyses and scrubbing will proactively stave off problems that may turn out to be critical to the vcloud operations. NewCo utilizes a centralized, redundant Syslog system for all management virtual machines and applications for error analysis and compliance. Logs captured in Syslog are readily available for analysis for 60 days and available via archive for a minimum of 12 months. Page 39 of 59

40 Figure 6. vcloud Logging Organization Page 40 of 59

41 6.3.1 VMware vcloud Director Logging Two external syslog servers are configured and used as a duplicate destination for the events that are logged locally to vcloud Director. Each vcloud Director cell has been configured to redirect syslog messages to syslog servers with the following log4j.properities changes to %VCLOUD%/etc. This is in addition to importing the copy of the response.properties and global.properties files during the initial installation of subsequent cells. log4j.appender.vcloud.system.syslog=org.apache.log4j.net.syslogappender log4j.appender.vcloud.system.syslog.sysloghost= log4j.appender.vcloud.system.syslog.facility=local3 log4j.appender.vcloud.system.syslog.layout=com.vmware.vcloud.logging.custompatte rnlayout log4j.appender.vcloud.system.syslog.layout.conversionpattern=%d{iso8601} %- 8.8p %-25.50t %-30.50c{1} %m %x%n log4j.appender.vcloud.system.syslog.threshold=info Save the file and restart the vcloud Director cell using service vmware-vcd restart. To enable centralized logging in all the vcloud Director cells, repeat the procedure for each cell Syslog Configuration Depending on your network architecture, it may be valuable to transmit logs to multiple hosts for redundancy. Syslog is UDP-based and stateless, so anywhere there can be network failure, logs transmission is not guaranteed. Some redundancy can be achieved by setting the syslog targets in %VCLOUD%/etc/global.properties and %VCLOUD%/etc/responses.properties to (localhost), and then modifying /etc/syslog.conf to retransmit those syslogs elsewhere, allowing you to send syslogs to two targets. For example, the following line could be placed at the top of the syslog.conf file: *.* ip.address.syslog.host This assumes all logs are wanted. VCD event logs are logged at the user.notice facility and level. If you redirect things such as the Debug and Info logs, those facilities are specified in the log4j.properties file. Such a configuration will also transmit all other logs received by syslog, regardless of facility. Because the logs received from VCD are considered to be remote (as they are sent via a network socket to localhost), the file /etc/sysconfig/syslog must also be modified to give syslogd the correct startup parameters. This line: SYSLOGD_OPTIONS="-m 0" Can be modified to: SYSLOGD_OPTIONS="-r -h -x -m 0" Which instructs syslogd to accept logs remotely, and to re-forward logs received from remote sources. The x flag disables name lookups, which can prevent syslogd from consuming lots of extra resources on name resolution. Page 41 of 59

42 Table 30: vcloud Director Log Locations Log Type Location Host(s) Collection Method VCD Debug Logs %VCLOUD%/logs/* VCD cells %VCLOUD%/logs/vcloudcontainer-debug.log and %VCLOUD%/logs/vcloudcontainer-info.log are redirected to syslog by modification of %VCLOUD%/etc/log4j.properties VCD Syslog Events Sent via Syslog VCD cells Set syslog targets are configured in %VCLOUD%/etc/global.properties and %VCLOUD%/etc/responses.propert ies VCD System Logs Standard Linux Log locations; /var/log/messag es and /var/log/secure VCD cells Syslog targets via syslog.conf or agent retrieval API Web Access Logs %VCLOUD%/logs/*.request.log by date VCD cells Periodical retrieval (logs organized by date. For example: 2010_08_09.request.log) vsphere Host Logging Remote logging to a central host provides a way to greatly increase administration capabilities. Gathering log files on a central server facilitates monitoring of all hosts with a single tool as well as enabling aggregate analysis and the ability to search for evidence of coordinated attacks on multiple hosts. Within each ESXi host, Syslog behavior is managed by leveraging esxcli. These settings determine the central logging host that will receive the Syslog messages. The hostname must be resolvable using DNS. For this initial implementation, all of the NewCo management and resource hosts are configured to send log files to two central Syslog servers residing in the management cluster. Requirements for this configuration are: Syslog.Local.DatastorePath A location on a local or remote datastore and path where logs are saved. Has the format [datastorename directory/filename], which maps to /vmfs/volumes/<datastorename>/<directory>/<filename>. The default datastore location is []/scratch/log/messages. For more information on scratch, see Creating a persistent scratch location for ESXi ( externalid= ). Syslog.Remote.Hostname A remote server's DNS name or IP address where logs are sent using the syslog protocol. DNS name for syslog: mgmt-syslog.example.com. Page 42 of 59

43 Syslog.Remote.Port A remote server's UDP port where logs are sent using the syslog protocol. Default is port 514. #Configure NewCo syslog servers. esxcli system syslog config set --default-rotate 20 --loghost udp://mgmtsyslog1.example.com:514,udp://mgmt-syslog2.example.com:1514 #Configure ESXi logs to send to syslog. esxcli system syslog config logger set --id=hostd --rotate=20 --size=2048 esxcli system syslog config logger set --id=vmkernel --rotate=20 --size=2048 esxcli system syslog config logger set --id=fdm --rotate=20 esxcli system syslog config logger set --id=vpxa --rotate= vcenter Orchestrator Logging vcenter Orchestrator uses Apache log4j, which allows for granular logging at runtime without modifying the application. The target of the log output is configured by default to a set of files, but has been routed instead to the NewCo Syslog servers. The log configuration file is named log4j.xml and is located in install_directory\appserver\server\vmo\conf. The following section was added to enable logs to be routed to Syslog with X.X.X.X changed to the Syslog server and LOCAL2 replaced with the correct log facility. <!-- ============================== --> <!-- Append messages to the syslog--> <!-- ============================== --> <appender name="syslog" class="org.apache.log4j.net.syslogappender"> <param name="sysloghost" value="x.x.x.x"/> <param name="facility" value="local2"/> <param name="facilityprinting" value="true"/> <layout class="org.apache.log4j.patternlayout"> <param name="conversionpattern" value="%t %5r %-5p %-21d{yyyyMMdd HH:mm:ss,SSS} %c{2} [%x] %m %n"/> </layout> </appender> The <root></root> section must include: <appender-ref ref="syslog"/> Each of the vcenter Orchestrator application server components and plug-in adapters provides logs at different levels including fatal, error, warning, info, and debug. During normal operations it is recommended to use the default info level to maximize the server performance while still keeping quite detailed level information. Using the debug level for a single component or for all server components is recommended for troubleshooting. When troubleshooting vcenter Orchestrator with vcloud Director, setting the vcloud Director plug-in in debug mode logs the REST calls and responses. This can be changed in the following section as follows: <category additivity="true" name="com.vmware.vmo.plugin.vcloud"> <priority value="debug"/> </category> Page 43 of 59

44 During overall server troubleshooting it is recommended to set the server and all the components in debug mode. This can be done by changing the log level in the three following sections. In the section: <appender class="org.jboss.logging.appender.rollingfileappender" name="file"> Update this line as follows: <param name="threshold" value="debug"/> In the section: <appender class="org.apache.log4j.consoleappender" name="console"> Update this line as follows: <param name="threshold" value="debug"/> In the section: <!-- VMware vco --> Update this section as follows: <category additivity="true" name="ch.dunes"> <priority value="debug"/> </category> 6.4 vcloud Monitoring Monitoring a vcloud instance gives the service provider insight into the health of their vcloud services to help meet SLAs and proactively notify regarding any potential capacity shortfalls. vcloud management systems can be monitored using vfabric Hyperic as a single dashboard integrated with agents installed on the management virtual machines. Table 31. VMware vcloud Director Monitoring Items Scope Item System Leases Quotas Limits vsphere Resources CPU Memory Network IP address pool Storage free space Virtual Machines/vApps Not in scope Page 44 of 59

45 6.4.1 vcloud Director As of VMware vcloud Director 1.0 monitoring is performed via custom queries to VMware vcloud Director using the Admin API to capture the summary consumption data on organization virtual datacenters, through MBeans, and through standard Linux and JMX monitoring services running on the vcloud Director guest. Some of the components in VMware vcloud Director can also be monitored by aggregating the Syslog-generated logs from the different VMware vcloud Director cells that would be found on the centralized log server vcenter Server Multiple vcenter Servers are present within a vcloud instance to manage the virtual resources within the management and resource groups. Use of vcenter alerts and health of the vpxd service provides notice when vcenter Servers are resource constrained or have suffered a fault. Use of the Hyperic plug-in for vsphere enables detailed monitoring for the vcenter Servers, ESXi hosts, and virtual machines running within them. See the Monitoring vsphere Components guide ( for more information about available objects and alerts vcenter Orchestrator The vcenter Orchestrator application server is monitored using Java Management Extensions (JMX). To enable JMX for a vcenter Orchestrator server started as a service: 1. Create a jmxremote.password file in install_directory\appserver\server\vmo\conf. The file takes the form of a username and a password separated by a space on each line. An example is: monitor password. 2. Create a jmxremote.access file in install_directory\appserver\server\vmo\conf. The file takes the form of a username and a permission separated by a space on each line. An example is: monitor readonly. 3. Secure each of the jmxremote files on Windows as described in the Oracle Java documentation: How to Secure a Password File on Microsoft Windows Systems ( and for non-windows systems simply use the following command: chmod 600 jmxremote.password jmxremote.password Note Essentially, each of the jmxremote files is only accessible to the owner. Final security properties on a Windows server will show Full Access to the owner (default is Administrators group) and no other users, groups, or SYSTEM listed for access. Additionally, windows explorer adds a lock icon next to the filename. 4. Edit install_directory\app- server\bin\wrapper.conf by adding the lines below after the line "wrapper.java.additional.9...": wrapper.java.additional.10=-dcom.sun.management.jmxremote.authenticate=true wrapper.java.additional.11=-dcom.sun.management.jmxremote.password.file=../server/vmo/conf/jmxremote.password wrapper.java.additional.12=- Dcom.sun.management.jmxremote.access.file=../server/vmo/conf/jmxremote. access Page 45 of 59

46 wrapper.java.additional.13=-dcom.sun.management.jmxremote.ssl=false wrapper.java.additional.14=-dcom.sun.management.jmxremote.port=1099 wrapper.java.additional.15=- Djavax.management.builder.initial=org.jboss.system.server.jmx.MBeanServerBui lderimpl wrapper.java.additional.16=-djboss.platform.mbeanserver Note The numbers in wrapper.java.additional.## must be in order without any gap between them; otherwise the wrapper ignores them. If the last property is wrapper.java.additional.10 then all above properties should be shifted by 1 starting at wrapper.java.additional.11 JMX monitoring will be available after restarting the vcenter Orchestrator server and is set up during the initial vcloud deployment Testing of the Monitoring JConsole is a GUI application from the Java Development Kit that is designed for monitoring Java applications. The jconsole executable is in JDK_HOME/bin, where JDK_HOME is the directory where the JDK is installed. If this directory is in system path, the tool can be started by typing jconsole in a command (shell) prompt. JConsole lists local processes and provides the option to enter a remote one with the hostname:port syntax. After connecting you can monitor the memory, threads, and managed beans. Table 32 represents a subset of MBeans that can be are used for monitoring the performance of a vcenter Orchestrator instance. Table 32.vCenter Orchestrator Monitored MBeans Workflow Execution Mbean Description Attribute ExecutorsActiveCount ExecutorsQueueSize ch.dunes.workflow.engine.mbean.workflowengine Statistics about active workflows. Description Number of currently active workflows. Number of workflows queued. Web views Mbean Description Attribute accesscount cachemaxsize jboss.web:type=cache,host=[hostname],path=/vmo webview statistics Description Number of accesses to the cache. Max size of resources which will have their content cached. Page 46 of 59

47 cachesize desiredentryaccessratio hitscount Max size of resources which will have their content cached. Entry hit ratio at which an entry will never be removed from the cache. Number of cache hits. Apache Tomcat Global Request Processor Metrics Mbean Description Attribute bytessent bytesreceived processingtime errorcount maxtime requestcount jboss.web:type=globalrequestprocessor,name=http-[hostname]-[port] Apache Tomcat Global Request Processor Metrics Description Bytes sent by all the request processors running on the Apache Tomcat container. Bytes received by all the request processors running on the Apache Tomcat container. Total processing time (in milliseconds) since startup. Error count on all the request processors running on the Apache Tomcat container. Maximum time it took to process a request. Request count on all the request processors running on the Apache Tomcat container. vco web service Mbean jboss.web:type=manager,path=/vmware-vmo-webcontrol,host=[hostname] Description Attribute activesessions expiredsessions maxactive processingtime sessionaveragealivetime sessioncounter sessionmaxalivetime Description Number of currently active sessions. Number of sessions that have expired. Maximum number of sessions that have been active at the same time. Total processing time (in milliseconds) since startup. Average time (in seconds) that expired sessions had been alive. Total number of sessions created by this manager. Longest time (in seconds) that an expired session had been alive. Page 47 of 59

48 WebViewEngine Mbean jboss.web:j2eetype=servlet,name=vso- WebViewEngine,WebModule=//localhost/vmo,J2EEApplication=none,J2EE Server=none Description Attribute maxtime processingtime sessionmaxalivetime requestcount Description Maximum time taken (in milliseconds) for processing a request. Total processing time (in milliseconds) since startup. The longest time (in seconds) that an expired session had been alive. Total number of requests served since startup. Web Tread pool Mbean jboss.web:type=threadpool,name=http-[hostname]-[port] Description Attribute currentthreadcount currentthreadsbusy Description Number of threads created on the Apache Tomcat container. Number of busy threads on the Apache Tomcat container. Page 48 of 59

49 7. Extending vcloud 7.1 Hybrid vcloud A hybrid vcloud is a vcloud infrastructure composed of two or more vcloud instances (private or public) that remain unique entities but are bound together by standardized technology that enables data and application portability (for example, cloudbursting for load balancing resources between vcloud instances). NewCo allows organizations to extend their existing private virtual environment through cloudbursting and IPSEC VPN between organizations to extend the datacenter. See Hybrid VMware vcloud Use Case for details on how private and public vcloud instances can be associated with each other. 7.2 vcloud Connector VMware vcloud Connector (vcc) is an appliance that allows vsphere administrators to move virtual machines from vsphere environments or vapps from a vcloud to a remote vcloud. The origination and destination vcloud can be a public or public vcloud. Figure 7 provides an overview of communication protocols between vcloud Connector and vcloud Director: Figure 7. vcloud Connector CB Server On Premise - Private Cloud vcloud Director vcloud Director CB Server vcenter Server VCC Appliance REST APIs REST APIs REST APIs Off Premise - Private Cloud VCC Plug-In for vsphere Client On Premise vsphere Virtualization vcloud Director CB Server Public Cloud vcloud Connector Design Considerations vcloud Connector requires vsphere 4.0 or later and administrative privileges. The appliance requires a dedicated static IP address. It is recommended that the appliance reside on the same subnet as vcenter Server. Ports 80, 443 and 8443 must be open on any firewall to allow communication between vcenter and the vcloud Connector appliance. Page 49 of 59

50 Table 33. vcloud Connector Components VMware Components VMware vsphere 5.0 vcloud Connector (vcc) appliance vsphere Client Microsoft Internet Explorer 7 or higher Description VMware virtualization software. vcc appliance deployed on vsphere vcc plug-in installs in vsphere Client. Requirement for vsphere plug-in. 7.3 vcloud API There are two ways to interact with the vcloud Director cell; via the browser-based UI or via the vcloud API. The browser-based UI has limited customization capability. To enhance the user experience a service provider or enterprise may want to write their own portals to integrate with vcloud Director. To enable integration, the vcloud API provides a rich set of calls in VMware vcloud Director. vcloud APIs are REST-like (which allows for loose coupling of services between the server and consumer), are highly scalable, and use HTTP/S protocol for communication. The APIs are grouped into three sections based upon the functionality they provide and type of operation. There are several options available to implement the custom portal using the vcloud API: VMware vcloud Request Manager, VMware vcloud Orchestrator, or by using third-party integrators. Some of these may require customization to design workflows to satisfy customer requirements. Figure 8 shows a use case where a service provider has exposed a custom portal to end users on the Internet. Figure 8. vcloud API Logical Representation Page 50 of 59

51 End-users log into the portal with a valid login/password and are able to select a predefined workload (from a catalog list) to deploy. The user s selection, in turn, initiates a custom workflow that deploys the requested catalog item (vapp or media) in the vcloud. Currently, the vcloud API is available in the form of a vcloud SDK with Java, C-Sharp, and PHP language bindings. 7.4 VMware vcenter Orchestrator Because vcloud Director leverages core vsphere infrastructure, automation is possible through vcenter Orchestrator. vcenter Orchestrator provides out-of-the-box workflows that can be customized to automate existing manual tasks. Administrators can use sample workflows from a standard workflow library that provides blueprints for creating additional workflows, or create their own custom workflows. vcenter Orchestrator integrates with vcloud Director through a vcloud Director plug-in that communicates via the vcloud API. vcenter Orchestrator can also orchestrate workflows at the vsphere level through a vsphere plug-in, if necessary. Figure 9. vcloud Orchestration Managing vcloud Resources vcenter Orchestrator provides numerous plug-ins for managing a vcloud environment from the compute and storage layer through vsphere and the vcloud Director instance. NewCo utilizes workflows contained in the plug-ins to assist in adding vcloud compute capacity and customer management including onboarding, offboarding, and user management to reduce operations associated with a vcloud system administrator. Page 51 of 59

52 7.4.2 vcenter Orchestrator Active-Passive Node Configuration High availability for vcenter Orchestrator can be used to reduce downtime during a planned service maintenance event or unplanned service outage on one or all of the following components: vcenter Orchestrator application server. vcenter Orchestrator Database. vcenter Orchestrator host operating system. vcenter Orchestrator host virtual machine. Cluster hosting the virtual machine. Datacenter/site failure. The vcenter Orchestrator server application is a Windows service that can be controlled with scripts using the command line interface. The vcenter Orchestrator server application is stateless. The workflows and their state are stored in a database. The vcenter Orchestrator server application implements checkpointing. It can resume running workflows from their saved state. Only one vcenter Orchestrator server application node can run per database. The application server has a local file based configuration required to start the service and the orchestrated systems. When making vcenter Orchestrator highly available the first thing to implement is multi-master or master-slave database replication. Also called cold standby, this provides a fully redundant instance of each node, which is only brought online when its associated primary node fails. As long as a copy of the database is available, a vcenter Orchestrator application server with the appropriate configuration can resume workflow operations. Specific database vendor best practices must be followed to implement database high availability. This is the configuration that suits best vcenter Orchestrator. A third-party clustering application can be set up to check on the server availability (for example, by monitoring the Web service), and upon failure stops the primary node and starts one of the secondary nodes. This requires that all of the vcenter Orchestrator application servers have the same plug-ins installed and, except for the IP address in use, the same configuration. This can be done having each node maintain its own copy of the cluster configuration data. The configuration on the nodes can be initially set using the vcenter Orchestrator web configuration application and exported manually to the other nodes, and then upon configuration change updated using file replication scripts on the <vco Installation Folder>\appserver\server\vmo\conf and <vco Installation Folder>\appserver\server\vmo\plugins directories. Alternatively, installing vcenter Orchestrator on the quorum drive is possible, but requires scripting the update of the IP address configuration of the Orchestrator application server (in <vco Installation Folder>\app-server\bin\boot.properties) for the new host as part of the automated failover and failback operations. Page 52 of 59

53 The first approach is recommended because compared to the alternative this: Allows recovery when the application server file structure integrity is compromised. Guarantees optimum performance for geographically dispersed clusters. Improves traceability by separating the file-based logs. Permits resumption of availability in thirty seconds to two minutes (the time required for a vcenter Orchestrator server to start). Page 53 of 59

54 8. vcloud Metering VMware vcloud Architecture Toolkit To track resource metrics for vcloud entities, vcenter Chargeback sets allocation units on the imported vcloud Director hierarchies based on the allocation model configured in vcloud Director. Table 34 shows which allocation units are set. Table 34. Allocation Units for vcloud Hierarchies Based on Allocation Model Entity Pay-As-You-Go Allocation Pool Reservation Pool Organization virtual datacenter None CPU Memory CPU Memory Storage Storage vapp None None None Virtual machine vcpu vcpu vcpu Memory Memory Memory Storage Storage Storage Template Storage Storage Storage Media file Storage Storage Storage Network DHCP DHCP DHCP NAT NAT NAT Firewall Firewall Firewall Count of Networks Count of Networks Count of Networks 8.1 Cost Models Installing vcloud Director and vshield Manager data collectors also creates default cost models and billing policies that integrate with vcloud Director and vshield Manager. Billing policies control costs assessed to resources used. Default vcloud Director billing policies charge based on allocation for vcpu, memory, and storage. Costs are charged for on an hourly, daily, weekly, monthly, quarterly, biannual, and yearly basis. Instead of modifying default billing copies and cost models, make copies and modify the duplicates. For more information, see the vcenter Chargeback User s Guide ( for vcenter Chargeback version Page 54 of 59

55 Rate factors allow the scaling of base costs for a specific chargeable entity. Example use cases include: Promotional rate A service provider offers new clients a 10% discount. Instead of modifying base rates in the cost model, apply a 0.9 rate factor to reduce the base costs for client by 10%. Rates for unique configurations A service provider decides to charge clients for special infrastructure configurations using a rate factor to scale costs. VM instance costing assigns a fixed cost to a hard bundle of vcpu and memory. This option is only available with the Pay-As-You-Go allocation model. Use VM instance costing to create a fixed cost matrix for different virtual machine bundles. 8.2 Reporting vcenter Chargeback generates cost, usage, and comparison reports for hierarchies and entities. The vcenter Chargeback API provides the capability to export reports to XML. Developers use XSLT to transform the raw XML into a format supported by the customer s billing system. Reports run from the vcenter Chargeback user interface are available in PDF and XLS format. Service accounts with read-only privileges have been created to run reports from the vcenter Chargeback UI or API. 8.3 Metering Internet Traffic Internet traffic is network traffic that extends beyond the vcloud environment to the Internet. For routed external organization networks, internet traffic is the traffic sent and received through the vshield appliance. vcenter Chargeback pulls network metrics sent through vshield Edge devices (send and receive) from vshield Manager. A usage model bills for the network bandwidth use by applying base rates to Network Received and Network Transmit metrics. This is the default billing policy type. A fixed cost-based cost model allows billing for different types of Internet services and usage based on an agreed upon fixed price. Example fixed costs include: Monthly fixed rate for a specified bandwidth cap Instead of charging for actual usage, the client is billed a fixed fee for Internet usage through the creation of a fixed cost in vcenter Chargeback. Basic monthly fixed costs on top of Internet usage (application monitoring tools and reports supplied by a solution provider). Additional fixed costs that incurred due to upfront infrastructure needed (for example, a new router for the client). Figure 10 provides an example of a one-off router cost of $150. Page 55 of 59

56 Figure 10. One Time Router Cost Example (vcenter Chargeback UI) Additional chargeable vcloud resources include: Count of networks that belong to an organization or a vapp. NAT service. DHCP service. Firewall service. VPN service. External network bandwidth measured in Mbps/hour. 8.4 Aggregator Reporting Public vcloud providers under the VMware Service Provider Program (VSPP) are required to report on the hourly virtual machine vram usage within the resource groups. NewCo has deployed the vcloud Usage Meter to meter the resource groups and report back vram usage to the aggregator on the fifth of every month. vram data collected from the resource groups is kept on file or within the vcloud Usage Meter database for a minimum of 12 months in the event of an audit by the aggregator or VMware. Page 56 of 59