I T S E R V I C E S Son Truong Systems & Operations Unix Technical Lead November 2016 SERVER PROVISIONING: Linux Virtual Machine (VM) Provisioning on the Hyper-V Platform Standard Linux VM Introduction One of the most significant service offerings of the Systems and Operations (SysOps) Unix Team is to provide Linux server facilities for IT operational, research and teaching groups and individuals. Our server provisioning preference and facilities are in the form of virtual machines (VMs) on the Microsoft Hyper-V (virtualisation) platform. There are exceptions to this general preference rule, of course. One example would be the request for server facilities to host a high performing database. Where appropriate we will consider hardware server provisioning if virtual servers are not suitable. Purpose of this Document This document defines the Standard Linux VM offerings and outlines the procedure for users and groups to follow, to attain VMs for operational, research and teaching purposes. There is a need to follow a formalised process to enable SysOps Unix to bound the scope of our work so that we able to effectively and responsively support, maintain, upgrade and manage the complete life-cycle of the VMs that we provide. In the process of defining the Standard Linux VMs, we will provide resources, guidelines, and instructions for the complete process of applying for and obtaining Linux VMs such that you can run services or IT facilities for operational, research or teaching purposes. Standard Linux VM provisioning will be very simple and quick if all the information required is given clearly and accurately on application. Requests that deviate from the standard VM offerings will need a more involved discussion and negotiation to define the requirements. The level of support and resources needed will have to be agreed and the quantification and allocation of costs needs to be met. There is also a need to define server configurations and the support of the applications or service that run on the provisioned VM(s). There will be a definition of the level of responsibilities and who in terms of SysOps Unix or the VM owner will be performing the installation, configuration and maintenance of the applications or service that are needed. Page 1 of 7
1. Standard Linux VM Definitions Firstly, we are going to define the supported Linux distribution(s) on which the VMs run. We will then define the application stacks that we can implement and support along with the standard VMs. Secondly, we will outline how the OS, the application stacks and other VM requirements such as security and updates, are implemented using a configuration management facility. Lastly, we will give a definition of Self-supported VM and in what circumstances this type of VM is most suited for your use. The flexibility of the self-supported VM comes with responsibilities which the owner must take on, so some of them will be outlined for your attention. 1.1. Linux Operating System - Distribution and Versions We need to standardise on a Linux operating system (OS), the distribution and its versions, purely for affordability, supportability and future-proofing. CentOS Linux 1 is the Community Enterprise Operating System and it is based directly on Red Hat Enterprise Linux (RHEL) from the sources freely supplied by them to the community. As such, CentOS Linux aims to be functionally compatible with RHEL and as a subsequent supported by Red Hat in terms of downstream security patches, software updates and version releases. CentOS is no-cost, free to use and distribute. Due to the above reasons and together with RHEL release and support lifecycle, CentOS Linux is ideal for IT Services use as the preferred OS and distribution for Linux server provisioning. To read and find out more about RHEL lifecycle please follow this link: https://access.redhat.com/support/policy/updates/errata Currently, RHEL version 6 and 7 are both in their production lifecycle, so we are able to provision VMs with CentOS version 6 or 7. We prefer to provision new VMs with CentOS 7, and still support and run existing VMs on CentOS 6. 1.2. VM Resource Specifications Virtual machines have three resource options: vcpus, memory and disk space. These three resources and their amount is determined by the two types of resource profiles: Production and Test/Dev 2 VMs. The reason for the split of resource allocation into the groups is based on their criticality and the underlying support infrastructure. Production VMs are hosted on resilient and high performance hardware, storage and network infrastructure. Test/Dev VMs maybe hosted on less performant or resilient infrastructure with smaller capacity due to the nature of their critical needs. Below is the current resource allocation of production and test/dev Linux VM. The production specification defines one Linux VM unit and so when more resources are need when requesting VMs, they will be measured in terms of this base unit for the purpose of costing and recharging, if and when these policies come into play see Section 2.5. 1 For more information on CentOS please visit: https://wiki.centos.org/ 2 Dev denotes Development. Beta and demo(stration) VMs fall into the Test/Dev VM profile. Page 2 of 7
The standard production VM running CentOS 6 is: 4 vcpus, 4GB RAM, 60GB (incl. OS) of disk space. The standard test/dev VM running CentOS 6 is: 2 vcpus, 2GB RAM, 60GB (incl. OS) of disk space. An additional request for disk space can be made of up to 150GB for production VM and 50GB for test/dev VM and is mounted under /srv: CentOS 6 Production Test/Dev vcpu 4 2 Memory 4 GB 2 GB Diskspace 60 GB (< 150GB) 60 GB (< 50GB) The standard production VM running CentOS 7 is: 4 vcpus, 4GB RAM, 100GB (incl. OS) of disk space. The standard test/dev VM running CentOS 7 is: 2 vcpus, 2GB RAM, 100GB (incl. OS) of disk space. An additional request for disk space can be made of upto 300GB for production VM and 100GB for test/dev and is mounted under /srv: CentOS 7 Production Test/Dev vcpu 4 2 Memory 4 GB 2 GB Diskspace 100 GB ( < 300GB) 100 GB (< 100GB) 1.3. Application Stacks Web servers and web technology: this is our most supported and popular VMs requests. Apache 2 is our preferred web server technology (not ngix) and the following associated application stacks can be provisioned: LAMP - Linux, Apache, MySQL, PHP (and variations, e.g. PHP, Python) Django - Apache, Python, Django (and variations, e.g. Perl, PHP) Java - Apache, Tomcat, Java (and variations, e.g. PHP, Perl, Python) A combination of these technologies can be provided together with owner installable software which means that these stacks should be able to provide users with the tools to form their service. The only caveat is that only the versions provided by the OS release repository are supported by the SysOps UNIX team. For example, if the CentOS 7 distribution provides PHP at version 6, then the standard application stack that includes PHP will be the one at that version. If users require another version or PHP 7, then they must ensure they provide their own system administration resource to set up and configure their VM such that the standard version is removed and their required version installed. Page 3 of 7
Please note that the MySQL in the LAMP stack will be provided on the Central MySQL Cluster service see Section 9. This facility may require you to create another service ticket requesting for the database. Certainly the client software or application will be installed as part of the LAMP stack. 1.4. Puppet Configuration Management To enable SysOps Unix to deliver, support and maintain a large number of provisioned VMs, we use Puppet Enterprise as the configuration management tool. Puppet allows us to deliver standard VMs, configured as desired quickly, effectively and securely. Puppet configuration management also ensures that critical and secure settings are not accidentally modified or removed by those with the right permissions. Puppet also allows some VM settings, such as user access and permissions, be changed by other IT support staff. This allows a devolution of administration to Zonal IT staff so that support is readily available and can direct for you. The use of Puppet will be the mechanism that installs and maintains the application stacks defined in (2.3). Puppet will be able to guarantee the software to be always present and if there are services associated, e.g. Apache httpd, then it will ensures that the service is also always up and running. It will not however, manage the application s configuration for you. You must have application configuration and management skills to be able to set up and main the service for your needs. The standard Linux VM is provisioned with the OS configured and maintained by the SysOps Unix Puppet base modules, forming base roles and profiles that are applied. Standard services such as NTP, timezones, LDAP integration as well as security features, such as the local firewall, and user access and permissions are present and enabled. Security updates are applied daily, so any patches or software updates which are pushed to the repositories are applied. To find out more about SysOps use of Puppet Configuration Management please read the SysOps Unix Puppet Enterprise wiki, available here. 1.5. Self-Supported VM A Self-supported VM is one that is provisioned with a bare-bones CentOS distribution that can be selfadministered to enable the setup of any application and software that is needed to test, develop and run a service or provide the needed function. You can request a self-supported VM if the standard VM with the available application stacks defined in section 2.2 and 2.3 is not suitable for your use. The OS on a self-supported VM is still managed by SysOps Unix, with the Puppet base role or profile enforcing system updates, access and security policies. Owners of these VMs will have full sudo permissions and can choose to install and configure their software as they see fit. The self-supported aspect of these VMs, is the support of any non-standard (to the distribution) or third-party software and configuration that is required to run on them. The owner of the VM is not only responsible for the installation and setup, but also the continuing maintenance, update and security patching of them. Please see section: Support and Responsibilities for further detail. The important aspect of a self-supported VM is that SysOps Unix can only provide reasonable efforts help and resources, and ultimately we do not have an infinite pool of resources, knowledge or skills to be able to solve any technical or resource issues that might arise. Page 4 of 7
With this in mind, if you are requesting a self-supported VM, it is explicitly expected that you have the systems and application administration and maintenance skills to be able to install, configure and run the software or applications that you require. Whatever software or application used and deployed, they and/or their use should not contravene the University and IT Services terms and conditions of acceptable use of IT. An extra important point in the adoption or use of software, applications or technologies that is new, innovative, or specialised, is that their use must be approved by the University s IT Services Technical and Enterprise Architecture designs and standards. Please see the ITAS Standard papers and best practice guides, available here. There are also best practices and guidelines to follow, especially in terms of resilience and reliability, when running critical production services. There is also greater importance and emphasis put on the consideration of data security, data protection and copyright issues related to any services you provide. 1.6. Infrastructure and Networking Our current virtualisation infrastructure is Hyper-V. The hypervisor and its management technologies should not be too much of a concern for the VM owner. The infrastructure is architected for redundancy, resilience and reliability, which is designed to deliver enterprise class IT services. The VM infrastructure is sufficiently costly for IT Services to run such that VM requests, although we provide without charge, there will be some cost and resources will be consumed and must be replenished. Requesting one or two VMs per project is a reasonable demand but requesting 10 or 15 VMs require negotiation and external (to IT Services) resourcing. The Hyper-V infrastructure allows the hosting of critical services on VMs which can replicated from one datacentre to another. This is purely for disaster recover purposes only if the primary datacentre Hyper-V infrastructure fails, then the replicated VM which lives in the second datacentre can be activated to immediately recover the service within a minimal downtime period. If your service criticality level is deemed as Silver or Gold, you must inform SysOps Unix upon request such that we automatically replicate the provisioned VM. Standard VMs are provisioned on private network behind the F5 firewall/load-balancers and as such run on a network that is partially isolated from other parts of the University s network and services. There are a set of standard ports open for certain traffic to flow in and out of this private network but not all ports can be open. For example, access to RDSF is not currently possible from within this private network due to policies implemented by the providers. It is also possible to request standard VMs that run on a different network/subnet, but this requirement must be detailed at the time of the request. However, not all networks/subnets are available to VMs that are created on the Hyper-V infrastructure. For external access to services that run on these provisioned VM, we need to use the F5 firewall/loadbalancers to forward traffic back and forth between service/external networks to private network. The most popular example of this is the forwarding of HTTP/HTTPS traffic from and to this private network and the internet. Using the F5 for such traffic control allows you to architect your services for reliability and resiliency so bear this in mind when designing your services. Where your service is suitably designed and the F5 supports this design, then the use of the F5 can allow outages of individual VMs without disrupting the availability of your service. Page 5 of 7
2. Support and Responsibilities 2.1. SysOps Unix The responsibilities of SysOps Unix are based mainly on the operating system, networking and its security aspects when regarding the actual Virtual Machine. The Hyper-V virtualisation infrastructure on which the VMs run is mainly the responsibility of the SysOps Microsoft team. Hyper-V replication is available for critical service VMs but this high availability facility must be discussed and arranged. When SysOps provision a standard VM, we are responsible for its OS, networking, and security setup and configuration. Whenever there is a fault, failure or misconfiguration, a TopDesk ticket must be raised so that we can investigate the problems and fix them accordingly. File/Networker backup of Production VMs is performed automatically. However, for development or test VMs, owners will need to explicitly request file-based backup if this is needed. Security and software updates of packages from official distribution repositories (OS and other repositories such as EPEL) are done automatically each day (using Puppet). Other critical patching and security updates might need to be done when and if they are needed - with or without notification. Access control and restrictions are applied implicitly at the local VM level which means the use of the local firewall. All ports are closed with only those needed opened for use and access. 2.2. Zonal Support and Responsibilities It is expected that there be some pre-consultation by the requester/owner with their Zonal support team before making a VM request - even if it is a standard VM. This pre-consultation should be of benefit to you as the Zonal support team should be able to give you procedural and technical advice before requesting and owning a VM. Enterprise business support and a customer focused service is what we aim to provide as IT Services to the rest of the University. This Zonal contact and support is also an important step for SysOps Unix and this enables us to link-up the service and support from end-to-end. 2.3. VM Requester/Owner It is important that when requesting a standard VM, the requester (and eventual owner, administrator or service manager of that VM) understands what SysOps are providing (2.1. above) and what is expected of the owner. The provision of a standard VM does not guarantee any kind of gold service and 100% uptime or availability. If this is the level of requirements that your service needs, then this should not be a standard VM request. A standard VM request allow us to quickly and efficiently provide you with a VM that can fulfil your simple needs immediately. Whether this request is for a test, development or production VM you should be able to install and setup your service quickly and have a Linux server that can satisfy this simple requirement. Where there s a need for more permanent, higher service-level, or resource demanding VMs, then these requirements deviate from the definition of the standard VM and the non-standard VM request Page 6 of 7
procedure should be initiated and a more appropriate VM or alternative physical machine provisioning can be made. It is the responsibility of the owner to initiate this non-standard VM request by reviewing the changed requirements of a pre-provisioned VM and talking with SysOps Unix. The main responsibility of the VM requester/owner is for the service that is run on the provisioned VM. Unless the service is a SysOps Unix provided and maintained one (see section 4, above), any services that are being provided on a provisioned VM are the responsibility of the VM owner. What this means is that we are unable to get involved with the installation, setup and configuration of software packages that are needed for this service. This especially applies for self-supported VMs or standard VMs that deviate from their initial request or have a complicated software setup or design. As SysOps Unix we are very knowledgeable on the operating side of the CentOS Linux distribution, but have limited knowledge on its software packages and the services that can be configured to provide. It is assumed that the requester of the VM is already proficient and knowledgeable on the configuration and implementation of the services. The emphasis, again when setting up and running any service is its security and safety aspects. The VM owner must ensure that your data (and the underlying OS) is secure and protected against exploitable vulnerabilities at the service level. You must ensure that the services abide by the standards and recommendation set by the Enterprise and Technical Architecture boards run by the University and IT Services. 2.4. Project Go Live Process and Checklist If your VM request is part of a pilot or bigger project please provide full details of this at the time of application. It is important that you proceed and adhere to the IT Services Project and Go Live processes and that the Go Live checklist must be read and completed to ensure what is required by you and that you contact the parties involved who needs to sign off the various IT aspects for your Go Live approval. Please attach all relating document to the VM requests such that we can check to see if we can fulfil the criticality and service level requirements of your service/project. 2.5. Resources, VM quota and Cost/Charging Model Currently, VMs are provisioned without charge for the compute and system administration resources they consume. However, this situation might change. You will need to ensure that your project will be able to meet cost and charging if they are introduced. Page 7 of 7