Technical Plan Version 7.11

Transcription

1 Technical Plan Version 7.11 January 6, 2014

2 Table of Contents Table of Contents... Section 1: Environment : Network... Section 2: Hardware/Software : Storage : Hypervisor : Operating System : System & Operating System Licensing : Portal... Section 3: Architecture... Section 4: Life Cycle... Section 5: Incident Management... Section 6: Maintenance and Operations... Section 7: Security... Section 8: Operational Recovery : Backup : Infrastructure Disaster Recovery... Section 9: Operational Risk... Section 10: Operating Level Agreements : OTech s Responsibilities : Contractor s Responsibilities : Data Communication Responsibilities : Business Continuity Responsibilities : Availability Responsibilities... ii P a g e ii

3 10.2.4: Performance & Monitoring : Billing : Reporting... Section 11: Technical Plan Approval P a g e iii

4 Section 1: Environment CalCloud is a private, on-premise Infrastructure-as-a-Service (IaaS) service offering. The infrastructure is 100% virtual and will be hosted at the Office of Technology s (OTech s) data centers located in Rancho Cordova and Vacaville. Equipment may be housed in either the contractor provided racks or the OTech s existing racks on either of the raised floors. All equipment will utilize the OTech s power strips. The hardware will utilize 208V power while employing dual power supplies and dual power feeds in order to support redundancy. 1.1: Network CalCloud is aligned with best-practice industry security standards, and will reside on the OTech s 10Gb Local Area Network (LAN) behind the OTech s firewalls, load balancers, and intrusion prevention system (IPS). Primary CalCloud servers will be located at the OTech s data center in Rancho Cordova while Disaster Recovery (DR) will be located at the OTech s data center at Vacaville. Connectivity between the two data centers that are necessary for DR will utilize the OTech s dark fiber. CalCloud will utilize the existing Wide Area Network (WAN) connections to the OTech s data centers. Existing WAN connections include the California Government Enterprise Network (CGEN), the OTech fiber ring, and the OTech s data center internet feeds. Consumers who require DR are required to have CGEN WAN connections to both data centers at Vacaville and Rancho Cordova. Internet/CGEN OTech External OTech External CalCloud OTech Dark Fiber CalCloud OTech Internal Vacaville OTech Internal Rancho Cordova P a g e 1

5 Section 2: Hardware/Software All hardware to support the CalCloud environment will reside at the OTech s data centers located in Rancho Cordova and Vacaville. The OTech is responsible for providing all of the networking hardware required to set-up and maintain the CalCloud environment, including firewalls and load balancers. 2.1: Storage The CalCloud environment offers the CalCloud customers two options for storage depending on the performance requirements that their workloads demand. The first option is highly available and high performing, with an average read/write response time of 15 milliseconds (ms). This first option of storage will also support at the minimum four levels of Redundant Array of Inexpensive Disks (RAID). RAID Main Features Level 1 Good performance and excellent redundancy. Good performance and redundancy. Most common option; good 5 choice for heavy read databases. 6 Can handle two disk failures Excellent performance and redundancy. Best option for mission 10 critical applications and databases. For less pressing workloads, there is a long-term storage option to be used for archiving purposes, yielding a 100 ms response time. 2.2: Hypervisor The hypervisor is the single most important piece of virtualization and allows the environment to share a pool of resources. The CalCloud environment will align its hypervisor solution with the hypervisor technology currently employed by its customers, in order to leverage standardization and ease of migration. CalCloud will provide its customer an industry-proven hypervisor solution; one being at least 2 years old. The CalCloud hypervisor will provide advanced virtual networking and routing capabilities. Through the hypervisor, the CalCloud environment will be highly available, 99.9%, and allow for infrastructure DR. The hypervisor solution will feature: Feature Clustered service(s) Clustered file system(s) Multi-pathing Automated load balancing Benefit Aides in maintaining service and application availability, while also minimizing downtime for maintenance. Contributes to a highly available environment and increases scalability. Provides failover protection and increases system efficiency. Distributes workloads while optimizing resources, and minimizing response times. P a g e 2

6 2.3: Operating System As part of the CalCloud service offering the customer will have three (3) virtual machine (VM) images available for provisioning. Each VM image will appear as a separate service catalog entry within the CalCloud Portal. The VM image options include: 1. Microsoft Windows Server 2008** 2. Red Hat Linux RHEL 5** 3. AIX*** **more recent version may be available ***specific version to be determined 2.4: System & Operating System Licensing All licensing for the operating system (OS) and the CalCloud system are the responsibility of the contractor. 2.5: Portal Realizing the sensitive data and applications that customers may choose to host within the CalCloud environment, the CalCloud web portal will use two-factor authentication. Additionally, the portal will allow various role-based access levels to ensure that only appropriate personnel can make specific changes. Some of the role-based access levels include: Department User Groups Projects Applications Servers The portal will feature a standardized service menu in conjunction with offering the customer a shopping cart experience. Through the portal, the customer has the opportunity to provision new services and/or modify existing services. The service menu offers four (4) pre-packaged bundles and the option to choose between a Microsoft, RedHat, or AIX OS. After choosing a server size and picking the appropriate OS to reside on the server, the customer is able to subscribe to add-on components, such as Infrastructure DR and backup. The self-service web portal provides customers with the ability to have virtual servers in as little as a couple of hours and through a few clicks of a mouse. Customers can also add more resources (i.e., memory and storage) to their virtual environment(s) through the same portal. The portal provides the customer with the ability to customize their server needs as their business needs change. The first diagram illustrates CalCloud s service model and outlines how a new service is requested through the portal in three simple steps. The chart below the diagram details each of the pre-packaged bundles for the customer to choose between and corresponds to step one, Select your Base Server Size, in the service model diagram. P a g e 3

7 *CCU CalCloud Compute Unit. One (1) CCU is equivalent to 1 GHz of a 2012 or newer Intel Xeon Processor The two main features of the portal are scalability and control; the portal has the functionality to upgrade or downgrade an existing VM, increase or decrease add-ons (RAM, storage, and backup), and stopping existing infrastructure DR Services. The entire portal experience gives the customer more control of their virtual environments. The portal also offers field-level reporting that can be completed on an ad-hoc basis. Reporting capabilities include tracking performance and capacity metrics, as well as billing data. Further, the portal offers a real-time monitoring dashboard for all provisioned servers including details such as up/down status, current CPU, RAM, and storage usage, and total available infrastructure DR. P a g e 4

8 The CalCloud portal will integrate with the OTech s Change Management, Incident Management, and Escalation Management processes. The portal will provide notifications to the customer and obtain approvals from the OTech for provisioning efforts. Additionally, the portal will show the status of any open Incident ticket keeping the customer and OTech heavily involved. P a g e 5

9 Section 3: Architecture The CalCloud environment is highly available, and 100% virtualized. By presenting an instance-based service menu paired with a consumer-facing, self-service web portal, the OTech s customers can provision servers, scale resources, and manage capacity. Customers will share the CalCloud infrastructure, allowing CalCloud to be a multi-tenant environment realized by federal, state, county, education, healthcare and local government entities. Along with the multi-tenancy characteristic that CalCloud supports, there are five (5) essential features that any true cloud environment should exhibit: 1. On-demand self-service 2. Resource pooling 3. Rapid elasticity 4. Measured service 5. Broad network access Each of the five (5) essential features of a cloud environment, along with the multi-tenancy characteristic are dependent upon each other, and in some situations are pre-conditions of one another. For example, resource pooling is a pre-condition of multi-tenancy. The five (5) previously mentioned characteristics demonstrated by CalCloud supports the theory that the whole is greater than the sum of its parts. Initially, the OTech expects the CalCloud environment to utilize two racks at both of the data centers, Rancho Cordova and Vacaville. In anticipation of a growing adoption rate, the OTech has reserved space for up to eight (8) racks to house the hardware at each of the datacenters. Throughout the duration of the contact, as more customers continue to express interest and move their applications to the CalCloud environment can be scaled to support increased levels of capacity. P a g e 6

10 Section 4: Life Cycle The OTech will maintain a business relationship with the awarded contractor for five (5) years, with the option to continue for an additional two (2) years. The first phase of CalCloud is an IaaS service offering. The expected Go Live date for the first phase of CalCloud is January After a successful implementation of the IaaS phase, the OTech plans to use a building block strategy to continue forward with CalCloud s development. Over the next three years (3) years CalCloud is expected to roll out Platform-as-a-Service (PaaS), Application-as-a-Service (AaaS), IT Services Management-as-a-Service and virtual desktop infrastructure (VDI). The below timeline highlights how CalCloud expects to grow over the next several years: P a g e 7

11 Section 5: Incident Management As described through the OTech s Incident Management SIP Release 1: Detailed Process Steps document released January 4, 2013, An Incident is an unplanned service interruption, a reduction in the quality of a service, or a failure of a Configuration Item (CI) that impacts a service. Keeping CalCloud aligned with OTech s business objectives and business processes, any event that causes a disruption or reduction of quality of service will be handled in accordance with the OTech s current Incident Management procedures. The CalCloud portal will show the customer status of their open tickets. By following the industry-proven Incident Management procedures, the amount of time it takes to restore normal service is minimized, as well as any adverse impact on normal business operations. The OTech s main focus is its customer and maintaining a strong business relationship while delivering innovative and cost-effective services. The below diagram shows the workflow for how incidents are handled through their entirety: The OTech service desk and CalCloud customer will work together to determine the appropriate priority for any Incident. In addition, below is a table illustrating the amount of time that the contractor must respond and react to an incident based on the designated priority level (determined by the OTech and CalCloud customer). P a g e 8

12 The below definitions and guidelines are used to determine the priority level: Priority/ Impact Critical/ Widespread High/ Large Medium/ Limited Low/ Localized Definition A major system or virtual network outage, multiple sites or organizations down, or incident marked VIP. A single site or organization down, or significantly degraded. A single user is down, or services degraded, but operational. Minimal impact to services, including a CalCloud customer question or routine request. Response Time Escalation Intervals Assigned Status Pending or Resolved Status 15 minutes 15 minutes 4 hours 30 minutes 30 minutes 4 hours 2 hours 2 hours N/A 4 hours 4 hours N/A Realizing the importance of customer-initiated incidents, the contractor will be held to a high-level of support. While the OTech service desk will be opening any given incident on the customer s behalf, if the problem appears to be contractor related, the contract must respond and resolve incidents in an acceptable amount of time. Depending on the priority level, the response time and resolution time has been calculated to accurately reflect the significance of the incident. The below service level agreements (SLAs) were developed to keep the contractor responsible for any service degradation that they are responsible for. Service Measurement SLA - Definition Target Target Service Credit Amount Incident Response Incident Resolution The percent of time within a calendar month that the Contractor must respond to a CalCloud Consumer incident within the acceptable response, times according to incident priority level. The percent of time within a calendar month that the Contractor must resolve a CalCloud Consumer incident within the acceptable response times according to incident priority level. 95% 95% Critical High Medium Low Critical High $650/ hour $520/ hour $390/ hour $260/ hour $650/ hour $520/ hour P a g e 9

13 Section 6: Maintenance and operations The contractor will be responsible for maintenance and operations for both the hardware and software used to support CalCloud. All maintenance will be performed during a regularly scheduled weekly or monthly maintenance window. A patching schedule will be developed, entailing patching at least once a month for all CalCloud customers. Although the contractor is responsible for such support, the OTech will be involved in all incident and change management, for both planned and unplanned maintenance. It is expected that the customer will receive the highest level of maintenance and support as a CalCloud customer. Any emergency maintenance and patching would require an incident ticket to be opened with the OTech s service desk. P a g e 10

14 Section 7: Security The CalCloud service is built upon industry best-practice security standards. The portal will boost encrypted, two-factor authentication for all CalCloud customers, including: 1. During all remote sessions to the cloud and virtualization management infrastructure, including remote access to all user/management portals. 2. As an available service to CalCloud Consumers for management of their respective virtual datacenter infrastructure including protecting shell or remote desktop protocol (RDP) access to the CalCloud Consumer s VM s. All patching efforts, applying to both virtual and physical infrastructure elements, will comply with the Server Virtualization Security Standard. Following this standard ensures that all virtual servers are hardened, minimizing the risk of security attacks that could infect virtual devices. To further support the Server Virtualization Security Standard, documentation of hardening of elements will take place. The CalCloud system will log all administrative actions that change the configuration state of the infrastructure. The log will detail the following: Action Time Resources involved Individual taking the action State of the environmental change prior to the change State of the environmental change after the change All log streams will be tamper-resistant and will not be altered by the contractor administrators before being passed to the OTech s Security Information Event Management (SIEM)/Syslog function. Encryption at rest for storage services is available for all CalCloud customers. While encryption for data at rest is available through CalCloud, the required encryption keys will be provided initially by the contractor, but will be controlled by the respective CalCloud customer going forward. The CalCloud infrastructure will logically isolate CalCloud customer applications and system zones from other CalCloud customer applications and system zones with no capability of a customer violating that isolation without prior approval. Without proper approval from all parties involved, including the contractor, the OTech, and any affected customer, there is no possibility that logical separation can be violated. This isolation will extend to both the VM as well as allocated storage for those VMs. With the CalCloud infrastructure, CalCloud customers will be allowed to create isolated security tiers within their application and/or system zone. The security tiers available include a web tier, application tier, database tier, etc. Further, the CalCloud customer will manage the communication policy and configuration between the tiers as well as ingress/egress traffic via the CalCloud portal. P a g e 11

15 Section 8: Operational Recovery Disaster preparedness remains a prudent business practice within the IT industry. Realizing so, CalCloud will provide a highly available disaster recovery and backup server environment. By utilizing hypervisor technology, the OTech is able to offer IDR out of the box. The OTech s Vacaville data center will be used as the backup and recovery site. 8.1: Backup All workloads under the CalCloud service must provide backup services. CalCloud offers two tiers of backup services: Backup Tier Feature(s) RPO Target Tier 1 Instant replication < 1 hour Tier 2 Daily incremental and weekly full < 24 hours 8.2: Infrastructure Disaster Recovery The infrastructure for DR will be housed at the Vacaville data center and will utilize the existing OTech dark fiber for network connectivity between the two data centers. Infrastructure DR, similar to backup, is a multi-tiered service offering. CalCloud offers two tiers of DR services: Infrastructure DR Tier Feature(s) RTO Target Tier 1 Active/Active < 1 hour Tier 2 Over-subscribed < 96 hours Note: The over-subscribed infrastructure DR will be done at a ratio of at least 1:5. P a g e 12

16 Section 9: Operational Risk As with every project, there are operational risks that can arise, commonly these risks are caused by a collapse in internal procedures and between people and system. The main way that CalCloud reduces operational risk is through virtualization. Through virtualization, the environment becomes more manageable, with fewer pieces of hardware to operate and maintain. Additionally, human risk is minimized through the CalCloud system by observing the principal of least privilege and separation of duties. For example, administrative staff is given only those privileges absolutely needed, with technical role duties (e.g. hypervisor/infrastructure management, CalCloud Consumer OS management, virtual/physical network management, storage management, etc.) spread across staff to reduce the risk of autonomous damaging access. To make sure that all staff are trained and knowledgeable of the procedures used in cooperation with the CalCloud environment, adequate training will be used throughout the entire life of the contract. The requirement to implement two-factor authentication also minimizes the operational risk, disallowing the wrong personnel from being able to make changes implicating the system. Further, automation is a significant feature of the CalCloud system. Workflows affecting change management are expected to be integrated within the system, ensuring that human intervention is kept a minimum. P a g e 13

17 Section 10: Operating Level Agreements The following operating level agreements, also referred to as the service level agreements (SLAs), are the objectives that the contractor must meet or exceed in order to keep the highest level of service for the OTech s CalCloud customers. These goals are intended to ensure a high quality of service. Any missed objective which negatively affects the CalCloud environment in entirety or any single CalCloud customer will result in service credits and when appropriate, will be passed on directly to the customer. The following SLAs will be in effect for the duration of the business relationship between the OTech and the contracted contractor. Refer to Section 4: Life Cycle for specific length. Throughout the life of the business relationship, the procedures are to be followed 24 hours a day, 7 days a week, 365 days a year, unless explicitly stated. Note: Regular State Business Hours are 8 am 5 pm, Monday through Friday, pacific standard time (PST), excluding California State holidays. 10.1: OTech s Responsibilities The OTech is responsible for opening tickets with the contractor on the CalCloud customer s behalf. In accordance with the OTech s existing incident management, the OTech is responsible for determining the appropriate incident priority level in cooperation with the customer, along with providing necessary information in order to resolve incident. 10.2: Contractor s Responsibilities A contractor-designated service manager must be available to act as the single point of contact for any issues that require escalation in regards to open and ongoing incidents. The contractor is to provide the highest-level of maintenance and support for all contractor-provided hardware and software as specified within Section 2: Hardware/Software. In addition, the contractor must provide service desk support 24/7/365, to mirror OTech s level of commitment to the OTech s customers. The contractor is only responsible for SLAs within the CalCloud environment. Any dependencies outside the CalCloud environment that may impact CalCloud service availability are not the contractor s responsibility : Data Communication Responsibilities There are three (3) main data communication responsibilities that the contractor must deliver to the CalCloud customer: 1. Backups can be performed at any time of the day as long as there is no impact to services or workload. 2. CalCloud customers data shall be preserved for three (3) months after exit transition has been completed. 3. Backup and Archiving has to be performed at either of the OTech s data centers, in Rancho Cordova or Vacaville. P a g e 14

18 10.2.2: Business Continuity Responsibilities Service Backup Measurement Tier 1 Backup Tier 1 RPO Backup Tier 2 Backup Tier 2 RPO Archiving Infrastructure DR RTO SLA - Definition Target The percent of time that Target < % Service Credit $780/ Amount hour data must be replicated to Vacaville.* The maximum tolerable period in which data may be lost. The frequency which backup data is to be replicated to Vacaville before the next backup occurs. The maximum tolerable period in which data may be lost. The maximum time allowed before beginning Archive. The amount of time the Contractor must provide DR Recovery. Continuous; < 1 hour 24 hours 24 hours (Full/Incremental/or Snapshot) 72 hours IDR Tier 1< 1 hour IDR Tier 2< 96 hours** Note: This time includes overprovisioning/oversubscription (time to provide additional dedicated infrastructure hardware) $780/ hour $390/ hour $390/ day $390/ hour $780/ hour $390/ hour * Not to be impacted by uptime of server/app. ** First come, first serve basis. P a g e 15

19 10.2.3: Availability Responsibilities Service Measurement SLA - Definition Target Target Service Credit Amount Portal Availability Service Availability Server Provisioning Availability The percent of time within a calendar month that the Portal is available to end users. The percent of time within a calendar month that the portal infrastructure is available (to and from) for use by the CalCloud Consumer. The maximum amount of time any quantity of servers 20 must be provisioned once requested via the Portal. 99.5% $390/ hour 99.9% $780/ hour < 4 hours $390/ hour Storage Provisioning For quantities 21 and over, the SLA Target time increases by four (4) hours for every additional 20 servers. The maximum amount of time any amount of storage 500 TB must be provisioned once requested via the Portal. < 2 hours $390/ hour Memory Provisioning Availability IDR Availability Backup / Restore Availability Archiving Availability For quantities 501 TB and over, the SLA Target time increases by two (2) hours for every 500 TB of SAN. The maximum amount of time any authorized amount of memory must be provisioned once requested via the Portal. The maximum amount of time Infrastructure DR must be available as a service once requested via the Portal. (not to complete DR) The maximum amount of time backup/restore capability must be available as a service once requested via Portal. (not to complete backup) The maximum amount of time archiving capability must be available as a service once requested via Portal. < 2 hours < 2 hours < 2 hours $520/ hour $520/ hour $520/ hour < 3 day $390/ day P a g e 16

20 10.2.4: Performance & Monitoring Service Measurement Root-cause Analysis SLA - Definition Target Target Service Credit Amount The maximum number of days from 14 days $260/ day incident resolution which root-cause analysis shall be completed. 8 hours $1,040/ hour Cumulative Outage The maximum number of hours within a 30 day rolling period that the service is unavailable. Excessive Outage Service Uptime (everyone, single or multidepartments/apps) Server / Application Uptime Storage Uptime Monitoring Uptime Security Breach Notification Patch Updates The maximum number of outages occurring within one calendar month. The percent of time within a calendar month that the entire infrastructure is operational. The percent of time within a calendar month that any server is operational. The percent of time within a calendar month that the read/write operations should meet optimal standards*. The percent of time within a calendar month that the infrastructure is to be monitored. The maximum numbers of hours allowed between a suspected or confirmed breach and notification from Contractor to State. The number of days from release which patches should be applied or rolled through environments. 4 Outages per Month $5,200/ outage 99.95% 8hours $1,820/ hour >8hours $5,200/ hour 99.95% 8hours $650/ hour >8hours $1,300/ hour % $780/ hour 99.5% $650/ hour 4 hours $650/ hour 30 days $650/ day *Storage read/write performance will operate at optimal standards 15 milliseconds P a g e 17

21 There are specific durations, which any compliance issue must be resolved. If any issue arises, once the contractor is made aware they are responsible for remediating the problem within the respective time frames : Billing Priority Remediate Period Service Credit Amount Critical 3 months $780 per finding per week Important 6 months $520 per finding per week All Others 12 months $390 per finding per week Service Measurement Service Credits Monthly Bill SLA - Definition Target Target Service Credit Amount The maximum number of days, in which service credits shall be made available to the State. The maximum number of days from the end of a billing cycle/calendar month that the bill needs to be received by State. 30 days [1/8 * credit amount] / [day] 3 days $260/ day : Reporting Reporting Interval Delivery Method Service Credit Amount Weekly Electronically delivered $520 / late occurrence Monthly Electronically delivered $1,040 / late occurrence P a g e 18

22 Section 11: Technical Plan Approval Name Title Project Approver Signature Date Name Title Project Sponsor/Lead Signature Date Name Title Service Manager Signature Date P a g e 19