CUNY Graduate Center Information Technology IT Provisioning for Business Continuity & Disaster Recovery Effective Date: November 14, 2018

Transcription

1 CUNY Graduate Center Information Technology IT for & Effective Date: November 14, 2018 Introduction Organization Information Technology (IT) is the division of the Graduate Center responsible for voice, video and data systems and services. The mission of this unit is to promote, facilitate and support the effective use of technology in instruction and learning, in research, and in processing and accessing institutional information. Organizationally, IT is comprised of three divisions: Administrative Services, Client Services and Systems Services. Facilities The Graduate Center s primary site at 365 Fifth Avenue ( GC/Fifth ) includes data centers located on the second floor directly connected to the CUNY ring. The secondary site, at the GC Advanced Science Research Center ( ), includes a main data center at ground level and as well as small data center on the 5 th floor and is directly connected to the CUNY ring using infrastructure currently overseen by CUNY central office CIS. Scope of This document outlines the provisioning currently in place by Information Technology to safeguard ongoing functionality for select IT systems and services specifically identified herein. The scope and suitability of such provisioning is reviewed on a regular basis as systems and services are decommissioned, added and changed. Definitions We distinguish between the phrases business continuity and disaster recovery by viewing a business continuity plan as essentially a proactive approach to safeguarding ongoing daily operations while a disaster recovery plan must attempt to react to the scope and nature of a calamity. That is, our business continuity provisioning guarantees that we have systems in place that are backed up and fail over, ensuring that key services stay up and running, that business processes remain operational, in the course of minor disruptions which are reasonably anticipated. recovery, in which an unlikely but catastrophic incident has rendered the GC/Fifth data center or the main data center lost in part or in total for an extended period of time, may call for wholesale reconstitution of facilities, resources and services depending on the nature and specifics of the disaster. Context For the purposes of this document, IT services are considered to be centered at GC/Fifth, with business continuity provisioning intended to safeguard that perspective and disaster scenarios envisioned deleteriously impacting that location. IT services based at are not addressed in this document. Related Policies & Procedures IT Backup and Restore Policies IT Incident Management Plan 1

2 Essential IT Services and BC/DR This section outlines the provisioning currently in place by Information Technology to safeguard ongoing functionality for select essential IT systems and services. Two types of provisioning are identified, business continuity and disaster recovery, for the essential IT services identified below. GC Data Center: The data center at GC/Fifth consists of one dedicated room. Access to the GC/Fifth data center is via restricted card-entry system, security cameras monitor the entrances and public safety routinely patrols the adjacent hallways. o The data center is on the second floor, above ground level, and set apart from the general traffic patterns used by the community of individuals occupying the building on a daily basis. o Supported by individual redundant cooling systems, with continuous monitoring and alerting in place for Facilities and Engineering. o Supported by individual power systems provisioned with UPS backup, with continuous monitoring and alerting in place for Facilities and Engineering. o Supported by individual sprinkler systems. o Supported by temperature, humidity and water alerting, monitored by IT staff and Facilities and Engineering. Access to the main data center is via restricted card-entry system and public safety routinely patrols the adjacent hallways. o The data center is on the ground level, and set apart from the general traffic patterns used by the community of individuals occupying the building on a daily basis. o Supported by individual redundant cooling systems, with continuous monitoring and alerting in place for Facilities and Engineering. o Supported by individual power systems provisioned with UPS backup, with continuous monitoring and alerting in place for Facilities and Engineering. o Supported by individual sprinkler systems. o Supported by temperature, humidity and water alerting, monitored by IT staff and Facilities and Engineering. Subject to the nature of the disaster, facilities at or GC/Fifth may be used as emergency relocation centers for restoration of targeted GC IT services. 2

3 GC IT Network Infrastructure: The network infrastructure at GC/Fifth and consists of core switches and related componentry in the data center, connected via fiber risers to multiple IDFs on each floor housing edge switches serving end-user devices. A wireless network infrastructure rides on top of this framework. GC/Fifth In the data center, core switches, distribution switches, firewalls, server-region switches and switches connecting the internal network to the CUNY ring are all deployed in pairs, with failover provisioning, providing high redundancy. From the data center, the IDF on each floor is supported by redundant fiber connections, however these are encased in the same pathway. The stack of edge switches in each IDF on each floor are configured for failover, however there are single horizontal paths from the edge switch to the individual enduser wall ports. There typically are multiple data ports in any given room. The CUNY ring provides two paths, in opposite directions, for redundancy. There is a single path from the data center to the external connection to the actual ring. There is no secondary internet connection in place at GC/Fifth. In the data center, core switches, distribution switches, firewalls, server-region switches and switches connecting the internal network to the CUNY ring are all deployed in pairs, with failover provisioning, providing high redundancy. From the data center, the IDF on each floor is supported by redundant fiber connections, however these are encased in the same pathway. The stack of edge switches in each IDF on each floor are configured for failover, however there are single horizontal paths from the edge switch to the individual enduser wall ports. There typically are multiple data ports in any given room. The CUNY ring provides two paths, in opposite directions, for redundancy. There is a single path from the data center to the external connection to the actual ring. There is no secondary internet connection in place at GC/Fifth. Subject to the nature of the disaster, facilities at and GC/Fifth may be used as emergency relocation centers for restoration of targeted GC IT services. GC domain (faculty & staff) for GC staff and faculty uses the gc.cuny.edu domain, and is implemented in a Microsoft Exchange environment, hosted locally at GC/Fifth. in the legacy asrc.cuny.edu domain is hosted on the CUNY central office CIS platform. gc.cuny.edu domain (faculty & staff) The Exchange platform is maintained in a dedicated cluster environment housed in the GC/Fifth data center, made up of multiple fault-tolerant servers, provisioned for fail-over redundancy. 3

4 GC domain (faculty & staff) for GC staff and faculty uses the gc.cuny.edu domain, and is implemented in a Microsoft Exchange environment, hosted locally at GC/Fifth. in the legacy asrc.cuny.edu domain is hosted on the CUNY central office CIS platform. ProofPoint and other supplementary systems apply additional security processing to incoming/outgoing for the purpose of safeguarding operations; these systems are likewise configured as a cluster of multiple faulttolerant servers, provisioned for fail-over redundancy. Backups are executed daily, stored onsite; backups are copied weekly to asrc.cuny.edu domain This is currently the purview of CUNY central office CIS accounts for ASRC-based employees are now provisioned in both the GC and the asrc.cuny.edu domain. gc.cuny.edu domain Notwithstanding weekly Exchange backup copies at, it will take time to operate gc.cuny.edu from the ASRC: o Appropriate servers will need to be setup o Mailboxes will need to be restored from backup o DNS changes will need to be made o Given the possibility of migrating to a CUNY cloud-hosted solution, dedicating resources to DR does not seem prudent asrc.cuny.edu domain This is currently the purview of CUNY central office CIS Externally hosted and managed IT services: Services such as , file storage and collaboration services for GC students (using the gradcenter.cuny.edu domain), CUNYfirst HR and finance services, Blackboard and NetCommunity are externally hosted. GC student , file storage and collaboration services (using the gradcenter.cuny.edu domain) are implemented in a Microsoft Office 365 environment, hosted externally by Microsoft and controlled by CUNY central office CIS. This is a Microsoft-hosted and supported environment, accessible to users from any internet-connected location. CUNYfirst and Blackboard are hosted systems overseen and managed by central office CIS. NetCommunity is a system contracted for by the GC Development office and hosted externally by the vendor. 4

5 Externally hosted and managed IT services: Services such as , file storage and collaboration services for GC students (using the gradcenter.cuny.edu domain), CUNYfirst HR and finance services, Blackboard and NetCommunity are externally hosted. CUNY OneDrive storage hosted externally by Microsoft and controlled by CUNY central office CIS. This is a Microsoft-hosted and supported environment, accessible to users from any internet-connected location. Subject to the nature of the disaster, these externally-hosted and supported environments are expected to remain operational and accessible to users from any internet-connected location. CUNYfirst, OneDrive, and Blackboard are the purview of CUNY central. GC database services: Electronic databases underpinning file systems and applications are hosted locally at the GC. GC/Fifth The MS SQL database environment is maintained in a dedicated cluster environment housed in the GC/Fifth data center, made up of two fault-tolerant servers, provisioned for fail-over redundancy. It is backed up daily and then copied to. In addition, transaction logs are backed up every 4 hours and until a 3 rd standby/passive cluster node is created at the ASRC (see below) transaction log backups are then copied to. Although the mariadb database environment at GC/Fifth is not clustered, it is backed up daily by 2 independent backups and these 2 different backups are copied daily to. In a disaster scenario until a 3 rd standby/passive cluster node is created at the with live updates from GC/Fifth-- a MS SQL VM will be restored from latest Veeam backup and latest transaction logs In a disaster scenario until a standby/passive VM is created at the with live updates from GC/Fifth -- the mariadb VM will be restored from latest Veeam backup. Future planning: Standby/Passive MS SQL and mariadb cluster nodes will be created at the with real-time updates. If a management priority, we can try to test failover/failback to/from the MS SQL database server on January 2018 Shutdown Day and perform a similar test with the mariadb database server on July 2019 Shutdown Day. 5

6 GC file services: Electronic file services for GC faculty and staff, commonly known as S: and R: drives, are hosted locally at GC/Fifth. (File services for GC students are discussed above.) GC/Fifth The system for file access is maintained in a dedicated cluster environment housed in the GC/Fifth data center, made up of multiple fault-tolerant virtual servers, provisioned for fail-over redundancy. The back-end system for file storage is a SAN system housed in the GC/Fifth data center. The file server cluster is backed up daily and copied weekly to. In addition, the S: drive and R: drive but not 4 large archive folders! are synchronized daily to a standby file server at The back-end system for file storage is a SAN system housed in the data center. The file server cluster is backed up daily and copied weekly to. Ongoing preparedness: Servers at GC /Fifth are patched and maintained on the same schedule as those at. In a disaster scenario, GPOs that map S: and R: will be changed to use the replica file server at. If a management priority, (and if GC workstations are granted access to the standby server at the ASRC), we can test failover to the ASRC on a future Shutdown Day. Incoming/outgoing internet connectivity: The IT infrastructure ( CUNY ring ) traversed by incoming and outgoing traffic between the internet and the IT infrastructure internal to GC/Fifth, GC/Apt and is controlled by CUNY central office CIS. GC/Fifth is connected to the CUNY ring. is connected to the CUNY ring. The CUNY ring is the purview of CUNY central office CIS. The CUNY ring is the purview of CUNY central office CIS. 6

7 GC-hosted Windows-based websites & web-based services: Resources such as the primary GC website (gc.cuny.edu), Password Reset, Track-IT, and the web front-ends are hosted locally at GC/Fifth. The asrc.cuny.edu website is hosted by CUNY central office CIS. The front-end systems are maintained in a dedicated cluster environment housed in the GC/Fifth data center, made up of multiple fault-tolerant virtual servers, provisioned for fail-over redundancy. The back-end databases are MS SQL and mariadb, discussed elsewhere in this document. If content of the front-end web server changes frequently, the server is backed up daily onsite and there is a daily offsite backup copy to GC\ASRC. If content is static, the server is backed up weekly and there is a weekly offsite backup copy to GC\ASRC. asrc.cuny.edu website This is currently the purview of CUNY central office CIS. Content residing on the CIS platform will be migrated to the GC s primary infrastructure by the end of gc.cuny.edu website Ongoing preparedness: Servers at GC/Fifth are patched and maintained monthly. In a disaster scenario, if passive/standby MS SQL and mariadb servers do not yet exist at GC\ASRC, first MS SQL and mariadb database servers will be restored from Veeam backup and second critical web servers will be restored from Veeam backup. Next, new IPs will be assigned and DNS entries updated. In the future, we will test Global Load Balancing to reduce the time to failover critical web servers to GC\ASRC asrc.cuny.edu website This is currently the purview of CUNY central office CIS. In the future, content residing on the CIS platform will be migrated to the GC s primary infrastructure by the end of GC-hosted Windows-based IT services: Resources such as WSUS, SCCM, EPO, software license server, print server, and Active Directory/DNS are hosted locally at GC/Fifth. These systems are housed in the GC/Fifth data center. There are multiple Domain Controllers and DNS servers for redundancy/high-availability. The other servers in this category do not have redundancy/high-availability. 7

8 GC-hosted Windows-based IT services: Resources such as WSUS, SCCM, EPO, software license server, print server, and Active Directory/DNS are hosted locally at GC/Fifth. If content of the server changes frequently, the server is backed up daily onsite and there is a daily offsite backup copy to GC\ASRC. If content is static, the server is backed up weekly and there is a weekly offsite backup copy to GC\ASRC. GC Domain Controller/DNS servers replicate to Domain Controller/DNS servers located at the ASRC. Except for the SCCM image server, the WSUS, SCCM (front-end), antivirus EPO and software license server are backed up daily with an offsite backup copy to GC\ASRC Although 3 GC\Fifth print servers are backed up, the backups are not copied offsite to GC\ASRC because GC\ASRC has its own printers and print server. GC-hosted Linux-based websites & web-based services: Resources such as GC Web Services, the CUNY Academic Commons, and the GC Library website, as well as the NML, DSC, MLD, CUNY BA, RILM, Brook Center and RedMine websites are hosted locally at GC/Fifth. The GC Linux environment is purely virtual and consists of three layers: the MySQL database back-end, discussed elsewhere in this document, the file system and the front-end web services layer. The MySQL database back-end is discussed elsewhere in this document. The environment is maintained in a dedicated cluster environment housed in the GC/Fifth data center, made up of multiple fault-tolerant virtual servers, provisioned for fail-over redundancy. For critical, production linux-based servers, there are two independent, daily backups and the backups are copied daily to GC\ASRC. In a disaster scenario, critical VMs can be restored from Veeam backup at GC\ASRC, IPs can be changed and DNS modified. Certain existing supplementary systems may be considered non-essential and will not be restored. See above discussion of mariadb. 8

9 Desktop Computers: A collection of Apple ios Mac and Windows PC desktop computers are available at GC/Fifth to support faculty, staff and students. These resources are maintained and supported on an ongoing basis. A collection of Apple ios Macs and Windows PCs are available at to support faculty and staff. GC/Fifth For desktop computers currently deployed, systems are kept current and protected by way of central management, via SCCM for Windows PCs and Casper for Apple Macs. SCCM is noted elsewhere in this document. Casper resides on a physical server, with a redundant hardware platform. WSUS, noted elsewhere this document, is used to keep PC computers current with Windows updates and patches as well as Microsoft application software updates. Casper is used to keep Macs current with Apple ios updates and patches. Ongoing preparedness: Critical updates issued by Microsoft are automatically pushed to all Windows desktop computers via WSUS. Critical updates issued by Apple are automatically pushed to all Mac desktop computers via Casper. McAfee EPO, noted elsewhere in this document, is used to keep the end-point security updated on both Windows PCs and Apple Macs. This includes antivirus, full-drive encryption for portable storage and data loss prevention for designated sensitive data (for the latter, Windows only). The same utility is used to provision full-drive encryption on GC laptops. Ongoing preparedness: McAfee updates are automatically pushed to desktop computers on a routine basis. Ongoing preparedness: A small collection of PCs and Macs are retained in stock ( spares ), as a redundant fail-safe precaution should a desktop computer currently deployed fail beyond immediate repair. Standard images for PCs and Macs are maintained in SCCM for Windows and in Casper for Macs. Ongoing preparedness: These images are backed up routinely. Desktop computers at will be integrated into the fleet of desktop computers at GC/Fifth. In a disaster scenario, certain existing supplementary systems may be considered non-essential. Subject to the nature of the disaster, facilities at will be used as emergency relocation centers for staff workstations. Existing desktop computers at are suitable for such work. Future planning: DR capability to serve GC members operating at GC\ASRC will be enhanced by migrating ASRC MACs and PCs (from stand-alone and ASRC.cuny.adlan domain) to the GC domain. 9

10 GC telephone service: Telephony service for GC/Fifth is provided via a voice-over-ip system hosted locally at GC/Fifth, using circuits to Verizon and AT&T. Telephony service for GC/Apt is provided via Verizon. Telephony service for uses infrastructure currently controlled by CUNY central office CIS. GC/Fifth The telephony system is comprised of a collection of components; all components are provisioned for redundancy. Additional end-user hand-sets are also stored on-site. Voic services enabling a caller to leave a message are maintained in a dedicated cluster environment housed in the GC/Fifth data center, made up of multiple fault-tolerant servers, provisioned for fail-over redundancy. Voic services enabling the recipient to retrieve stored messages are not similarly provisioned. Backups are executed daily, stored onsite. backups were successful. Redundant routes exist to the PSTN, however these traverse the same pathway between GC/Fifth and the street connection. This is currently the purview of CUNY central office CIS. GC/Fifth A small number of telephones at GC/Fifth are provisioned independently of the IP network using traditional analog service. Telephone services at GC/Apt and are independent of GC/Fifth and therefore, subject to the nature of the disaster, may remain operational. In a disaster scenario, certain existing supplementary systems such as voic may be considered non-essential. Subject to the nature of the disaster, facilities at and GC/Apt may be used as emergency relocation centers for staff workstations. This is currently the purview of CUNY central office CIS. 10