University Information Systems. Administrative Computing Services. Contingency Plan. Overview

Transcription

1 University Information Systems Administrative Computing Services Contingency Plan Overview Last updated 01/11/2005

2 University Information Systems Administrative Computing Services Contingency Plan Overview Table of Contents i. Introduction...1 ii. iii. UIS Recovery Strategy...2 UIS Backup Strategy...3 iv. UIS Recovery Scope...5 v. UIS Contingency Planning Manual...6 vi. vii. UIS Recovery Organization...8 User Responsibilities...10 viii. Disaster at the User s Facilities...11 ix. Hypothetical Disaster Recovery Examples a. Initial Impact Assessment and Response...12 b. Hot Site Recovery Example...13 ii

3 i. Introduction The purpose of this document is to provide departments with important information regarding the University Information Systems (UIS) Contingency Plan. The overview is geared toward providing University departments an understanding of what happens if the UIS Computer Center was severely disabled. Computer users of the computer center should become familiar with this document. Users will find that the overview can be of value helping to define their own contingency planning procedures. This overview should provide sufficient information to enable departments to answer the following key questions: In the event of a recovery at the SunGard Hot site, what services and business applications will be recovered? When will these applications be recovered and what do computer users need to do? For recovery purposes only, how long should a department retain source documents used as mainframe input? Who is responsible for the recovery of departmental systems that have been severely disabled or destroyed? Departments that have any questions regarding the UIS Contingency Plan may contact the Director of Administrative Computing Services or the Assistant Director of Computer Services. 1

4 ii. UIS Recovery Strategy The current UIS Contingency Plan is based on the activation of a 'hot site' which will allow the University to re establish its administrative data processing environment in the event of an extended loss of services due to major damage at the UIS Computer Center. The University currently contracts with SunGard Availability Services for hot site recovery. The University's current recovery objective is to have a fully functional mainframe supporting production batch, on line, and WEB services (as outlined in the UIS Recovery Scope section) at the recovery center within forty eight (48) hours from the point in time when the disaster declaration is made by the University. A hot site is a functioning computer room which has been prepared with all necessary hardware including an operating mainframe, disk and tape storage equipment, communications equipment and printers. The only major component missing at the hot site is the University s data. The UIS Contingency Plan is tested twice a year by performing a full recovery of all production online and batch services (as noted in the UIS Recovery Scope section). By contract, the University can operate out of the hot site for up to a six week period. If services back at the University cannot be restored within this time frame, UIS will need to move its hot site to a 'cold site'. A cold site is a computer ready physical room with all appropriate environmental resources such as electrical power and air conditioning. A cold site does not contain any computer equipment. The University would need to acquire and move a new mainframe and all peripheral equipment into this site prior to the expiration of the six week period at the hot site. Processing at the cold site is available up to a six month period. 2

5 iii. UIS Backup Strategy Currently, the Administrative Computer Center executes a series of data backups, both full and incremental, on a daily and weekly basis. A full back up represents a complete copy of all data stored on our disk storage equipment. On a daily basis, all data that has changed since its last back up is also backed up. This process is defined as incremental back ups. In addition, ADABAS, our database management system, generates data images of records before and after database updating. The actual creation of backup files for the ADABAS images, known as protection logs, is based on transaction volume and pre determined times. All backup files, including ADABAS protection logs, are written to two separate automated tape libraries. One tape copy is created and stored at the UIS Computer Center and a second copy is created and stored at our electronic vault which is located in a different building. Local and remote back up copies are created automatically and concurrently to ensure readability and to facilitate recovery. In the event of an activation of our SunGard hot site, the data from the electronic vault site would be physically shipped to SunGard and restored on the recovery mainframe platform. The transportation of the recovery data to SunGard, along with the data restoration and technical configuration process, is time consuming and is the basis of the estimated recovery schedule. In order to minimize the disruption a catastrophic incident can cause, UIS has installed an additional safeguard known as the checkpoint back up process. As stated above, the ADABAS protection logs are generated on the basis of transaction volume and pre determined times. Therefore, the creation of back up files from the ADABAS logs is not predictable. To increase the confidence that a severe failure at the computer center does not result in a major timeconsuming user recovery, the checkpoint back up process was developed and implemented. This process executes twice a day, seven days a week, at 8:00 AM and again at 4:30 PM. The checkpoint back up process forces the generation of the ADABAS protection log files to be backed up. The benefit of this additional protection minimizes user recovery. For example, if a catastrophic incident at the center occurs at 5:00 P.M. on Tuesday, the expected disruption would only be back to 4:30 PM. Users would be required to re enter all data from 4:30 PM to 5:00 PM. Without this change to the standard approach, user recovery would have required the re execution of all batch processing and re entry of all on line transactions from the last protection log back up or even as far back as the previous evening s incremental back ups. UIS mainframe users should base their internal recovery procedures on the two forced ADABAS protection logs which are backed up at 8:00 AM and 4:30 PM daily. To be on the safe side, it is recommended that source documents used as input for mainframe processing which do not have any other retention requirements other than for recovery, should be retained for twenty four (24) hours. As an important side issue, it is highly recommended that the source documents that are retained for recovery purposes are stored in a protective environment overnight. This is important to be able to recover the source documents in the event that a catastrophic event occurs that damages both the UIS Computer Center and the affected University department. 3

6 iii. UIS Backup Strategy (continued) In addition, the data stored on the two automated tape libraries is refreshed via a daily process that ensures the integrity of the physical tape and that the data written on any tape is readable. Since any physical data storage media (tapes, floppy disks, etc.) is vulnerable to defects, the refresh process constantly moves and writes data from one tape volume to another to ensure that any such defects are found in a timely manner. Keep in mind, if the tape media containing a back up file is detected as being damaged during a refresh process, the non damaged copy, either the remote or local tape media, is used as input to recreate the second copy. The refresh process is very important to ensure that in the event that no back up files survive a disaster at the UIS Computer Center, all back up files located at the vault are able to be read and are able to provide for a successful restoration of the University s data at the SunGard Recovery Center. 4

7 iv. UIS Recovery Scope The following represents the planned restoration schedule for services and applications in the event the UIS Computer Center is severely disabled. The recovery plan is based on technology available at the SunGard Recovery Center and availability of resources from external service providers. 1. Services to be recovered in the first 48 hours: All Production on line processes (except those noted in items 2 and 3 below) All Development on line processes All Production batch processes All LINK and WEB access applications Campus network connectivity to the University Information Systems mainframe at SunGard 2. Services to be recovered from 48 hours to six weeks: Dedicated links for business functions with Data Exchange Partners (Banks, Credit Card Clearinghouses) may be restored if dictated by the end user s contingency plans 3. Services to be recovered at the cold site: AF/Remote Automated Problem Notification System Recovery of distributed systems platforms Degree Audit, Universal ID Card, Online fax requests, SEVIS, Touchtone Telephone System, EOS Thin Client (2005), and Host On Demand (2005) 5

8 v. UIS Contingency Planning Manual University Information Systems has created a set of documents instruct personnel on actions during and after a disaster or emergency. The UIS Contingency Planning Manual is comprised of the following documents: BU Charles River Campus UIS Emergency Procedures UIS Disaster Response Plan Administrative Computing Services Recovery Procedures SunGard Recovery Procedures The objectives of these documents are as follows: Provide for orderly and timely recovery from an interruption of data processing services Reduce confusion during any chaotic period by having a clearly defined course of action Specify steps necessary to relocate the Computer Center to an alternate site Identify those systems which require priority scheduling Identify offsite locations where backup files, software, and documentation are stored Identify equipment configurations and procedures for repair or replacement during a disaster Establish the personnel responsible for Contingency Planning Identify all vendors and service providers for critical products and services Identify all software components and the supporting group, organization or company Identify all hardware components and the supporting group, organization, or company Plans to be used for the recovery of the UIS mainframe system and the establishment of communication methods at the SunGard hot site location 6

9 v. UIS Contingency Planning Manual (continued) The UIS Contingency Planning Coordinator is responsible for ensuring that the plan is maintained and is current. In order to accomplish this, the Coordinator must ensure that the procedures and requirements outlined in the Contingency Plan are adhered to in accordance with the instructions of the plan. Semi annually, the Contingency Planning Coordinator reviews the entire Contingency Plan suite of documents and solicits updates from the University Information Systems community. Significant updates warrant a meeting of all team leaders to review changes. Reviews and tests of certain segments of the plan will be conducted throughout the calendar year. The Recovery Coordinator distributes a hard copy of the suite of documents to all ACS personnel after each update The documents are available to UIS staff online and are located in the ACS section within the University s WEB page. The hard copy is needed as personal computers may not be available in a disaster situation. Future plans for distributing the suite of documents involve creating a CD for distribution to all ACS personnel in addition to the hard copy. 7

10 vi. UIS Recovery Organization The UIS Contingency Planning Recovery Organization is made up of a hierarchy of teams. The individual recovery teams and roles are as follows: Disaster Management Team / Administrative Services Team The Disaster Management Team provides the initial activation authorization for the hot site. The team manages the overall direction of the University Information Systems recovery operations and restoration or relocation of the computing facility. The team provides a central focal point for all communications between recovery teams, vendors, internal service organization such as Buildings & Grounds and Information Technology, user departments and senior University management. The Administrative Services Team will provide overall administrative coordination of the disaster recovery effort and will serve as expediters of the administrative functions. Impact Assessment Team The Impact Assessment Team performs the initial assessment of damages to the Computer Center and equipment. They will provide a recovery recommendation to the Disaster Management Team. Applications Support Team The Applications Support Team provides the required production support between the end users, the Disaster Management Team and the hot site facility. User departments will receive updates on the status of the recovery from this team. Hardware Support Team The Hardware Support Team coordinates the procurement of replacement or substitution hardware for the Computer Center and the rebuilding of the original site. This team coordinates with SunGard and the hot site Recovery Team both primary and backup communications restoration with particular attention to restoring communications as soon as possible for the critical applications. The team will also coordinate the recovery of the offsite storage materials needed for hot site recovery. To facilitate 3270 access from the campus to SunGard, the University has contracted with SunGard to provide 100 personal computers available on a Quick Ship basis. These personal computers would be shipped to a location designated by the University within 48 hours of being requested. 8

11 vi. UIS Recovery Organization (continued) Hot Site Recovery Team The SunGard Hot site Recovery Team coordinates and manages the recovery of the SunGard hot site. This would include coordinating the delivery of the backup tapes, overseeing SunGard staff in the implementation of recovery procedures to restore normal production processing at the hot site, and verifying the establishment of the emergency communication links to the University's Boston campus. The team also performs any tasks required at the hot site facility until relieved, or until the original site has been restored to service Software Support Team The Software Support Team provides support services for the restoration of operating software and production data files at the hot site facility. This team would also be responsible for the restoration of operating software and production files upon return of the Data Center to the University. Computer Center Restoration Team / Buildings & Grounds Support Team This team provides for the reconstruction of the damaged UIS facility. Salvage and cleanup operations will also be coordinated by this team. 9

12 vii. User Responsibilities In the case of a catastrophic event that impacts the UIS Computer Center resulting in the activation of the hot site facility, certain emergency procedures need to take place at the departmental level. The following represents the user s role during a disaster recovery process: 1. Await notification from the Applications Support team: The Applications Support Team will notify all users that a disaster has occurred and keep end user groups informed as to the extent of the disaster and the progress of the recovery at the hot site. 2. Coordinate the replacement of special forms stock: The recovery staff will inform the departments of the loss of special forms stock stored in the Computer Center and will assist in the delivery of critical stock, such as checks, to the hot site. Arrangements will be made for deliveries of critical printed output from the hot site to the Boston campus. A minimum of a twelve (12) to twenty four (24) hour delay in receiving reports should be anticipated. Important: It should be noted that the use of the existing on line electronic reporting system, WSF2/ERD, becomes instrumental during this crisis period. On line viewing of reports will be available immediately upon completion of report job executions through ERD. In the near future, the web based product, EOS Thin Client will allow users to access their reports via any PC or laptop computer with access to the internet. EOS Thin Client is expected to be recoverable at a hot site in Retrieve all source documents that might be required: Some source documents might need to be re entered in to the system once it has been recovered at the hot site (see the section on Backup Strategy). User departments should have the last twenty four (24) hours worth of source documents retrieved from where they have been stored. 4. Coordinate the delivery of any outside agency tapes: All tapes that are normally created and sent out for processing or tapes that are brought in the computer center from outside agencies must have the deliveries re directed to the hot site facility. 5. Determine the quantity and location for the distribution of the Quick Ship personal computers: Departments will need to determine the number of 3270 mainframe connections required to perform business functions during a declared disaster and hot site activation. A limited number of personal computers will be delivered to the University by SunGard within forty eight hours. End users should provide a report on current critical business cycles should be communicated to the Applications Support Team and subsequently to the Disaster Management Team. The DMT and management will determine the distribution and installation based on the current critical business cycles. 10

13 vii. User Responsibilities (continued) 6. Perform data verification of the recovered hot site system: Once the connection to the recovered UIS mainframe has been established and the quick ship personal computers delivered, the Hardware Support Team will proceed to install the equipment and provide instructions on its use. After this has been accomplished, departmental staff will need to connect to the hot site and verify that the recovery has been successful by checking the appropriate records and files. 7. Notify their customers and outside agencies: The departments will need to notify their customers (students, staff, etc.) and outside agencies, such as banks, of the service outage and what services are being provided during and the estimated duration of the emergency. viii. Disaster at the User s Facilities The user department is responsible for the following recovery actions if a catastrophic event damages a user s department which disables or destroys the user s departmental computerized systems. Retention time of source documents used as mainframe input Replacement of damaged departmental system equipment Retrieval of departmental system back up files Restoration of the departmental systems Re entry of all data not restored While the scope of UIS Disaster Planning does not include the restoration of departmental facilities, UIS will make every effort to be of assistance to the end users during this time of need. 11

14 ix. Hypothetical Disaster Recovery Examples a. Initial Impact Assessment and Response The following is a hypothetical timeline for initial action during a possible disaster situation as defined by current UIS plans and guidelines. Time: 0 hour At the first indication of any total service outage UIS will initiate standard recovery and problem escalation procedures and attempt to resolve the problem at the operations level. The UIS Computer Center staff will escalate the problem to the appropriate management levels. Computer technical staff will attempt to expedite the problem resolution. Time: 0 24 th hour (concurrent with other actions) If the problem exhausts normal recovery/escalation procedures by the 24th hour or if damage to the facility/equipment appears severe any time prior to the 24th hour, the staff member or manager who is coordinating the problem resolution should notify the leader/alternate of the Impact Assessment Team. User departments are kept informed of events by the Application Support Team during this time period. Time: 1 st hour Upon receiving notification of an incident, the Impact Assessment Team Leader will notify the Impact Assessment Team that a major problem has occurred at the Computer Center and advise the team to precede onsite. Time: 2 nd hour The Impact Assessment Team will arrive and evaluate the situation. The Impact Assessment Team leader will contact the Disaster Management Team leader with an indication of the disaster category after an initial damage assessment has been performed. The Impact Assessment Team will continue with the preparation of a detailed Impact Analysis Report to be presented to the Disaster Management Team upon their arrival at the pre defined Control Center. The Contingency Planning Coordinator will notify the leaders of the Software Support Team, Hardware Support Team, Hot site Recovery Team, and Application Support Team that there has been an incident involving the computer center that may require a recovery process of declaration of a disaster. Each team leader will then be on call with the understanding that their team may be immediately activated pending management declaration of a disaster. The team leader would notify the members of his/her team of the situation. 12

15 ix. Hypothetical Disaster Recovery Examples (continued) a. Initial Impact Assessment and Response (continued) Time: 2 nd hour (continued) Upon notification, the Disaster Management Team leader will notify senior University Management of the potential disaster situation and communicate the initial Impact Assessment Team findings. Time: 3 rd hour All team leaders will assemble at the designated control center. Time: 4 th hour The Disaster Management Team leader will conduct a briefing of team leaders after which each team will proceed according to their documented contingency plans. b. Hot Site Recovery Example The following timeline represents the high level steps taken in the event the University was to declare a disaster and was required to relocate computer facilities to the hot site location. Time: 0 hour SunGard receives official notification of a Boston University disaster declaration. Packing of backup tapes from the UIS electronic vault for shipment to the hot site begins. Time: 12 th Hour Twelve hours represents the minimum expected arrival time for backup tapes at the hotsite facility. This could extend past 12 hours depending on the nature of disaster and the location of the hot site (Philadelphia mega center is the primary site) SunGard staff begins to recover production systems using the UIS Contingency Planning Recovery Procedures. SunGard restores all University production data from data shipped from the UIS electronic vault. Time: th Hour Production system recovery is completed by SunGard staff and UIS services are ready for verification and access. Time: 0 48 th Hour (concurrent with above actions) The process to deliver emergency computer equipment for 3270 terminal access begins. 13

16 ix. Hypothetical Disaster Recovery Examples (continued) b. Hot Site Recovery Example (continued) Time: 48 th Hour Six Weeks Restoration of the damaged UIS Computer Center and offices commences. A determination as to the extent of outage is made. In this hypothetical scenario, the UIS Computer Center will be restored prior to 6 weeks. Accordingly, a decision is made to not use the cold site but instead direct the ordering of all new computer center equipment to commence for delivery to the original facility. Time: Day 6 Six Weeks The original facility restoration and equipment hookup is completed. All original dedicated communication links are rebuilt. Planning now begins for the backup of all data created at the hot site facility and the cut over from the hot site back to the original facility. At the appropriate time, the hot site is shutdown and final backups are taken. That data is then restored on to the system at the UIS Computer Center. 14