Contingency Planning and Disaster Recovery Plan Introduction In response to Homeland Security concerns and following sound business practices, agencies were asked to develop a Continuity of Operations Plans (COOP) for review. In response to that directive, John Tyler Community College has developed a COOP as a comprehensive document to deal effectively with personnel safety issues and recovering critical business operations at an off site location. Companion documents to the COOP are the John Tyler Community College Campus Emergency Operations Plan (EOP), John Tyler Community College Information Technology Security Plan, Crisis Communication Plan (CCP), and the College Closing Policy 14.11. The John Tyler Community College Information Technology Security Plan includes, but is not limited to, the College s Business Analysis and Risk Assessment, Risk Assessment for Information Technology Infrastructure, and the Contingency Management Plan (Contingency Planning and Disaster Recovery Plan). Disaster recovery planning involves identifying, preventing and preparing for events that may interrupt the College s business functions due to the effects of disasters or other failure. Please note that these business functions include essential business functions as well as dependent business functions. Risks may come from many sources and could damage the College s image or reputation, cause harm to College assets, subject the College to legal ramifications, or weaken the College s business controls. The business impact analysis process was completed to identify those business functions which must be maintained to provide College services to the greatest extent possible in the event of a disaster. The recovery time objective identified is the maximum acceptable down time for the IT systems associated with these critical tasks. The risk assessment process has been completed to identify risks which may expose the College to these damages and apply proactive, corrective measures to mitigate those risks. The Business Impact Analysis results are located at: G:\itsc\JTCC Security Planning - BIA_RA College management has ensured the necessary allocation of resources for the development and maintenance of a contingency plan for critical information technology systems for the support of essential functions. John Tyler Community College s Information Technology Security Plan has been written to reflect the College s current network infrastructure and environment. This plan applies to all functions, operations and resources at the all campus locations. Page 1
Small non-networked systems, such as those located in office environments, are the responsibility of the individual departments to develop a more abbreviated and less formal plan. All plans will be operationally tested at a frequency commensurate with the risk and magnitude of loss or harm that could result from disruption of information processing support. Definitions A disaster can be defined as a total or partial loss of any or all of the following: physical space, servers, workstations, network infrastructure equipment, personnel, software eradication, or hostile intrusion of the College s network resources resulting in an interruption of services. An incident need not be a disaster to warrant attention. There may be various levels of incidents and each should be handled according to the level of damage the incident may cause to the College s business functions. Types of disasters may include natural disasters (earthquakes, floods, tornados, hurricanes, thunderstorms, and fire for example), system malfunctions (any unavailability due to actual equipment failure or loss of electricity, gas, or water for example), or human causes (deliberate or non-deliberate). Purpose The purpose of the John Tyler Community College Information Technology Security Plan is to ensure compliance with the Information Technology Security Standard as stated in ISO/IEC 27002:2005(E). This document provides the required disaster recovery procedures as well as plans for the continuity of operations which support the College s business functions. Assumptions This contingency plan is based on the following assumptions: On-site College Disaster: 1. Emergency resources and staff can be made available. 2. Users will return to manual operations while ITSC is out of service and will retain source data for ITSC updates when operations are resumed. Page 2
3. All ITSC staff will have access to the plan(s), have been properly trained and can manage the recovery if required. 4. In the event of loss of all ITSC personnel, College administration will contact the Vice Chancellor for Information Technology Services, Virginia Community College System, to request assistance to JTCC management to procure trained operations and technical staff from state wide VCCS resources or elsewhere to carry out ITSC duties. 5. Any surviving ITSC staff members will be able to manage the execution of the disaster recovery plan. 6. Technical personnel can use ITSC's off-site system back-up media and documentation to restore the College's ITSC department. 7. System back-up media are secure at the primary off-site location at the JTCC Midlothian Campus. 8. The college utilizes five Barracuda backup servers to back up the college s data. Three servers are located at Chester and two servers are located at Midlothian. Chester data is backed up locally to one of the Chester BBS appliances and Midlothian data is backed up locally to one of the Midlothian BBS appliances. Twenty-four hours after a backup cycle completes, the data is then replicated. a. Documentation and software located at JTCC Chester or Midlothian Campuses will be available. b. College departments are responsible for inter-departmental hard copy manually processed data and documentation if disaster is wide spread. All electronic data and programs residing on network servers will be retrievable from back-up media. c. Use of the current server environment existing at the Midlothian Campus and acquisition of additional servers acquired at the time of a disaster would provide sufficient server resources to support the College if the Chester site is lost. Reallocation of server definitions and data and program restoration (if required) would bring all services back on line as soon as possible. Current server environment existing at the Chester Campus would provide sufficient server resources if the Midlothian site was lost. d. Reconstruction of the ITSC department may be in a leased mobile office space (double-wide trailer). Location on campus will be determined by the Management Team. Page 3
e. All members of the four disaster recovery teams have been issued a copy of the Contingency Management/Disaster Recovery Plan. Off-Site Non-College Disaster: This would include locations that contain critical applications and data essential to College automated operations via remote based systems (i.e., Oracle, PeopleSoft or PMIS and CARS via VITA). College Environment John Tyler Community College has two campuses, one in Chester and the other in Midlothian. We also have faculty and staff at the College s Nursing Program Site. Chester Campus Street and Mailing Address John Tyler Community College 13101 Jefferson Davis Highway Chester, Virginia 23831-5316 Directions Map of Campus Midlothian Campus Street and Mailing Address John Tyler Community College 800 Charter Colony Parkway Midlothian, Virginia 23114-4383 Directions Map of Campus Nursing Program at CJW Johnston-Willis Campus Street Address 1401 Johnston-Willis Drive Richmond, Virginia 23235 (please do not send mail to this address) Mailing Address John Tyler Community College 13101 Jefferson Davis Highway Chester, VA 23831-5316 Directions Page 4
Infrastructure Environment All network drawings are housed at: G:\itsc\JTCC Security Planning - Configuration Management\Network\Diagrams Backup procedures are housed at: G:\itsc\JTCC Security Planning - Configuration Management\Barracuda\BBS Backup and Recovery\How to Recover Using Barracuda Backup Services - for Systems Engineers.docx Page 5
When an IT Disaster is recognized In the event of a disaster, the disaster planning coordinator, Michael Smith, College Technology Officer (CTO), will initiate disaster recovery procedures. If the disaster planning coordinator is unavailable or incapacitated, the order of responsibility for coordinating the recovery will be as follows: Disaster Planning Coordinator Smith, Michael College Technology Officer (CTO) ITSC Staff in order of succession Michael Russell Chief Information Officer Assistant Vice Chancellor for Enterprise Services Operations Information Technology Services Blackwell, Scott W. Greene, Jeffrey Scott MacPherson, Dennis Programmer Analyst Williams, Patricia Abu-Saleh, Emad Painter, Jill Mitchell, Lincoln Audio Visual/Multi-Media Specialist Chester-B133b; Midlothian-B216 (804) 706-5044 (804) 594-1421 msmith01@jtcc.edu System Office (804) 423-5365 mrussell@vccs.edu Chester-B133 (804) 706-5047 (804) 706-5046 sblackwell@jtcc.edu Midlothian-B212 (804) 594-1423 (804) 594-1426 FAX jgreene@jtcc.edu Chester-B130 (804) 706-5038 (804) 706-5248 FAX dmacpherson@jtcc.edu Chester-B133 (804) 706-5040 pwilliams@jtcc.edu Chester - B133 (804) 706-5049 (804) 706-5245 FAX eabu-saleh@jtcc.edu Chester - B133 (804) 706-5049 (804) 706-5245 FAX jpainter@jtcc.edu Chester-B131 (804) 706-5039 Page 6
lmitchell@jtcc.edu The first task of the coordinator is to secure a current copy of the JTCC Contingency Management/Disaster Recovery Plan. A current copy resides at Chester, B133B, and Midlothian Campus, B216. Additional copies can also be retrieved from: the Vice President of Financial and Administrative Services located at the Chester Campus. First Activity to be performed The disaster planning coordinator, or individual acting as the same, will: Perform a quick analysis of the situation and determine if the disaster recovery plan needs to be implemented. Notify: 1. The College Vice President of Administration 2. The College President 3. The ITSC staff 4. The VCCS Client Services Center, Chancellor and Vice Chancellor of Information Technology Services 5. Notify computer customers (staff, faculty, administrators, students, etc.) Call and place into service the Emergency Management Team. At this point in time, the disaster recovery teams will begin performing their appointed duties. IT Disaster Recovery Teams Disaster recovery teams will be utilized to restore automated computer/technology services. Four recovery teams lead by a disaster planning coordinator will participate in recovery activities based on the level of severity of the loss, recovery deemed necessary, and restoration order as deemed in the Business Impact Analysis and Risk Assessment processes. Team members may serve in multiple roles and be assigned to multiple teams. Page 7
The coordinator and four teams are: Disaster Planning Coordinator: Smith, Michael College Technology Officer (CTO) Chester-B133b; Midlothian-B216 (804) 706-5044 (804) 594-1421 msmith01@jtcc.edu Responsibilities of the Disaster Planning Coordinator are: Manage and coordinate all disaster plan activities. Contact all disaster recovery team members involved in the recovery effort. Ensure all recovery team members have a copy of the plan. Appoint replacement staff if necessary. Contact the College President, College Vice-President of Financial and Administrative Services. Initiate tasks as delegated by disaster recovery team responsibilities. Provide disaster recovery status via communication with College administrators. If necessary, assist planning for returning to normal conditions (renovations, new construction, etc.). Page 8
Emergency Management Team The team consists of the following personnel: Smith, Michael College Technology Officer (CTO) Michael Russell Chief Information Officer Assistant Vice Chancellor for Enterprise Services Operations Information Technology Services MacPherson, Dennis Programmer Analyst Blackwell, Scott W. Greene, Jeffrey Scott Williams, Patricia Abu-Saleh, Emad Painter, Jill Mitchell, Lincoln Audio Visual/Multi-Media Specialist Chester-B133b; Midlothian-B216 (804) 706-5044 (804) 594-1421 msmith01@jtcc.edu System Office (804) 423-5365 mrussell@vccs.edu Chester-B130 (804) 706-5038 (804) 706-5248 FAX dmacpherson@jtcc.edu Chester-B133 (804) 706-5047 (804) 706-5046 sblackwell@jtcc.edu Midlothian-B212 (804) 594-1423 (804) 594-1426 FAX jgreene@jtcc.edu Chester-B133 (804) 706-5040 pwilliams@jtcc.edu Chester - B133 (804) 706-5049 (804) 706-5245 FAX eabu-saleh@jtcc.edu Chester - B133 (804) 706-5049 (804) 706-5245 FAX jpainter@jtcc.edu Chester-B131 (804) 706-5039 lmitchell@jtcc.edu Responsibilities of the Emergency Management Team are: Conduct on-site assessment of the damaged areas. Page 9
Provide a detailed status of the disaster to the disaster planning coordinator as soon as possible. Contact necessary vendors to meet at the ITSC site to assess the damage and determine resources necessary to restore services to the damaged areas. Determine the priorities and the minimum time frame the college will function with degraded operations before the backup plan is implemented. Ensure other support staff is contacted to provide assistance as needed. Determine a time frame for when all services will be restored. Provide the College president and administration a briefing within 12 hours of the damage inflicted on the information technology environment, particularly in ITSC. This briefing may include: o What is affected o Can ITSC be restored at current location o What is salvageable in ITSC o What is probable length of time frame involved for restoration of services o Anticipated vendor support o If a security officer is required in the IT area o Can ITSC s operation be transferred and headquartered at Midlothian Campus (If Midlothian Campus is impacted, can Midlothian operations be transferred to Chester Campus) o Recommend priorities Notify the following individuals/vendors: VCCS Chancellor s Office Verizon Cisco Systems Current Cisco Certified Contract Vendor Cable Contractor Electrical Contractor Dell Computers Mobile office (trailer) vendor Miscellaneous vendors as needed Page 10
Technical Support Team The team consists of the following personnel: Smith, Michael College Technology Officer (CTO) Michael Russell Chief Information Officer Assistant Vice Chancellor for Enterprise Services Operations Information Technology Services MacPherson, Dennis Programmer Analyst Blackwell, Scott W. Greene, Jeffrey Scott Williams, Patricia Abu-Saleh, Emad Painter, Jill Mitchell, Lincoln Audio Visual/Multi-Media Specialist Chester-B133b; Midlothian-B216 (804) 706-5044 (804) 594-1421 msmith01@jtcc.edu System Office (804) 423-5365 mrussell@vccs.edu Chester-B130 (804) 706-5038 (804) 706-5248 FAX dmacpherson@jtcc.edu Chester-B133 (804) 706-5047 (804) 706-5046 sblackwell@jtcc.edu Midlothian-B212 (804) 594-1423 (804) 594-1426 FAX jgreene@jtcc.edu Chester-B133 (804) 706-5040 pwilliams@jtcc.edu Chester - B133 (804) 706-5049 (804) 706-5245 FAX eabu-saleh@jtcc.edu Chester - B133 (804) 706-5049 (804) 706-5245 FAX jpainter@jtcc.edu Chester-B131 (804) 706-5039 lmitchell@jtcc.edu The responsibilities of the Technical Support Team are: Page 11
Work with Emergency Management Team to conduct on-site assessment of the damaged areas to determine condition of network infrastructure equipment/servers, etc., and determine what computer hardware and software has been damaged. Responsible for making decisions based on information received from the Emergency Management Team. Inform Campus Security Force of their role in the disaster plan. Inform Application Owners of their role in the disaster plan and recovery process. Notify staff and vendors using "Emergency Notification Telephone List" o See the Emergency Contact List Determine via JTCC Information Technology Security Plan, Business Impact Analysis and Risk Assessment Executive Summary, which applications are mission critical and need to be on-line immediately and who is responsible for each application. o See BIA System Results. Schedule, coordinate, and communicate as needed with: o All College managers o Emergency Management Team o Special Projects Team o Customer Support Team o Vendors o Users o VCCS Review pre-disaster environment Network Infrastructure Schematics, VCCS and VITA Resources via Connectivity. Prepare the temporary mobile office trailer at the JTCC Chester or Midlothian Campus if required. Reconstruct pre-disaster environment or new environment in accordance with Emergency Management Team's proposal. This will include scheduling repair, replacement, and movement of equipment: 1. Salvage usable equipment 2. Document arrival of equipment 3. Prepare charts with expected installation dates, personnel responsible for duties, etc. 4. Document events as they occur 5. Configuration and moving of equipment Page 12
Reconstruct pre-disaster environment or new environment in accordance with Emergency Management Team's proposal. This will include installation of software: Salvage usable software: 1. Document re-installation of software 2. Prepare charts with expected installation dates, personnel responsible for duties, etc. Document events as they occur: Work with VCCS Customer Support Center to reestablish terminal dependent IMS access privileges. Work with VCCS Security Officer to reestablish terminal dependent VITA access privileges. Prepare a list of the essential items for use in the temporary mobile office (trailer). Special Projects Team The team consists of the following personnel: ITSC Staff College Lab Assistants Institutional Effectiveness College Web Master Associate Vice President of F&A Procurement Staff Accounting Staff Fixed Assets Staff Facilities Staff Executive Secretary, Financial and Administrative Services Responsibilities of the Special Projects Team are: Help with transportation needs, order state cars, trucks, etc. Assist managers in making phone calls as requested. Order supplies as requested. Borrow or purchase boxes for moving. Page 13
Coordinate the preparation of any needed paper work. Acquiring emergency purchasing means (charge card, delegated purchasing, contract information, etc.). Provide assistance to any support teams as needed. Customer Support Team The team consists of the following personnel: Smith, Michael College Technology Officer (CTO) Michael Russell Chief Information Officer Assistant Vice Chancellor for Enterprise Services Operations Information Technology Services MacPherson, Dennis Programmer Analyst Blackwell, Scott W. Greene, Jeffrey Scott Williams, Patricia Abu-Saleh, Emad Painter, Jill Mitchell, Lincoln Audio Visual/Multi-Media Specialist Chester-B133b; Midlothian-B216 (804) 706-5044 (804) 594-1421 msmith01@jtcc.edu System Office (804) 423-5365 mrussell@vccs.edu Chester-B130 (804) 706-5038 (804) 706-5248 FAX dmacpherson@jtcc.edu Chester-B133 (804) 706-5047 (804) 706-5046 sblackwell@jtcc.edu Midlothian-B212 (804) 594-1423 (804) 594-1426 FAX jgreene@jtcc.edu Chester-B133 (804) 706-5040 pwilliams@jtcc.edu Chester - B133 (804) 706-5049 (804) 706-5245 FAX eabu-saleh@jtcc.edu Chester - B133 (804) 706-5049 (804) 706-5245 FAX jpainter@jtcc.edu Chester-B131 (804) 706-5039 Page 14
lmitchell@jtcc.edu Responsibilities of the Customer Support Team: Assist users in recovery of non-network supported automated systems. Assist users in implementing manual procedures to accomplish mission critical application processing if immediate automated solutions will not be forthcoming. Assist users with hardware and software restoration, or relocation to new office site. Keep users knowledgeable of the restoration schedule, particularly as to the impact of there particular areas and services need for them to be productive. Page 15
Emergency Response Procedures John Tyler Community College (JTCC) Information Technology services are split between the two primary campus locations: Chester and Midlothian Campus. The foundation of the College s disaster recovery plan is to take advantage of JTCC s multicampus environment. If one site were to suffer a major disruption, the other location would become the College s disaster recovery site. The ITSC network administrator and staff would perform steps to establish the Midlothian Campus Hot Site as the College s primary WAN/LAN infrastructure domain. If the Midlothian Campus ITSC environment was to suffer a disaster the technology services would be established at the Chester Campus to support Midlothian technical activities. If equipment replacement is required due to the College suffering a debilitating disaster at both campuses, the normal emergency order procedures should produce the needed equipment within a reasonable time to support recovery activities. Page 16
Emergency Telephone List For security purposes, personal telephone information is not housed in this main document. Appropriate employees will have access to the document noted below. o See the EOP Emergency Contact List Smith, Michael College Technology Officer (CTO) College Contacts Michael Russell Chief Information Officer Assistant Vice Chancellor for Enterprise Services Operations Information Technology Services MacPherson, Dennis Programmer Analyst Blackwell, Scott W. Greene, Jeffrey Scott Williams, Patricia Abu-Saleh, Emad Chester-B133b; Midlothian-B216 (804) 706-5044 (804) 594-1421 msmith01@jtcc.edu System Office (804) 423-5365 mrussell@vccs.edu Chester-B130 (804) 706-5038 (804) 706-5248 FAX dmacpherson@jtcc.edu Chester-B133 (804) 706-5047 (804) 706-5046 sblackwell@jtcc.edu Midlothian-B212 (804) 594-1423 (804) 594-1426 FAX jgreene@jtcc.edu Chester-B133 (804) 706-5040 pwilliams@jtcc.edu Chester - B133 Page 17
Painter, Jill Raspiller, Dr. Ted President President's Office Susan Grinnan Vice President, Finance & Administration Finance and Administration Fiege, Dr. Bill Vice President, Learning & Student Success Dunaway, Gregory Director, Facilities Operations and Safety Facilities Operations Horning, Jerad Web Developer Creative Services Jimison, Nancy Procurement Manager Business Office Tanya Brown Assistant Director of College Safety and Security (804) 706-5049 (804) 706-5245 FAX eabu-saleh@jtcc.edu Chester - B133 (804) 706-5049 (804) 706-5245 FAX jpainter@jtcc.edu Midlothian-B206c; Chester- A105b (804) 594-1571 (804) 706-5013 (804) 594-1621 FAX traspiller@jtcc.edu Chester-A102c;Midlothian-B206e (804) 706-5016 (804) 594-1403 (804) 796-4163 FAX ftaylor@jtcc.edu Chester-N113; Midlothian B206f (804) 706-5004 (804) 594-1406 wfiege@jtcc.edu Midlothian-P101 (804) 594-1430 (804) 594-1430 (804) 594-1438 FAX llaclair@jtcc.edu Midlothian-E124G (804) 594-1526 (804) 594-1531 FAX jhorning@jtcc.edu Chester-A102a (804) 706-5024 (804) 796-4163 FAX njimison@jtcc.edu Chester-N111b (804) 706-5017 Page 18
Facilities Operation (804) 897-6678 (804) 796-4163 FAX tbrown01@jtcc.edu Patteson, Beki Fixed Asset Control Specialist Finance and Administration Mitchell, Lincoln Audio Visual/Multi-Media Specialist Chester-G118 (804) 706-5019 (804) 706-5019 (804) 706-5245 FAX bpatteson@jtcc.edu Chester-B131 (804) 706-5039 lmitchell@jtcc.edu Page 19
Maintaining and Testing the Plan The Disaster Planning Coordinator is responsible for maintaining the VCCS Information Technology Disaster Prevention and Recovery Program. The College Coordinator has developed a plan to minimize disruptions of critical functions and the capability to recover operations expediently and successfully. The Coordinator performs the following activities to maintain the plan: Develop a time table to test the College plan as required. ITSC staff will support the Coordinator to ensure the plan is continuously updated with the most recent information so as not to render the plan outdated and useless. Update the position description of the employee assigned to be the Disaster Planning Coordinator and include the responsibilities in the employee annual Performance Evaluation. Coordinator conducts annual planning meetings with each team prior to annual testing. The purpose of testing the plan is to ensure the plan will work when needed. Goals are to identify the weaknesses and incorporate the information obtained to improve the plan. The testing will assess the following: Verify that the plan is complete and accurate. Evaluate that all participants know their roles and responsibilities. Measure the overall performance of the plan. ITSC will participate in all college-wide testing. Information or documentation obtained from all testing will be used for detailed analysis and improvement of the plan. Training will be held at various ITSC staff meetings and informally periodically to ensure IT staff is aware of their individual roles and responsibilities, notification and warning procedures, location and use of emergency equipment, and emergency shutdown procedures. Input from staff at these meetings may be used to update the plan accordingly. Updates will be made periodically as recovery strategies or technology environments change. Page 20
Technology Team information from the current COOP. The College Management Team guides and directs all phases of the COOP process and evaluates the transition between the Responses, Recover, Resume, and Restore phases. The College Management Team will also monitor and evaluate the Continuity Teams and determine when specific teams can either be disbanded or restructured. The Director of Information Technology is part of the College Management Team and will: Contact the IT staff and provide a status report of the current situation. Call the Business Recovery Center contacts and provide a status report of the current situation. Assign Incident Command Post contacts the tasks of preparing the Recovery sites for use. Recommend establishing a Call Center with appropriate personnel. Announce fax numbers and contact numbers on the agency web site and TV/radio. Technology Team The Technology Team provides the essential technical and telecommunications support functions during the recovery and resume processes. The Technology Team is activated by a call from the College Management Team. The Technology Team reestablishes technology services. The Technology Team consists of individuals who are knowledgeable about information systems and infrastructure, telecommunications and database technologies. This group is responsible for the recovery and resumption of automated systems, networks, and telephony for John Tyler Community College. In accordance with the active Security Plan, the following will be recovered: Telephones: Determine what systems are intact and what needs to be recovered, if possible. Data Communications: Determine which networks are functioning and which are not. Take necessary steps to reestablish communications in coordination with the Special Projects Team. Information Services: Restore critical applications at Business Recovery Center site, as needed in the order of priority established by JTCC. Page 21
Step Activity 1 Respond to the Special Projects Team call in response to an event. Get as much information as you can on the nature and the circumstances of the emergency. 2 Phone the other Technology Team members, as required. Convey to them what you know about the event. 3 Identify Client Services personnel who will work with Public Information Office on all communications associated with the disaster event. 4 Call the contact at the Business Recovery Center and schedule the restoration. 5 Contact Critical Vendors to provide assistance at the Business Recovery Center. 6 Determine the extent of the Technology and Data procedures that must be followed as identified in the current John Tyler Community College Information Technology Security Plan. 7 Re-establish critical data center services through hardware reconfiguration, ordering of hardware, or use of other state facility computing resources as specified in the in the John Tyler Community College Information Technology Security Plan. 8 Report recovery progress to Special Projects Team for the Business Recovery Center. John Tyler Community College is not required to establish any Memorandums of Agreement for Business Recovery locations. All Business Recovery Center sites are John Tyler Community College locations. These consist of the following locations to be used in the event any one or two of the other John Tyler Community College locations experience an emergency. Recovery Team Info (Note: ITSC employees may serve on more than one team) Page 22
College Management Team Disaster Site Business Recovery Center(s) Technology Recovery Site Coordination & Recovery Team Special Projects Team Technology Team Administrative Support Team Technology Team Division Teams Page 23