SOP Unplanned Outages/Major Incidents

Transcription

1 Table of Contents Definitions... 2 Incident... 2 Declaring... 2 Incident Levels... 3 Priority Priority Priority Priority Mission Critical Services... 3 Authentication... 3 Computer Labs Network... 4 Wireless Network... 4 Power Disruptions to Campus... 4 Storage Area Network (SAN) Disruption... 4 Roles and Responsibilities of Team for Priority 1 Outages/Major Incidents... 4 CIO or Backup... 4 Incident Commander... 4 Technical Lead... 4 User Support Lead... 4 Communications Lead... 5 Field Coordinator... 5 Service Owners... 5 Subject Matter Expert (SME)... 5 Communications... 5 Internal Communications Options... 5 External Communications Options

2 Reporting... 6 Meeting Location... 7 Logistics... 7 Execution of Work Steps for Priority1 Major Incident... 8 Incident commander duties General duties Incident commander timed checklist Check List Review the Emergency Prep roles and procedures! Username: it Password: Ic3 Definitions Incident Whenever a user is not receiving an expected level of service from an IT service. Expected levels of service are based on Service Level Agreements (SLA). Major Incident/Outage A major incident is defined as a significant event, which demands a response beyond the routine, resulting from uncontrolled developments in the course of the operation of any establishment or transient work activity. Declaring Mission critical (university or internal) IT service(s) are not performing at the expected level for a period of 30 minutes unless defined differently in the SLA or designated otherwise by this plan. 2

3 Incident Levels Priority 1 Mission critical services are not performing for the University. All appropriate resources will be dedicated to restore service(s). Priority 2 Mission critical services are not performing for departments or computer labs. Service(s) is not performing at a campus or enterprise level. Appropriate services owners will be dedicated to restore service(s). Priority 3 Address problem and escalate as necessary. These incidents do not require the dedication of level 1 or 2. Priority 4 There is a known work around for the issue. Does not require dedicate resources to resolve. Mission Critical Services Authentication Accepting authentication requests and responses for the following systems: Blackboard Campus desktop computers Central IT maintained computer lab machines CUSIS portal MyUCCS Portal Wireless System Computer Labs Disruptions to IT maintained computer lab machines not allowing customers the ability to utilize systems. messages not flowing in or out of the following systems: Exchange on premise Office 365 Cloud Solution Note: Unless the service outage is determined to be an exclusive Microsoft issue and UCCS IT personnel have no control to participant in a resolution, than this will not follow the full Major Critical procedures. Conceivably only the communication plan will be followed. 3

4 Network Disruptions to campus network systems to include: Campus Firewalls Campus Routing Campus Switches Connections in and out or within the El Pomar Data Center Connections in and out or within the Columbine Data Center Connections in and out or within Main Hall and Cragmor Hall External internet connectivity Wireless Network Disruptions to the wireless system not allowing customers to utilize the network Power Disruptions to Campus Any power disruption to the El Pomar or Columbine Data Centers lasting longer than 10 minutes. Storage Area Network (SAN) Disruption Any disruption to data flowing in or out of the campus SAN solution. Roles and Responsibilities of Team for Priority 1 Outages/Major Incidents CIO or Backup Authorize resources for the major incident; direct communication with Chancellor and UCCS Leadership team; and if needed communications with Presidents office. Incident Commander Coordinate plan; oversee response; lead meetings; organize meals; and provide funding; See below for detailed description. Technical Lead Examine situation; confirm major incident; attempt to identify root cause; work to find technical options; present technical options to team; and participant with plan where needed. User Support Lead Provide information from user s perspective; provide user support options; contact specialized users; and participant where needed. 4

5 Communications Lead Create plan for messaging including frequency; provide messaging to campus; update UCCS.info; point person for internal communication; and participant where needed. Field Coordinator Provide information from the field; deliver support from the field; participant where needed. Note: depending on the major incident this role may not be needed. Service Owners Provide information on services effected; work with technical lead to create options for plan of action. Subject Matter Expert (SME) An individual with a high-level of overall knowledge of the service impacted, both in terms of general architecture and business service provided. Communications Internal Communications Options Communications should be sent out from the helpdesk@uccs.edu address if possible outage@uccs.edu - hosted on lists.uccs.edu (Communigate server local infrastructure must be working) (Texting and ) outage@uccs.info - hosted through Bluehost.com (Texting and ) uccshelpdesk@gmail.com - help desk communications sent when exchange is not available CenturyLink Conferencing Audio Conferencing USA: USA /Canada (toll free): This will be the Major Incident main line: Web / sharing desktops GotoMeeting.com External Communications Options Students - student-1@uccs.edu (lists.uccs.edu is required) Faculty - faculty-l@uccs.edu (lists.uccs.edu is required) Staff staff-l@uccs.edu (lists.uccs.edu is required) UCCS.info {needing login information} Automatically posts to IT Twitter UIS itccop@cu.edu Housing Lists: Summit-l@uccs.edu Alpine_l@uccs.edu Timberline-l@uccs.edu 5

6 UCCS leadership - Only CIO or backup communicates with leadership team University Relations Hutton, Tom Executive Director University Advancement - University Communications and Media Relations MAIN 301A thutton@uccs.edu UCCS Twitter Denman, Philip Assistant Director University Advancement - University Communications and Media Relations MAIN 301 pdenman@uccs.edu Website Alerts {Craig needing information for posting in Ingeniux or how this should be handled} Rave (Must first check with Tim Stoecklein before post message with system) Stoecklein, Tim Program Director of Emergency Management Public Safety Department - Emergency Management DPS 208 tstoeckl@uccs.edu Phones Help Desk ACD message Sidecars if necessary Media University Relations will be the only organization allowed to speak to the media. Reporting When to report Who to report to UIS Other CU campuses Chancellor's office President's office 6

7 Meeting Location EPC 139, IT Conference Room Location needs: Phone Laptop/Projector White board Table Room and chairs for 10 people Extra Ports Power Logistics Review Mission Critical Services Communications expectations plan Communication templates Define essential personnel and backups Personnel expectations during major incident and after Essential personnel is expected to participant in major incident/outage response. If incident is after hours essential personnel is expected to participant if available. Working Time: 16 hours working max or 2 a.m. At the start of 14 hours, or midnight appropriately, technical lead must start to create plan for providing rest to employees. Discuss of break/meal every 4 hours. Food/Drink coordination After major incident/outage is resolved and work was conducted after normal business hours, employees will be given hour for hour flex time. The employee is expected to take the time and must be used within one month from when the work was performed. Incident Commander will work with employee s supervisor to coordinate flex time. Equipment needs Funding Equipment needs shall be coordinated by the Incident Commander. Will be coordinated by the Incident Commander. 7

8 Execution of Work Steps for Priority1 Major Incident Service restoration target is two hours for a Priority 1 Major Incident. Task Description 1. Notification and Confirmation Priority1 Major Incident has been identified 2. Notify IT Outage Group Send or text message to outage@uccs.info or outage@uccs.edu Text Template: UCCS IT internal alert (ServiceName) is experiencing a service interruption. See for more details. Sent by: (Name) (PhoneNumber) Time Incident as past the trigger points Within 10 Minutes Template: Subject: (ServiceName) is experiencing a service interruption. Current symptoms includes: - SYMPTOM1 Known workarounds include: - WORKAROUND1 IT is working to restore service and will provide more information as it becomes available. The next communication will be sent by XX:XX a.m./p.m. 3. Contact Computing Services Directors: Kirk Moore Cell: (719) House: (719) kmoore@uccs.edu or uccsit@moorei.com Within 10 minutes of initial contact Greg Williams Cell: (719) House:(719) gwillia5@uccs.edu or 8

9 If directors have not been reached contact associate directors: Rob Garvie Cell: (719) House: (719) Mike Belding Cell: (719) House: (719) If directors or associate directors have not been reached contact CIO: Jerry Wilson Cell: (719) House: (719) Open Technical Bridge (toll free): Within 15 minutes of initial contact Host and Guest passcodes stored in LastPass under Incident Response 5. Initial Triage 1. Start assessment 2. Replicate issue 3. Review monitoring and logs 4. Try to identify workarounds If no quick solution or workout is discovered then declare a Priority 1 Incident. 6. Declare Priority 1 Incident 1. Select Incident commander 2. Start calling in personnel to meeting location or conference bridge 3. Define roles of team 4. Start external communication Set Message on UCCS.info using Template Within 20 minutes of initial contact Within 30 minutes of initial contact Text Template: UCCS IT Alert: SERVICENAME is experiencing a 9

10 service interruption. IT is working to restore service and will provide more information as it becomes available. Template: Subject: UCCS IT Alert: SERVICENAME is experiencing a service interruption. Body: Current symptoms includes: - SYMPTOM1 Known workarounds include: - WORKAROUND1 IT is working to restore service and will provide more information as it becomes available. The next communication will be sent by XX:XX a.m./p.m. 5. Communicate to CIO or person acting as backup and they will contact Chancellor s office 6. Determine if service should remain active or be brought down. 7. Determine whether individuals aiding in incident restoration should convene in person to aid in restoration efforts. 8. Determine whether vendor involvement or escalation is required. 9. If incident resolution is not expected within 15 minutes, establish time frame for next status update. 10. Continue to facilitate conversation as appropriate to ensure focus is on restoring service. 7. While Priority 1 Incident is occurring 1. Request current status of restoration efforts. 2. Instruct communication lead to send a notice using the following templates: Template: Subject: UCCS IT Alert: SERVICENAME is experiencing a service interruption. Every 60 minutes 10

11 Body: Current symptoms includes: - SYMPTOM1 Known workarounds include: - WORKAROUND1 Update: UCCS IT is working to restore service and will provide more information as it becomes available. The next communication will be sent by XX:XX a.m./p.m. 8. Upon Service Restoration 1. Request the technical lead to verify that service has been restored and report on the current state of operation. 2. Confirm the communications lead will: Update UCCS.info Remove ACD message on Help Desk phone line. Send an incident restoration message with the following content: Upon service restoration Template: Subject: UCCS IT Alert: SERVICENAME Service Now Available Body: UCCS IT has identified and addressed root cause. As of XX:XX a.m./p.m. service has been restored. Thank you for your cooperation we worked to resolve this issue. 3. The technical manager or director of the failing service or component that caused the incident will: Hold a debrief meeting Prepare and deliver incident report to CIO, Directors, and campus IT partners within three business days. 4. Formally state to participants on the technical bridge that the incident status is downgraded, everyone is standing down and the technical bridge is being closed. 5. Document resolution and close incident management (IM) ticket. 6. Open problem management ticket to track ongoing root cause analysis efforts and document any known workarounds. 11

12 Close Major Incident 1. Hold debrief meeting with three days 2. Prepare Major Incident Response report within five days with the help from those participating 3. {Rachel - needing report template} 4. Distribute report Incident commander duties An incident commander serves to keep an incident project on track for process, maintain focus on the problems, facilitate analysis and interactions, and verify that the incident response team s needs are being met (resources, information, etc.). To that end, the individual has several duties outlined below. General duties Open the incident phone bridge line: (toll free): Host Passcode: Guest Passcode: Monitor incident phone bridge line or assign duty. Assign a participant to set up any required A/V resources (projecting monitoring data, etc.). Briefly recap incident process at the beginning of incident room level events. Coordinate efforts within the room to minimize confusion and reduce the risk of inadvertent or simultaneous changes. Draw focus back together when conversations become unproductively fragmented. Document notable events and steps taken in a visible incident log (to be recorded electronically by designated in-room scribe). Solicit approval and/or consensus on decisions to bring services up/down and to make changes to production services. Ensure that the appropriate UIS employees and vendors are engaged and working the issue, tasking people to escalate as needed. Initiate brainstorming during troubleshooting and ensure that identified paths of investigation (hypotheses) or actions are assigned to individuals and given an order/priority. Facilitate communications efforts (both to broad groups of customers and executives) by ensuring that the appropriate communicators have timely and accurate information. Record employees hours worked and ensure they take their flex time. 12

13 Incident commander timed checklist Every 30 minutes: o Request status from teams working issues. What current hypotheses are being investigated and have any been eliminated or verified. What actions have been completed or are in progress. o Provide a verbal update within the incident room and update the incident log. Every hour: o Check in with communications staff regarding next status update steps. o If the list of hypotheses has been exhausted, initiate a new cycle of brainstorming, documenting, assigning tasks, etc. At 11:30am and 5:30pm: o Request that business office (if available) order some food for those working the issue in the incident room. Be sure to cover dietary needs (vegetarian, etc.). o Encourage participants to use mealtime as an opportunity to leave the room for a little while, allowing for coverage if needed. At 9pm: o Request that directors/managers begin their plans for staff rotation during the night if ongoing work is required. Check List 1. Notification of Priority 1 incident 2. Confirmation of major incident/outage 3. Priority 1 incident has been determined 4. If Level 1 priority 1 incident a. Has incident crossed trigger points i. No continue to monitor situation ii. Yes 1. Create problem in Cherwell 2. Determine which individuals are needing to evaluate the situation 3. Define roles for individuals participating 4. Tools a. Last pass Cloud Service for password management b. Monitoring c. Testing environment 5. Build action plan: a. Define scope / timeframe b. Develop technical plan 13

14 c. Define personnel needed d. Determine return on investment e. Assign tasks 6. Communication plan a. How do we communicate with each other? b. How and who do we communicate with externally? c. Recording communications d. Confirming communications postings e. How often do we need to communicate f. Communicating to UCCS Leadership (Role of CIO) 7. Document going progress and issues, record in Cherwell 8. Resolved a. Documenting issue, response and fix 9. Closing response a. Hold debrief meeting with three days b. Prepare Major Incident Response report within five days with the help from those participating c. {Rachel - needing report template} d. Distribute report 14