NORTH CAROLINA MANAGING RISK IN THE INFORMATION TECHNOLOGY ENTERPRISE NC MRITE Nominating Category: Nominator: Ann V. Garrett Chief Security and Risk Officer State of North Carolina Office of Information Technology Services Enterprise Security and Risk Management Office
Executive Summary In 2008-2009, the State of North Carolina (NC) faced many challenges as it tried to improve the State s continuity of Information Technology (IT) operations. In the summer of 2008, the brand new Western Data Center came online, giving the state two data centers. Legislative support and funding for the construction of this new data center were predicated on cost savings by conducting IT disaster recovery (DR) and business continuity activities in-state. NC was letting most of its outsourced recovery activity vendor contracts expire. The State essentially had to determine how to best maximize the use of both data centers to support ongoing operations and in-house disaster recovery and business continuity functions. As the planning for this major shift in the entire enterprise approach to IT Business Continuity Planning (BCP) progressed, we realized we did not have the information necessary from the agencies to perform analysis and make wise decisions on the optimization of State IT recovery resources. We desperately needed a way to collect and analyze data on critical IT systems and infrastructure from all State agencies. To better understand and address the business recovery needs of our IT community, we had to determine if there were common gaps impeding the recovery of critical applications. We asked ourselves if all processes and relationships were identified and properly prioritized with known contingencies. This approach highlighted weaknesses in our infrastructure. Recognizing this data only partially supported a comprehensive business continuity planning strategy, we partnered with the NC Division of Emergency Management and the US Department of Homeland Security to fully understand the risk universe that could impact our State and data centers. Prior to 2009, the 30 Executive Branch Agencies submitted annual business continuity plans (BCPs) to the State Chief Information Officer, as required by General Statute 147-33.89; however, while viable at the agency level, they lacked a standardized enterprise approach. The plans were a disparate collection with little common ground for an enterprise-wide analysis to reach an understanding of statewide strengths and weaknesses. Compiling an enterprise view to identify priorities and gaps was nearly impossible. North Carolina needed a universal toolset and risk management methodology to produce a cohesive solution while retaining independent plan administration at the agency level. By March 2010, NC MRITE accomplished the following: improved, standardized and centrally managed business continuity planning tools; evaluated the data from a new enterprise toolset to identify common gaps in BCPs for the recovery of critical applications; identified a means to track critical dependencies to plan for adequate support infrastructure; partnered with the US Department of Homeland Security and the NC Division of Emergency Management on security measures for both data centers; and provided agencies with the opportunity to conduct four formal business continuity tests at the new data center. State agencies successfully completed their 2009 BCPs using a standardized approach and enterprise toolset. NC now has a solid IT recovery infrastructure. We count on improving our approach year after year. - 2-
Description of the Business Problem and Solution: Business Problem: The State needed to transition from an outsourced disaster recovery vendor to the new Western Data Center; however, good information on IT needs at the enterprise level was lacking because agencies were following their own business continuity approaches. NC did not have a strategy for migrating agencies from independent methodologies to a singular functional planning approach to support enterprise-wide analysis of strengths and weaknesses in the State s readiness to prepare for and respond to IT crises. Successfully addressing this challenge would require a common toolset and statewide standards and policies for developing and maintaining BCPs while meeting the individual needs of 30 executive branch agencies. State agencies were preparing business continuity plans with disparate methodologies and lacked a standardized approach. These inconsistencies across the State made identification of critical IT infrastructure and application dependencies difficult to identify and placed a burden on statewide IT service support. When the existing business continuity planning software was due for renewal, the client-server version agencies were familiar with was no longer available. An upgrade to a web-based version was necessary. The upgraded BCP application qualified for hosting in a virtual machine environment. North Carolina began embracing virtualization in 2008 and wanted to leverage this technology option. The State had to decide between upgrading 30 individual agency licenses or converting to an enterprise-wide concurrent user license for all agencies. Agencies favored upgrading the individual licenses; however, this option came with a projected cost exceeding $1 million for 30 separate installations of the required three-tier web, application and database server environment. Converting to an enterprise-wide concurrent user license required a single version of the three-tier environment plus standardized policies and rule sets. This opportunity for standardization meant addressing security and access controls to keep agencies data separate while providing flexibility to meet the needs of small and large size agencies, both consolidated and non-consolidated. Business Solution: A small team of subject matter experts within the State s Enterprise Security and Risk Management Office led the way with the identification and implementation of five key IT goals to improve the State s approach to business continuity planning and promote critical IT service continuity management. Goal 1: Improve, standardize and centrally manage business continuity planning tools. Standardization and cost reductions were recognized by converting from 30 agencies individually hosting the business continuity planning application to one enterprise-wide virtual installation of the software. If hosting remained at the agency level, a minimum of 90 servers would have been necessary with an implementation cost exceeding $1 million in the first year. A proof of concept pilot for virtual hosting was completed in November 2008. Based on the positive results of the pilot, the State chose to proceed with a more cost effective virtual hosted solution. The virtual hosting solution is $42,000-3-
per year. The BCP software contract was renegotiated in November 2008 to convert the 30 individual application licenses to one enterprise-wide concurrent user license for use by all agencies. The contract renegotiation resulted in a cost reduction of $36,725 at contract year-end December 31, 2008, and a further reduction of $10,000 at contract year-end December 31, 2009. A pilot test with a small group of agencies was conducted between December 2008 and February 2009. Full on-boarding to the new standardized BCP application was completed between February and June 2009. Executive branch agencies submitted their first business continuity plans following the standardized enterprise format in September 2009. Enhanced application security allows BCP Administrators the ability to grant/restrict users specific privileges via roles that define access to screens, features, plans, data, documents and reports. Conditional plan and record access can be set for individuals, meaning the users will automatically only see the information that is pertinent to their plan building, such as all plans and employees associated with a specific department. Goal 2: Use the data from enterprise business continuity planning tools to help identify common gaps in BCPs for the recovery of critical applications. Starting in 2009, plans were standardized using Crystal reports; all agency BCPs followed the same table of content structure, resulting in all plans having the same look and feel. This common approach makes it easier to locate critical plan elements and compile statewide results. Agencies were asked to capture critical application data in the enterprise Application Portfolio Management (APM) tool. APM tool data was identified as another key resource in our toolset which produced valuable new metrics for senior State managers including Agency Heads. Reports were provided to the Legislature on application prioritization. See recovery metrics below. - 4-
Goal 3: Identify critical dependencies and plan for adequate support infrastructure. The numbers of factors in a continuity plan are staggering. Even the most complete continuity plans can suffer if one process is not properly prioritized, creating a chain effect that can seriously impact an organization s downtime. Understanding the relationships between BCP elements specifically processes, applications and hardware, and how each element depends on another greatly reduces the likelihood of leaving important contingencies out of a plan. It avoids issues caused by overlapping priorities, over-allocation of resources and under utilization of space. This enterprise approach provides better integration and capability to roll up agency BCPs within an agency as well as statewide views. The enhanced web-based business continuity application provides graphical dependency mappings, allowing the data center IT management to see the connections between the data center infrastructure and the agency s applications. Agencies have the means to identify their critical application dependencies using the new planning tools, paving the way for IT Operations to use these dependency maps to refine infrastructure support services. For example, the network, Internet access and hosting environment need to be restored before a critical web-based application can be restored. This infrastructure order is hard to identify at all levels. The more intricate the dependency relationships for critical applications that are not fully documented, the greater the service restoration risk. The first baseline view of critical dependencies was completed with the September 2009 BCP submissions. Goal 4: Work with the US Department of Homeland Security and the Division of Emergency Management on security measures at the Eastern Data Center and Western Data Center. We invited the US Department of Homeland Security to help us assess our security measures in place at both of our data center facilities during April 2009. We have documented our infrastructure strengths and are focusing on opportunities for improvement. We continue to work closely with the NC Division of Emergency Management to stay abreast of regional risks that could adversely impact our State. Planning for the continuity of critical IT systems operations and security measures is now an integral part of the State s continuity planning process. Goal 5: Provide agencies with the opportunity to conduct both informal and formal business continuity tests and maintain records of agency critical application BCP tests. Semi-annual formal test cycles are in place and agencies may request alternate test windows. The first formal disaster recovery test was conducted at the new Western Data Center in June 2008. Formal testing cycles were also conducted in December 2008 and June and December 2009. The Western Data Center is poised to host the next disaster recovery exercise in June 2010. Tracking of business continuity and disaster recovery testing is now more robust with standardized plan structures and test plans. Test results are documented in the BCP application for use in plan analysis and - 5-
validation, audit assessments, and reporting to management. Agencies are encouraged to conduct table-top exercises at any time throughout the year. Significance of the Project: The State now operates from a position of strength because we have the knowledge gained from a unified functional approach for guiding State agencies in business continuity plan preparation, application criticality ranking, and dependency mapping. Statewide critical infrastructure dependencies are now documented in a common toolset, allowing State IT management the much needed insight to provide the best IT service solutions. Statewide standards and policies for developing and maintaining BCPs have been developed and are being followed. The State now functions under one congruent enterprise-wide business continuity approach, replacing disparate practices at the agency level. Secure access to business continuity plans from any location via the web supports plan exercises, relocation to alternate sites, travel, and the expanding teleworking community within NC government. Disaster recovery was brought in-state during 2008-2009 utilizing the new Western Data Center, removing the dependency on an outsourced vendor. Our understanding of enterprise risks that could impact IT has been strengthened by our partnering with the NC Division of Emergency Management and the US Department of Homeland Security. Completion of these goals and ongoing support are considered key IT initiatives for NC. Achieving these goals was supported by continuous open dialogues with the agencies. Formal communications announcing the projects and detailing key dates and deliverables for the goals were sent to Agency Heads, Chief Information Officers (CIOs) and BCP Administrators in advance. Hands-on training and awareness meetings were held five times starting in late 2008 through April 2010. A communications listserv is utilized to distribute FAQs, address concerns, and maintain a sense of open communication at all times throughout the project. Benefit of the Project: North Carolina recognized an immediate benefit by discontinuing disaster recovery services through an outsourced vendor. The millions of dollars sent out of state for this service and related travel now remains in-state and new jobs were created to support the Western Data Center. The risk of potentially being bumped out of the vendor s disaster recovery center is no longer a factor. The State initiated a new virtual hosting solution for all executive branch agencies business continuity planning application. This eliminated the need for each agency to purchase a three-tier server environment and responsibility for IT support and application administration. This approach provides homogeneous application administration with a common toolset complete with over 100 standardized reports, allowing each agency to maintain independence in their plan development as well as the capability for a statewide view. The standardized plan content structure provides a consistent look and feel to all agency plans which facilitates plan reviews, training, auditing, application and infrastructure support. - 6-
Significant financial benefit was realized with the cost avoidance at the individual agency level. A typical three-tier (web, application, and database) server installation was estimated to cost each of the 30 agencies over $16,000 annually for hosting above initial setup costs. Cost reduction to the State with centralized hosting, reduced licensing fees and the standardization of the BCP resulted in a net savings exceeding $1 million the first year. Savings for the first three years are projected to exceed $1.9 million. Actual savings through May 2010 are $1,796,011.17. BCP Application Hosting Options 3 Year Cost Projections Licenses Servers Option 1 Individual $419,940.00 $2,082,715.80 = $2,502,655.80 Option 2 Enterprise $383,215.00 $126,000.00 = $509,215.00 Projected cost savings over 3 years = $1,993,440.80 The scope of this statewide IT initiative received high-level visibility and generated interest from the Office of the Governor, Agency Heads, and the Legislature. Enterprise Risk Management included bringing a new data center online, conducting a statewide critical application gap analysis, and running a pilot hosting solution from proof of concept to full implementation in a virtualized environment. The State provided a solid management structure, including training for agency BCP Administrators on how to use the new BCP toolset to prepare their plans and procedures following a standardized approach, and provided a formal test environment at the new State data center to conduct their continuity and disaster recovery scenarios. The State CIO is now recognized as a valuable asset to the State s Business Continuity Planning team. NC MRITE improved, standardized and centrally managed business continuity planning tools; evaluated the data from this enterprise toolset to identify common gaps in BCPs for the recovery of critical applications; identified critical dependencies to plan for adequate support infrastructure; partnered with the US Department of Homeland Security and the NC Division of Emergency Management on security measures for both data centers; and provided agencies with the opportunity to conduct four formal business continuity tests to-date at the new data center. State agencies successfully completed their 2009 BCPs using a standardized approach and enterprise toolset. Agencies have taken tours of the new data center to see advanced technologies in operation, and are embracing it as their alternate IT processing and DR facility. NC MRITE aligns with many of NASCIO s 2010 State CIO priorities. These include budget and cost control, consolidation, shared services, security, infrastructure, governance, virtualization, broadband and connectivity. Ultimately, the citizens of NC will benefit from the State s ability to sustain IT support for critical services in an emergency. - 7-