Ensuring Stability to our Rapidly Changing (DR) World. Larry Heck October 7, 2014 Case Study

Transcription

1 Ensuring Stability to our Rapidly Changing (DR) World Larry Heck October 7, 2014 Case Study

2 What cards have you been dealt?

3 Problem Statement 1 mile between datacenters No clear visibility of DR program compliance Poorly implemented notification tool Overprovisioned data storage Lack of resiliency

4 Problem #1: Existing Datacenters Desirable NYC work locations but questionable designated DC locations Good neighborhoods but poor DC environmentals Nice recipe for a geographic disaster!

5 New Datacenter Part A $650 million facility opened Jan 1, ,000 sq ft research facility 3,000 sq ft datacenter Upper East Side location

6 New Datacenter - Part B CU Ithaca Campus Free shared datacenter space 250 miles away from Manhattan

7 Datacenter Reflection What is your dreaded datacenter disaster scenario? Should you be investigating other datacenter solutions?

8 Problem #2: DR Program Compliance Evolution process: - Phase 1: Charm approach (Previous State) - Phase 2: Structured approach (Current State) - Phase 3: Integrated work mgmt. approach (in Development) - Phase 4: Baked into Operations (Future State) Framework: - DR Program established DR policy (anchor) - DR Governance accountability (essential) - DR Prioritization tiered services (what matters most) - DR Recoverability up-to-date, tested DR plans (wisdom) - DR Mitigation willingness to fight for resources (using our influence) Internal Goal: - 80% Compliance during each annual compliance period!

9 Current DR Compliance Reporting

10 Why IT Work Mgmt. Approach? Some reasons - Managements commitment - Leverage integration points - Known structure for controlling, reporting and dashboarding - Helps in the document management - Use same notification mechanism

11 DR Program Compliance Reflection What are your compliance pain points? Right escalation process for non-compliance? How do you measure success? Adequate foundation to build upon?

12 Problem #3: Notification Process Evolution process: - Phase 1: Call Tree - Phase 2: Basic notification (Current State) - Phase 3: Enhanced utilization (In Development) - Phase 4: TBD Dilemmas: - Controlled by another department - Different budget pocket - Philosophical differences

13 Notification Process Challenges: - Mobile workforce - Decentralized environment - Mix of students and staff - Regulatory requirements

14 Notification Reflection Are you caught up in clashing philosophies? Is your notification process doing everything you want e.g., staff check-in?

15 Problem #4: Data Storage Evolution process: - Phase 1: Offsite tape replication (Prior State) - Phase 2: Selected application replication (Current State) - Phase 3: Enhanced file and block replication (In Development) - Phase 4: TBD Dilemmas: - Overprovisioned storage - Enforcing storage policy - Unproved tape recovery capability - Geographic risk

16 Data Protection Tiers Tier Criticality RTO Datacenter Disk Tape 0 Self-Healing Immediate BRB & Ithaca 2 copies 2 copy 1 Mission Critical < 24 hrs BRB & Ithaca 2 copies 2 copy 2 Highly Critical < 72 hrs BRB only 1 copy 2 copy 3 Critical < 5 days BRB only 1 copy 2 copy 4 Non-Critical < 2 Weeks BRB only 1 copy 2 copy 5 Deferrable > 2 weeks BRB only 1 copy 2 copy Near-line data replication between BRB and Ithaca Copy = directly, service useable version (no restoration is required)

17 Data Storage Reflection Solution meet your business requirement? Monitoring adherence to your storage policies? How are you managing the data explosion (and keeping it safe at the same time)?

18 Problem #5: Evolution process: - Phase 1: In-house solution (Prior State) - Phase 2: Hosted locally offsite (Current State) - Phase 3: Cloud-based solution (In Development) - Phase 4: TBD Dilemmas: - Multiple institutions - Less downtime tolerance - Geographic risk

19 Reflection Vulnerable with your service? Ready to outsource ?

20 Recap: Learning Lessons Our business environments are changing rapidly. We need to remain agile. We need to be clear about our unchanging DR/BC role. We need to continue to use our DR/BC influence in exposing risks. Our plugged-in ness will determine our institutional value. We need to avoid resting on yesterday s successes. Wonder what the next five years will bring?

21 Roller Coasters

22 My DR Role at WCMC

23 Perspective on our world

24 Q&A Larry Heck

25 Cornell Tech Campus

26 About the Speaker Larry Heck is the Disaster Recovery Manager in the Information Technologies and Services Department at Weill Cornell Medical College in New York City. Larry works to develop and maintain the DR program that cares for the clinical, research and academic operations. Previously, he served as a Business Continuity and Disaster Recovery Manager at Avaya. Prior to his BC/DR involvement, Larry worked in various financial roles at Nabisco and other global corporations. He has a MBA in Finance and a BA in Accounting as well as CBCP with DRII. Personally, he is married with two adult children and lives in Northern NJ.