High Availability and Disaster Recovery

Transcription

1 High Availability and Disaster Recovery

2 Klaus Schmidt High Availability and Disaster Recovery Concepts, Design, Implementation With 83 Figures and 26 Tables 123

3 Klaus Schmidt Stockheimer Str Frankfurt Germany Library of Congress Control Number: ISBN Springer Berlin Heidelberg New York ISBN Springer Berlin Heidelberg New York This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilm or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable for prosecution under the German Copyright Law. Springer is a part of Springer Science+Business Media springer.com Springer-Verlag Berlin Heidelberg 2006 Printed in Germany The use of general descriptive names, registered names, trademarks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. Typesetting: Joachim Schrod, Net & Publication Consultance GmbH, Rödermark Production: LE-TEX Jelonek, Schmidt & Vöckler GbR, Leipzig Cover design: KünkelLopka Werbeagentur, Heidelberg Printed on acid-free paper 45/3142/YL

4 For Helga, Andreas, and Katrin

5 Preface During the last 15 years I was involved in planning, deployment, and operations of IT systems for major companies. Those systems are missioncritical: the customers business depends on their availability. The systems are required to be highly available and to protect against all kinds of problems, like hardware failures, software issues, human errors, through to physical disasters. I learned that there are misunderstandings between customers, planners, and vendors about what high availability is, what can be achieved with IT systems, and where their limitations are. I also recognized that disaster recovery is only a feature of high availability, but is often seen as an independent topic. This book addresses this area with an end-to-end view and makes it available as single piece of material: from requirements gathering to planning, implementation, and operations. Another missing piece is supplied, an approach to develop an architecture which leads to highly available systems that are robust and are able to recover from the relevant failure scenarios. But identification of failure scenarios is still a kind of art, mostly based on individual experiences. Selection of a solution is driven by a process, and not by products that claim protection but do not consider the whole picture. With that end-to-end view, we get a structured approach that leads from requirements to possible failure scenarios to a successful solution. That was the motivation to write this book. It addresses these topics and is targeted at all parties involved, enabling them to speak a common language and manage their mutual expectations. The goal of this book is to explain and discuss architecture, technology, solutions, and processes. Since products and features are too short lived for the aim of this book, it does not review, compare, or recommend any particular products.

6 VIII Preface I thank my employer Electronic Data Systems (EDS) for the excellent work environment that I have had over many years, for a very successful working relationship, and for support in writing this book. I would like to thank the following friends and colleagues for their fruitful discussions, guidance, and reviews: J. Bankstahl (EDS), F. Haberhauer (Sun), H. Haupt (AES), F. Krämer (IBM), S. Krampen (EDS), F. Mittelbach (EDS), and Dr. F. Renner (Opel). I also thank all my many colleagues at EDS, especially from the Enterprise Technology Office. Their experience and insights helped me in understanding the full picture. Special thanks to Joachim Schrod from Network and Publication Consultance (NPC). Without his significant contributions, this book would not have been possible. Frankfurt, März 2006 Klaus Schmidt

7 Contents 1 Introduction Audience Roadmap of This Book Real-WorldExamples Elementary Concepts Business Issues Business Continuity as the Overall Goal Regulatory Compliance and Risk Management System and Outage Categorization High Availability Handling Minor Outages Availability Reliability Serviceability Disaster Recovery Handling Major Outages Quantifying Availability: %andReality Service Level Agreements Basic Approach: Robustness and Redundancy Layered Solution with Multiple Precautions Summary Architecture Objectives Conceptual Model System Model System Design Base Concepts SystemStack Redundancy and Replication RobustnessandSimplicity... 74

8 X Contents Virtualization Solution Roadmap List Failure Scenarios Evaluate Failure Scenarios Map Scenarios to Requirements DesignSolution Review Selected Solution Against Scenarios System Solution Patterns System Implementation Process Systems for All Process Steps UseCase:SAPServer Hardware Components and Computer Systems Disk Storage Raid Redundant Array of Independent Disks Storage Systems SANvs.NAS Journaling Is Essential for High Availability Virtualization of Resources Vendor Selection and Purchasing Decisions SystemInstallation System Maintenance and Operations Making Our Own Statistics Operating Systems Failover Clusters HowDoesItWork? Failover Cluster Implementation Experiences Load-Balancing Clusters Load-Balancing Approaches Target Selection for Load Balancing Cluster and Server Consolidation Virtualization and Moore s Law HostVirtualization Databases and Middleware Middleware Categories Database Servers High-Availability Options for Database Servers Disaster Recovery for Databases Web Servers Application Servers Messaging Servers

9 Contents XI 8 Applications Integration in a Cluster on the Operating System Level High Availability Through Middleware High Availability From Scratch Code Quality Is Important Testing for High Availability Infrastructure Network NetworkDevices LAN Segments DefaultGateway RoutinginLANsandWANs Firewalls and Network Address Translation Network Design for Disaster Recovery Infrastructure Services Dynamic Host Configuration Protocol (DHCP) Domain Name Service (DNS) Directory Server BackupandRestore Monitoring Disaster Recovery Concepts Approach Conceptual Design Scenarios for Major Outages Disaster-Recovery Scope Primary and Disaster-Recovery Sites State Synchronization Shared System, Hot or Cold Standby Time to Recovery Failback to the Primary Site Solutions MetroCluster FastRestore Application-Level or Middleware-Level Clustering Application Data Mirroring Disk Mirroring Matching Configuration Changes Disaster-Recovery Tests Test Goals and Categories Organizational Test Context Quality Characteristics Holistic View What Is Needed Besides Technology? Command Center and War Room

10 XII Contents Disaster-Recovery Emergency Pack A Prototypical Disaster-Recovery Project System Identification the Primary Site Business Requirements and Project Goals BusinessView SystemDesign Implementation Failover to Disaster-Recovery Site or Disaster-Recovery Systems General Approach Example Checklist for a Database Disaster- Recovery Server FailbacktothePrimarySystem A Reliability Calculations and Statistics A.1 MathematicalBasics A.2 Mean Time Between Failures and Annual Failure Rate A.3 Redundancy and Probability of Failures A.4 Raid Configurations A.5 ExampleCalculations A.6 Reliability over Time the Bathtub Curve B Data Centers B.1 RoomInstallation B.2 HeatandFireControl B.3 PowerControl B.4 ComputerSetup C Service Support Processes C.1 Incident Management C.2 Problem Management C.3 Configuration Management C.4 Change Management C.5 Release Management C.6 Information Gathering and Reporting References Index