Copyright 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. Presentation_ID.scr 1. IPS _05_2001_c1

Transcription

1 2001, Cisco Systems, Inc. All rights reserved. 1 Presentation_ID.scr 1

2 Introduction to High Availability Networking Session 2001, Cisco Systems, Inc. All rights reserved. 3 Agenda Introduction Building an HA Network Network Outages Human Factors Recommendations 2001, Cisco Systems, Inc. All rights reserved. 4 Presentation_ID.scr 2

3 Introduction 2001, Cisco Systems, Inc. All rights reserved. 5 Availability vs. Reliability Reliability implies that the system performs its specified task correctly Availability means that the system is ready for immediate use An airplane needs to be reliable; It doesn t need to available to fly 24 hours a day; However, it must be reliable when the user determines it is ready to be available Today s networks need to be available 24 hours a day, 7 days a week, 365 days a year; They need high availability 2001, Cisco Systems, Inc. All rights reserved. 6 Presentation_ID.scr 3

4 Why Does Availability Matter Increased demands on the internet from business critical applications Minimize lost revenue due to downtime Competitive pressure Loss of goodwill Loss productivity when network is down Keep your job Maximize ROI of deployed infrastructure Increased revenue from existing services Reduced operating costs 2001, Cisco Systems, Inc. All rights reserved. 7 High Cost of Internet Outages Corporations can lose millions of dollars in just one hour if a mission-critical application becomes unavailable or does not run correctly, quickly or completely Business Brokerage Operations Credit Card Authorization Average Cost per Hour $6.45 million $2.6 million Catalog Sales $90,000 Internet Back in one hour Airline Reservations $89,500 Tele-ticket Sales $69,000 Package Shipping $28,000 ATM Fees $14,000 Source: Contingency Planning Research 2001, Cisco Systems, Inc. All rights reserved. 8 Presentation_ID.scr 4

5 What Is High Availability? Availability DPM Downtime Per Year (24x7x365) % Days 15 Hours 36 Minutes % Day 19 Hours 48 Minutes % Hours 46 Minutes % Hours 23 Minutes % Minutes % 10 5 Minutes % 1 30 Seconds DPM Defects per Million (Customers) 2001, Cisco Systems, Inc. All rights reserved. 9 Availability Availability A M T T R Mean Time to Repair Mean Time Between Failure M T B F 2001, Cisco Systems, Inc. All rights reserved. 10 Presentation_ID.scr 5

6 Availability Definitions Availability = MTBF/( MTBF+MTTR) = 1 - (Total connection outage time total in service connection time) = 1 - [? (No. connections affected in outage i x duration of outage i )] (No. connections in service x operating time) Unavailability = 1 - availability Typically averaged over a month's time 2001, Cisco Systems, Inc. All rights reserved. 11 Example Last month there were 30 users down for 60 minutes There are 1000 users There is 720 hours in a month Outage time 30 users * 1 hours/user = 30 hours Service connection time 1000 users * 720 hours/month = 720,000 User hours/month Availability = (1-30 hours/720k User hours/month) = (99.996%) month/user 2001, Cisco Systems, Inc. All rights reserved. 12 Presentation_ID.scr 6

7 Defects Per Million (Customers) A linear method for measuring availability DPM = (1 %Availability) * 10 6 For PVCs DPM = S (#conns*outage minutes) S (#conns*total minutes) For SVCs or phone calls DPM = S(#existing calls lost + #new calls blocked) total calls attempted For connectionless traffic DPM = S (#end users*outage minutes) S (#end users*total minutes) Example: ( ) * 10 6 = 40 DPM 2001, Cisco Systems, Inc. All rights reserved. 13 Factors Impacting Availability Change management Process consistency Operational Errors 40% Software Application 40% Network 20% Platform problems The network Operating system or hardware Application bugs (ie DNS) Misconfiguration Source: Gartner Group, CNET News.com Jan 26, , Cisco Systems, Inc. All rights reserved. 14 Presentation_ID.scr 7

8 Measuring Availability in Your Network Things for you to consider Interested in end users, end ports, or nodes? Differentiate by circuit speed? Separate planned downtime from unplanned downtime Separate core, access, and LAN What does available mean to you? End users can successfully communicate? Port can pass traffic? Router can pass traffic? Node can respond to ping? Does it have to be reliable? Meeting CIR, jitter, and delay? Define for your environment stay consistent 2001, Cisco Systems, Inc. All rights reserved. 15 Measure! However, Not Always Easy Complex and expensive Multiple networks, protocols and applications; Testing availability for each of these becomes complex and expensive Overhead Proactively measuring true availability measurements require consistent testing or measurement from every point in the network to every other point in the network Application complexity Different applications and protocols have different performance and response time requirements to determine whether a service is available Geophysical complexity Different availability requirements based on location, geography, applications or type of service Consider keeping it simple 2001, Cisco Systems, Inc. All rights reserved. 16 Presentation_ID.scr 8

9 Building an HA Network 2001, Cisco Systems, Inc. All rights reserved. 17 Building an HA Network Introduction Network Outages Hardware resiliency Software reliability/features Link/carrier resiliency Environment/power Network design Human Factor User-error/process Recommendations 2001, Cisco Systems, Inc. All rights reserved. 18 Presentation_ID.scr 9

10 Network Outages Operational Errors 40% Network 20% Software Application 40% 2001, Cisco Systems, Inc. All rights reserved. 19 Hardware Resiliency and Reliability 2001, Cisco Systems, Inc. All rights reserved. 20 Presentation_ID.scr 10

11 Types of Redundancy Parallel Series Redundancy N=2, M=1 N=2, M=1 N=3, M=1 Si IP Cloud Si Availability = (1 - (1-A 1 ) 2 )? (1 - (1-A 2 ) 2 )? (1 - (1-A 3 ) 3 ) (1 - ( ) 2 )? (1 - ( ) 2 )? (1 - ( ) 3 ) = % 2001, Cisco Systems, Inc. All rights reserved. 21 Types of Redundancy Module Redundancy vs. Chassis Redundancy Failover redundant modules only Operating system determines failover Typically cost effective No design limitations architecture limitations All modules are redundant Protocols determine failover Increase cost and complexity Limitations for point-topoint networks 2001, Cisco Systems, Inc. All rights reserved. 22 Presentation_ID.scr 11

12 Hardware Availability Features and Considerations Will the solution allow for load sharing? Which components are redundant? (processor, power, fans, line-cards, bus) What active/standby fault detection methods are used? What is the card MTBF? Should it be made redundant? How long does it take to recover a failed node? How long does it take to do an upgrade? Is hot swap and Online Insertion and Removal (OIR) available? 2001, Cisco Systems, Inc. All rights reserved. 23 Hardware Resiliency and Reliability Conclusions Understand your MTBF for components and devices Know reliability of key paths Understand key availability factors and recovery times Build MTTR processes that fit business requirements 2001, Cisco Systems, Inc. All rights reserved. 24 Presentation_ID.scr 12

13 Software Reliability 2001, Cisco Systems, Inc. All rights reserved. 25 Increased Stability over Time 2001, Cisco Systems, Inc. All rights reserved. 26 Presentation_ID.scr 13

14 Software Availability Features and Goals Minimal interruption for upgrades Fast route processor switchover NSF (non-stop forwarding) capabilities Line card switchover Reduced route convergence times Increase MTBF 2001, Cisco Systems, Inc. All rights reserved. 27 Software Reliability Conclusion Consider new features gained (time to market) vs time tested (time in the field) 2001, Cisco Systems, Inc. All rights reserved. 28 Presentation_ID.scr 14

15 Link/Carrier Availability 2001, Cisco Systems, Inc. All rights reserved. 29 Carrier Management Understand carrier network Model and understand carrier availability Understand carrier diversity strategy Consider multi-homing to different vendors Monitor carrier availability Enhanced services Utilize carrier SLAs Notification and escalation procedures 2001, Cisco Systems, Inc. All rights reserved. 30 Presentation_ID.scr 15

16 Cable Management Color coding Device card/chassis access Cable labels Path diversity Documentation 2001, Cisco Systems, Inc. All rights reserved. 31 Link/Carrier Availability Conclusions Model carrier availability ROI analysis on redundant services Insurance factor for large outages Usefulness of carrier SLAs Review carrier notification/escalation procedures to reduce repair times Cable management to facilitate problem isolation and equipment swap 2001, Cisco Systems, Inc. All rights reserved. 32 Presentation_ID.scr 16

17 Power/Environment Availability 2001, Cisco Systems, Inc. All rights reserved. 33 Availability Affects of UPS and Generator Raw AC 5 Min UPS 1 Hour UPS UPS w/generator Power Array w/generator Event Outages 15 Events 1 Event.15 Events.01 Events.001 Events Annual Downtime 189 Minutes 109 Minutes 10 Minutes 1 Minute.1 Minute Availability 99.96% % % % % Data assumes staffed 1 hour response for UPS Additional downtime for remote UPS service Source: American Power Conversion, Tech Note # , Cisco Systems, Inc. All rights reserved. 34 Presentation_ID.scr 17

18 Affect of Temperature on Failure Rate Failure rate increases by factor of 1.7 to 2.2 for every 10 o C Normal equipment operating temp up to 40 o C, (104 o F) in standard temperature controlled environment 2001, Cisco Systems, Inc. All rights reserved. 35 Power/Environment Practices for High Availability Power/environment facilities planning and management Power circuit to device mapping and management Remote power management Power facilities auditing/consulting Disaster recovery plan (facilities access) 2001, Cisco Systems, Inc. All rights reserved. 36 Presentation_ID.scr 18

19 Network Design 2001, Cisco Systems, Inc. All rights reserved. 37 Complexity and Manageability Defined modules Repeated templates Backbone Standard configuration Standard hardware Internet Standard software No single points of failure WAN Server User Access 2001, Cisco Systems, Inc. All rights reserved. 38 Presentation_ID.scr 19

20 Consider Control Plane Resources CPU Memory Buffers Backplane utilization Bus utilization Messaging/query storms 2001, Cisco Systems, Inc. All rights reserved. 39 Consider Protocol Attributes Complexity to manage and maintain Convergence properties Hold times Signal overhead 2001, Cisco Systems, Inc. All rights reserved. 40 Presentation_ID.scr 20

21 Network Design Conclusion Reduce complexity, increase modularity and consistency Consider solution manageability Minimize failure domain size (reduce single points of failure) Consider control plane resource requirements and the affect of busy CPU/memory Consider protocol attributes 2001, Cisco Systems, Inc. All rights reserved % of All Outages Are Due to Human Error Based on Carnegie-Mellon Usability Study Operational Errors 40% Network 20% Software Application 40% 2001, Cisco Systems, Inc. All rights reserved. 42 Presentation_ID.scr 21

22 Human Error and Process We Have Met the Enemy and He Is Us - Pogo, Original Earth Day Cartoon 2001, Cisco Systems, Inc. All rights reserved. 43 Network Life-Cycle Management New Solution Planning and Deployment Security Management Plan/ Design Implementation Configuration Management Performance and Capacity Management Operation Fault Management 2001, Cisco Systems, Inc. All rights reserved. 44 Presentation_ID.scr 22

23 Plan Hardware Pre-configure, test prior to upgrade Software Research release notes, defect support, new feature set, and device compatibility Configuration Test prior to deployment Have a back-out plan 2001, Cisco Systems, Inc. All rights reserved. 45 Implementation Make different types of changes one at a time Maker/checker model Understand contingency plan in event of failure Validate the change was successful 2001, Cisco Systems, Inc. All rights reserved. 46 Presentation_ID.scr 23

24 Operations Monitor Identifying change, who, what, when Audit trail Fault notification 2001, Cisco Systems, Inc. All rights reserved. 47 Human Error and Process Conclusion Training training training Process process process Plan plan plan Test test test Label label label Document document document 2001, Cisco Systems, Inc. All rights reserved. 48 Presentation_ID.scr 24

25 Recommendations 2001, Cisco Systems, Inc. All rights reserved. 49 Getting to 4 Nines 4 nines, even with redundancy, will be a challenge if you have any of these problems: Single point of failure (edge card, edge router, single trunk) Outage required for hardware and software upgrades Long recovery time for reboot or switchover No tested hardware spares available on site Long repair times due to a lack of troubleshooting guides Inappropriate environmental conditions 2001, Cisco Systems, Inc. All rights reserved. 50 Presentation_ID.scr 25

26 Getting to 5 Nines Cannot Get to 5 Nines if You Have Any of These Problems: High probability of redundancy failure (Failure not detected redundancy not implemented) High probability of double failures Long convergence time for rerouting traffic around a failed trunk or router in the core 2001, Cisco Systems, Inc. All rights reserved. 51 Long Outages Long outages make the evening news and the wall street journal because they affect people s lives now Long outages cause most DPM especially if they affect a large number of users too Longer outages lead to more and more expensive recovery mechanisms; more short outages are better than fewer long ones for the same DPM Need specific requirements to reduce outage on most common events like upgrades, switchover, trunk failures Reduction in MTTR requires better fault isolation tools, on-site spares, training 2001, Cisco Systems, Inc. All rights reserved. 52 Presentation_ID.scr 26

27 Remember No vendor including Cisco can claim to deliver % available networks by themselves This needs a lot of cooperation from customers DPM measurement program, hot spares, troubleshooting procedures, change control and so on 2001, Cisco Systems, Inc. All rights reserved. 53 References Please attend IPS-211 deploying highly resilient IP networks Other related sessions: Campus and LAN protocols and routing protocols Books "Performance and fault management, Cisco Pr; ISBN: "High availability network fundamentals", Cisco Pr; ISBN: Best practice white papers Carrier Class Availability Services (CCAS) , Cisco Systems, Inc. All rights reserved. 54 Presentation_ID.scr 27

28 Introduction to High Availability Networking Session 2001, Cisco Systems, Inc. All rights reserved. 55 Please Complete Your Evaluation Form Session 2001, Cisco Systems, Inc. All rights reserved. 56 Presentation_ID.scr 28

29 2001, Cisco Systems, Inc. All rights reserved. 57 Presentation_ID.scr 29