High Availability in Numbers

Size: px

Start display at page:

Download "High Availability in Numbers"

Joanna Armstrong
5 years ago
Views:

1 High Availability in Numbers mag. Sergej Rožman; Abakus plus d.o.o. The latest version of this document is available at:

2 High Availability in Numbers mag. Sergej Rožman Sistemi

Information System) ARBITER the ultimate tool in audit trailing APPM - Abakus Plus Performance

(services, VPN, QoS, security) open source, monitoring (Nagios, OCS, Wiki) Hardware: servers, SAN

3 Abakus plus d.o.o. History from 1992, ~20 employees Applications: special (DB Newspaper Distribution, FIS Flight Information System) ARBITER the ultimate tool in audit trailing APPM - Abakus Plus Performance Monitoring Tool Services: DBA, OS administration, programming (MediaWiki, Oracle) networks (services, VPN, QoS, security) open source, monitoring (Nagios, OCS, Wiki) Hardware: servers, SAN storage, firewalls Infrastructure: from 1995 GNU/Linux (18 years of experience!) Oracle on GNU/Linux: since RDBMS & Forms 3.0 (before Oracle!) >20 years of experience with High-Availability!

4 Router/Firewall!!!FAILOVER!!! FW1 FW2

5 HA Customers

6 IDC's Availability Spectrum Availability level Impact of Component Failure AL 1 Work stops; uncontrolled shutdown; data integrity ensured AL 2 User interrupted, but can quickly log on again; may need to rerun some transactions from journal; may experience performance degradation AL 3 User stays online; current transaction may need restarting; may experience performance degradation AL 4 Transparent to user; no interruption of work; no transactions lost; no degradation in performance

7 Availability Classes Availability Class Availability % Downtime per Year 1 90% < 36.5 days 2 99% < 3.65 days % < 8.77 hours % < 52.6 minutes % < 5.26 minutes % < 31.6 seconds % < 3.16 seconds

8 Class Feasibility Binomial distribution f n, p (k ) = n pk (1 p)n k k () (Bernoulli trials) probability (%) 90 p= single system 2-node cluster years

9 DEFINITON: Highly Available System The computer system is highly available when it has two following properties: no component is a single point of failure; overall, it is reliable enough that it can be repaired before something else breaks. Tone Vidmar; Informacijsko-komunikacijski sistem

10 Mean Time Between Failures operating time MTBF = failures Component MTBF (h) MTBF (years) Interface card; e.g., LSI MegaRAID SAS i4e SGL Cooling fan Disk drive; e.g., Seagate ST SS

11 MTBF of a Single System MTBF (h) MTBF (years) Server board Backplane board Power supply Cooling fans PCI riser card Front panel board Component All together 1 1 = MTBF MTBF n n

12 Probability of Failure Probability for failure Lifetime ~1 month ~5 years

13 Annual Failure Rate 1 AFR MTBF in-years Component MTBF (h) MTBF (years) AFR ,0055 Interface card ,0095 Cooling fan ,035 Disk drive AFR can be interpreted as a probability that the component fails within one year.

14 Annual Failure Rate 1 AFR MTBF in-years Component MTBF (h) MTBF (years) AFR ,0055 Interface card ,0095 Cooling fan ,035 Disk drive expected_failures = components AFR time in-years For example: SAN system with 200 disk drives: expected failure rate is 1.1 disk drives per year!

15 Mean Time to Repair MTTR of a disk drive in a RAID configuration: 8h with a hot-spare 48h without a hot-spare AFR RAID5 AFR RAID10 n(n+ 1) MTTR MTBF MTTR n MTBF * from [2] but not entirely true, did not consider that neighbouring time slot failures are problematic as well

16 MTBF, MTTR & AFR MTBF = h MTTR = 8 h using 1 TB disk drives 0,001 AFR 0,0001 RAID5 RAID10 0, , capacity (TB)

17 About Disk Drives Array disk failures are highly correlated, making RAID 5 two to four times less safe than assumed Enthusiast-oriented reports such as AnandTech, quoting an»8% chance of complete data loss using RAID 5 with 200GB spindles«the»father of DEC StorageWorks«(now HP EVA) quoting that»if you have one petabyte of desktop drives with RAID 5, you could lose data twice a year«

18 Simultaneous Multiple Failures Birthday paradox n... # of components, e.g. disk drives t... # of MTTR intervals in lifetime t = MTBF / MTTR n! t n p(n) = 1 tn () lifetime MTTR t n probability 10 years 48 h % 10 years 8h % 30 years 48 h % 30 years 8h % 1 day people 50.7% birthdays: 1 year

19 Guidelines Make full failure mode analysis of your system(s). The AFR for RAID group should be smaller than This maximizes the number of disks to 8 (7 + 1) for RAID5 and 16 (8 + 8) for RAID10. (Google claims: AFR of a single disk drive is >4%) Use hot-spare disks.

20 IT Infrastructure primary data center 10G-eth switch secondary data center 10G-eth switch 10G-eth switch virtualization server 1 SAN 1, 2, 3: SSD drives for Oracle DB SAS drives for other applications (virtualization) SATA drives for backup virtualization server 2

21 High Availability Traditional Way Clustering Grid Application service failover Application Instance 1 Application Instance 2 Application Cluster Physical Hosts

22 High Availability Inventive Way Virtualization Cloud Transparent system failover XEN remus KVM kemari failover App App App App App App App App App OS OS OS OS OS OS OS OS OS Virt. Infrast. Virt. Infrast. Resource Pool Physical Hosts Virt. Infrast.

23 HA Work in Progress massive virtualization cloud: using low cost commodity hardware disaster site: on-line transparent stateful switchover/failover (and switchback again) as many times as you like

24 References [1] Intel; Calculated MTBF Estimates ( [2] Klaus Schmidt; High Availability and Disaster Recovery: Concepts, Design, Implementation [3] Tone Vidmar; Informacijsko komunikacijski sistem [4] Eduardo Pinheiro, Wolf-Dietrich Weber, Luiz Andre Barroso Google Inc; Failure Trends in a Large Disk Drive Population ( [5] Johan De Gelas; Server Guide part 2: Affordable and Manageable Storage ( [6] Robin Harris; StorageMojo (

25 High Availability in Numbers Questions mag. Sergej Rožman ABAKUS plus d.o.o. Ljubljanska c. 24a Kranj sergej.rozman phone: abakus.si

APEX Shared Components. Aljaž Mali, Abakus Plus d.o.o.

APEX Shared Components. Aljaž Mali, Abakus Plus d.o.o. APEX Shared Components Aljaž Mali, Abakus Plus d.o.o. 2 Quick introduction IT Solutions Architect at Abakus plus, d.o.o SIOUG - Vice President APEX Meetups HTMLDB just a toy?, SIOUG, Portorož 2004 APEX