High Availability and Performance Computing: Towards non-stop services in HPC/HEC/Enterprise IT Environments

Transcription

1 CSC469/557: Winter 2006 High Availability and Performance Computing: Towards non-stop services in HPC/HEC/Enterprise IT Environments Chokchai (Box( Box) ) Leangsuksun, Associate Professor, Computer Science Director, extreme Computing Research Group Center for Entrepreneurship and Information Technology Louisiana Tech University Box s s 1 minute Bio MS and PhD in CS (1995): MS Thesis: Parallel C compiler PhD Thesis: Resource management/allocation in Heterogeneous Parallel Distributed Computing 7 years in Highly Reliable Software/system industry (Lucent) Architect, PM, Tech lead (15-30 team size) R&D -> > 4 major network management products Associate Professor in CS since graduate students (4 PhD) Collaborations with national and industry labs (e.g.ornl, Intel, Ericsson, Dell, CRAY etc) This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary. 1

2 Tux idol contestants tux02 tux03 cenit Outline z z Background and Motivation Current R&D & Educational projects ¾ ¾ ¾ z HA-OSCAR Architecture, infrastructure and System management Design & Dependability Analysis Education in HAPC Conclusion cenit This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary. 2

3 High Performance Computing HPC- Hardware and Software techniques devised, for building computer systems to quickly perform large amounts of computation HPC is not the same as high throughput Example of HPC app Page is excerpted from David Klepacki s presentation This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary. 3

4 Example of HPC app Page is excerpted from David Klepacki s presentation HPC goes mainstream Dual core is available now Multi core is imminently available At SC 05, Bill Gates gave a keynote as HPC goes mainstream MS are in HPC cluster (windows) More critical applications requires HPC This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary. 4

5 How to achieve HPC Work hard add more powerful unit(s). Faster CPU More CPU, parallel architecture Faster connectivity Work smart better algorithms to take advantage of parallelism Multiple-programming programming processing (Unix fork) Multi-threading threading Parallel programming (MPI, openmp,, PVM) Parallel Architectures Hardware Architectures for HPC and their Parallel Programming Models Distributed Memory Systems, MPP, Clusters - Message Passing Shared Memory Systems (SMP), Shared Memory Programming Specialized Architectures, Vector Processing Data Parallel Programming (SIMD). The Grid, Grid Computing This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary. 5

6 TOP500.org The chart is from top500.org TOP500.org The chart is from top500.org This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary. 6

7 TOP500.org The chart is from top500.org TOP500.org The chart is from top500.org This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary. 7

8 TOP500.org The chart is from top500.org TOP500.org The chart is from top500.org This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary. 8

9 TOP500.org The chart is from top500.org This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary. 9

10 Stop here: 11/30/2005 Production HPC system in the real world. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary. 10

11 Availability density of each node in the ASC White (was one of the top HPC) The majority of the availability of the each node is above 0.95 with a few of them below A 1 Availability for White The average is 0.98, with standard deviation This indicates Availability that, 0.6 compared to others, Average= some nodes 0.4 If the runtime systems manifest STDEV= are outages not aware of 0.2 these nodes unreliability, more. it may result in low system total performance, extended application completion or 0 failure Nodes index Nodes MTTF density (in hours) MTTF The maximum is hours Nodes MTTF for White 0 The minimum is 230 hours Node index The average is 3923, with standard deviation 1217 Mean=3923 STDEV=1217 This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary. 11

12 Nodes Downtime (in hours) Nodes TDT for White Some failure events cost more time due to a prolonged repair process, 8000 and thus increase the total average 6000 TDT. TDT 4000 The average is 355 hours, with standard deviation 56 Mean=355 STDEV= Most of the total down time 451for each node is around 100 hours. Node index HPC Buildout High Performance Buildout High Availability High serviceability This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary. 12

13 Availability A measurement represents a ratio of uptime vs. total times High availability - ability of a system to perform its function continuously (without interruption) for a significantly longer period of time than the reliabilities of its individual components would suggest. High availability is most often achieved through fault tolerance. Degree of Availability System Type Unavailability (minutes/year) Availability (in percent) Availability Class Unmanaged 50, Managed 5, Well-managed Fault-tolerant High Availability Very High Availability Ultra Availability This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary. 13

14 Quiz. Find out for each 9 s9 System Type Unavailability (minutes/year) Availability (in percent) Availability Class Unmanaged 50, Managed 5, Well-managed Fault-tolerant High Availability Very High Availability Ultra Availability Quiz #2 Say if the machine costs $20M Availability is 92.1% What is the downtime and cost of downtime? This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary. 14

15 Unavailability = No performance and functionality Availability enables Performance Performance oriented Enterprise/Shared Major computing resources- 7/24/365 Losses of $195K - $58M with 3.5 hrs (Meta Group report) Service provider Regulation/Mandate FCC mandate (Class 5 local switch) Losses time and opportunities Life-threatening National Security (Home Land defense) What involves (non-stop services)? users Operation Config/Mgt Tool Training HW App/Service??? This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary. 15

16 Goals Towards Non-stop services in HPC/HEC environments High Availability (Reliability) High Serviceability (planned downtime) High Performance Computing (HPC) We want them all HA-OSCAR + : High Availability and Performance Computing Architecture and Management System +Open Source Cluster Application Resources (HAPC infrastructure and System management) This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary. 16

17 Beowulf: Linux HPC Cluster Unavailable = No performance and functionality pbs, maui, SGE, nfs, etc HA-OSCAR Goals High Availability for HPC cluster Serviceability - Simplicity Transparent - Preserve existing investments, No change required, retrofitable Production-quality software release This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary. 17

18 Related Works Serviceability easy Beowulf Builders OSCAR, ROCKS, Scyld High Availability LinuxHA, Kimberlite, HP ServiceGuard, Redhat AS Monitoring Mon, netsnmp., Ganglia, Clumon, dproc HA-OSCAR Beowulf Optional Image servers This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary. 18

19 Key solutions High Availability Redundancy (at the head/service node) Monitoring & Self-healing (auto detection and recovery) Critical services (pbs( scheduler, maui,, grid, SGE, network, authentication, member etc) Resources (filesystem( filesystem,, cpu usage etc) Hardware and network Higher Serviceability Self-build, self-configuration Hot-upgrade cluster OS or services. Self-healing Schemes This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary. 19

20 Monitoring and recovery Enhancement based kernel.org MON, IPMI, and net-snmp framework Recovery Associative Response Local recovery, e.g. restart, checkpoint Failover (simple( or impersonate/clone) Admin-defined defined actions Adaptive Response Previous state and number retry Acceleration (Time-series) E.g. maui dies, restart. After 3 times reties within 3 mins, failover Adaptive recovery state diagram previous state, # counter, recovery failure Detect switch over & take control at the standby working threshold reached Alert. previous state, # counter, recovery after # retry Failover After the primary node repair, then optional Fallback This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary. 20

21 HA-OSCAR Cluster Self-build & Self-configuration How to build HA-OSCAR Can retrofit an existing Linux Beowulf Or start with OSCAR/ROCKS installation tool Wizard GUI based installation This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary. 21

22 HA-OSCAR installation Adopt ease of build (self-build, config w/o OS pre-loaded) 30 min 1.5 hrs installation (retrofit) Take almost the same time for disaster recovery step1 Step2 create head image Step3 Input and clone image Step5 (optional) web admin to add/config more services Step4 config & build Standby Hardware Management abstraction IPMI (Intelligent( Platform Management Interface) A widely accepted specification for server management. Defines an abstract interface for intelligent hardware that monitor the health of server s s hardware inventory. Alerting, power control, asset tracking, health monitoring, event logging are IPMI s management strategies. open IPMI and OpenHPI (SA forum) This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary. 22

23 Our early observations Can set thresholds in managed elements to trigger events with severity levels Automatic failure trend analysis -> prediction 01/25/ :31:19 Sys Fan 1 critical 01/25/ :31:19 Sys Fan 3 critical 01/25/ :31:19 Sys Fan 4 critical 01/25/ :31:19 Processor 1 Fan ok 01/25/ :31:20 Processor 2 Fan ok A failure prediction & policy-based recovery management Detections - the damage done! Predictions trend analysis Anticipate imminent failures Better handling More difficult for multiple events Example of IPMI events and trend analysis E.g. CPU temp raising too fast with 5 min -> prepare to checkpoint, failover and restart Memory bit error detected -> > take a node out This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary. 23

24 HA-OSCAR monitoring and recovery Restructure 01/25/ :31:19 Sys Fan 1 critical 01/25/ :31:19 Sys Fan 3 critical 01/25/ :31:19 Sys Fan 4 critical 01/25/ :31:19 Processor 1 Fan ok 01/25/ :31:20 Processor 2 Fan ok Retrieving os/sw states from (dead?) systems Storing/retrieving os/sw states on the local storage in stealth mode, e.g, message log (process state saving) Ability to resume application execution from dead nodes Cluster Passive monitoring "A Report of Invention has been filed and Tech is preparing a patent application." This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary. 24

25 HA-OSCAR Status HA-OSCAR beta was released in March 2004 Failure Prediction and policy based recovery proof of concept Prediction for known problems or events Single event type can be anticipated Active monitoring Head node is achievable Challenging for very large scale compute nodes and multi-events Automatic/Transparent Checkpoint/restart recovery for MPI app (HA-OSCAR + FT LAM/MPI) Roadmap Port to other platforms (e.g.?? SGI, cray) Federated HEC systems (see HEC future work) More sophisticate trend analysis and predictions (multi event correlations) Grid-aware HA-OSCAR Support other cluster distributions (ROCKS, SCYLD) Multi-head n+1 active-active Hot-upgrade cluster (OS/CMS) Fault-tolerant tolerant applications/services framework FCAPS Management & carrier grade This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary. 25

26 Three HA-OSCAR flavors Monitoring Pbs maui nfs httpd SGE NSF nis httpd gmond,gmetad Heartbeat (3 sec) CPU Fan Speed CPU Temperature CPU status Memory bit error HA-OSCAR Active-Hot Standby IPMI option IPMI option IPMI option IPMI option HA-ROCKS Active-Hot Standby (lab grade) IPMI option IPMI option IPMI option IPMI option HA-OSCAR 2+1 Active- Active (lab grade) IPMI option IPMI option IPMI option IPMI option Appeared in a front cover in two major Linux magazines, various technical papers, research exhibitions. web site: This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary. 26

27 HA-OSCAR DEMO at the end Design and Dependability Analysis Research This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary. 27

28 Reality Checks go to modelingm Great! We got HA Beowulf! But How much improvement? The total uptime? Aggregate Performance? Analytical model and prediction Statistical technique to compare uptime How many 9 s? 9 (downtime per/year) Stochastic Reward Net with SPNP package Identical hardware parameters between Beowulf and HA-OSCAR multi-heads Availability Model Server up Server down & repair Availability model S1 S1 S1&S2 S2 time HA-OSCAR dual head model This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary. 28

29 HA-OSCAR SRN model Server sub-model Switches Compute nodes Server Sub Model P Server up P Server down Failover P server repair Failback S is up and ready S takes control S Server down S repair This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary. 29

30 Instantaneous Availability Steady (A) = (36 min) vs. Beowulf (A) = (30 hr) More analysis & comparisons Availability HA -OSCAR solution vs traditional Beowulf MTTF(hours) Beowulf HA-oscar Both unplanned & planned downtime Schedule downtime=200 hrs Repair time = 24 hrs Monitoring intervals=10s Ours.9999 vs ~ K vs. 10M TFLOB (1T system) $ 70K vs $2 M ($20M system) Lost investment due to unavailability (based on $20M) HA -OSCAR solution vs traditional Beowulf Unavailable performance (based on 1 Tflop machine in a year) m illion $ MTTF(hours) Beowulf HA-oscar Gflop MTTF(hours) Beowulf HA-OSCAR This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary. 30

31 Some HA-OSCAR measurements 3-55 sec Manual failover time 0.9% CPU usage at each monitoring interval 300 HA-OSCAR Network load in Packets/Min measured by TCPtrace HA-OSCAR Mon polling interval (s) Comparison of network usages for HA-OSCAR different polling sizes This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary. 31