NCEP HPC Transition. 15 th ECMWF Workshop on the Use of HPC in Meteorology. Allan Darling. Deputy Director, NCEP Central Operations

Transcription

1 NCEP HPC Transition 15 th ECMWF Workshop on the Use of HPC Allan Darling Deputy Director, NCEP Central Operations

2 WCOSS NOAA Weather and Climate Operational Supercomputing System CURRENT OPERATIONAL CHALLENGE 2

3 NCEP Operational Forecast Skill 36 and 72 Hour 500 MB over North America [100 * (1-S1/70) Method] 36 Hour Forecast 72 Hour Forecast IBM 701 IBM 704 IBM 7090 IBM 7094 CDC 6600 IBM 360/195 CYBER 205 CRAY Y-MP CRAY C90 IBM SP IBM P690 IBM P655+ IBM Power

4 NCEP HPC Growth TeraFLOP per operational system

5 HPC Production HWM Utilization (02/16/2010) February :00:00 00:56:00 01:52:00 02:48:00 03:44:00 04:40:00 05:36:00 06:32:00 07:28:00 08:24:00 09:20:00 10:16:00 11:12:00 12:08:00 13:04:00 14:00:00 14:56:00 15:52:00 16:48:00 17:44:00 18:40:00 19:36:00 20:32:00 21:28:00 22:24:00 23:20:00 5

6 150 Production 08/27/ % Utilized (69.77% Prod/11.33% Dev) :00 00:27 00:54 01:21 01:48 02:15 02:42 03:09 03:36 04:03 04:30 04:57 05:24 05:51 06:18 06:45 07:12 07:39 08:06 08:33 09:00 09:27 09:54 10:21 10:48 11:15 11:42 12:09 12:36 13:03 13:30 13:57 14:24 14:51 15:18 15:45 16:12 16:39 17:06 17:33 18:00 18:27 18:54 19:21 19:48 20:15 20:42 21:09 21:36 22:03 22:30 22:57 23:24 23:51 6

7 May

8 FY11 On-Time Product Generation % Average On-Time Product Generation - Oct-July /27/2010: Storage failure 2/16/2011: Failed node /6/2010 and 10/7/2010: Job failures

9 FY12 On-Time Product Generation 99.35% Average On-time Product Generation - Oct Aug 2012 Impact worsened due to system load /7/2012, 2/8/2012 & 2/9/2012: System performance issue 2/18/2012: System hardware failure 6/9/2012: Model failure 8/28/2012 & 8/29/2012: System management loading issue /14/2012: Job failure 8/16/2012 & 8/17/2012: Hardware repair issue /18/2011 & 10/19/2011: Implementation error. 2/1/2012: DNS issue /10/2011: Facility power outage, requiring emergency system switch 12/24/2011 & 12/25/2011: Model failure 2/6/2012: HW issues leads to emergency system switch. 2/16/2012: Hardware issue 8/1/2012: Power loss at facility and emergency system switch /14/2012 & 2/15/2012: Implementation error 3/20/2012: Hardware 5/13/2012: Storage failure & emergency system switch

10 WCOSS NOAA Weather and Climate Operational Supercomputing System CONTRACT & SYSTEM OVERVIEW 10

11 WCOSS Contract Overview Sustains current operational systems during build, acceptance and transition to new systems Bridge contract expires September 2013 One 5-year base period, one 3-year option period, one 2-year transition option period Provides phased delivery of new Primary and Backup systems in the 5-year base period Will transition from IBM/Power6/AIX to IBM/Intel/Linux by end of FY13 Provides new sites Primary Reston, VA Backup Orlando, FL 11

12 Bridge Contract Current NOAA Weather and Climate Operational Supercomputer System Location Primary Gaithersburg, MD (IBM provided facility) Backup Fairmont, WV (GFE NASA IV&V facility) Configuration Identical Systems (per site) IBM Power 6/P575/AIX 73.9 trillion calculations/sec 5,314 processing cores 0.8 petabytes of storage Performance Requirements Minimum 99.0% Operational Use Time Minimum 99.0% On-time Product Generation Minimum 99.0% Development Use Time Minimum 99.0% System Availability Failover tested regularly Significance Where United States weather forecast process starts for the protection of lives and livelihood Produces model guidance at global, national, and regional scales Examples: Hurricane Forecasts Aviation / Transportation Air Quality Fire Weather Contract ensures no gap in operations Inputs and Outputs Processes 3.5 billion observations/day Produces over 15 million products/day 12

13 WCOSS Contract Future NOAA Weather and Climate Operational Supercomputer System Location Primary Reston, VA (IBM provided facility) Backup Orlando, FL (IBM provided facility) Configuration Identical Systems (per site) IBM idataplex/intel Sandy Bridge/Linux 208 trillion calculations/sec 10,048 processing cores 2.59 petabytes of storage Performance Requirements Minimum 99.9% Operational Use Time Minimum 99.0% On-time Product Generation Minimum 99.0% Development Use Time Minimum 99.0% System Availability Failover tested regularly Significance Where United States weather forecast process starts for the protection of lives and livelihood Produces model guidance at global, national, and regional scales Examples: Hurricane Forecasts Aviation / Transportation Air Quality Fire Weather Inputs and Outputs Processes 3.5 billion observations/day Produces over 15 million products/day 13

14 Key Performance Metrics Bridge At least 99.0% operational use time WCOSS At least 99.9% operational use time WCOSS and Bridge At least 99.0% system availability Sustain on-time generation of model guidance products at a rate of 99% or better; within 15 minutes of target completion times No degradation in product delivery to occur during planned, routine failover testing between the Primary and Backup WCOSS Delivery of committed computing capability upgrades objectively measured and accepted through the use of benchmark suite 14

15 WCOSS Locations 15

16 System Characteristics (per site) Bridge System WCOSS Phase 1 WCOSS Phase 2 Life Cycle Architecture OS Average Capability Average Capacity Cores TeraFLOP Oct Sep 2013 Power6 AIX X 1.0X 5, TF Planned accept Dec 2012 FOC Aug 2013 Planned accept Dec 2014 FOC Jul 2015 idataplex idataplex Linux (RHEL) Linux (RHEL) *2.0X over Bridge P6 *1.9X over Phase 1 *Estimated values; assumes FY13 NWS Reallocation Budget *2.3X over Bridge P6 *4.4X over Phase 1 10, TF *44,400 *920 TF 16

17 System Characteristics cont. (per site) Bridge System WCOSS Phase 1 WCOSS Phase 2 Life Cycle Operational Use Time Requirement Storage Useable Power KW Floor Space Oct Sep % 0.80 PB 689 KW 3,100 SF Plan accept Dec 2012 FOC Aug 2013 Plan accept Dec 2014 FOC Jul % 2.59 PB 469 KW 4,060 SF 99.9% *7.2PB *1050 KW 4,060 SF * Estimated values; assumes FY13 NWS Reallocation Budget 17

18 WCOSS HPC Growth TeraFLOP per operational system Built into the contract are capability and storage increases approximately every 3 years with no increase in funding NOAA modeling plans rely heavily on built-in contractual increases Projected

19 WCOSS NOAA Weather and Climate Operational Supercomputing System SCHEDULE & TRANSITION 19

20 Bridge Contract, WCOSS Contract and Transition from Bridge to new WCOSS Services FY11 FY12 FY13 FY14 Previous Contract Operations Transition activities Date Bridge Contract New WCOSS Contract Milestone Major Milestones Aug 2011 Sept 2011 Nov 2011 Apr 2012 Sep 2012 Nov 2012 Dec 2012 May2013 July 2013 Aug2013 Sep 2013 Awarded 2-year Bridge Contract Previous WCOSS contract expired Awarded new WCOSS ID/IQ Contract Installed early access system & started early testing with new architecture Complete installation of Primary system (Reston, VA) Complete installation of Backup system (Orlando, FL) Acceptance of all systems completed Achieve Authority to Operate (ATO) for new WCOSS (NOAA8866) Complete transition porting, testing, validation and tuning Go operational on new WCOSS (Ops on new WCOSS Contract) Close out Bridge contract 20

21 Transition End-to-End Validation Development Organizations Porting, Testing and Source Code Certification - $DEV Branch Chiefs certify New baseline of software systems and applications NCEP Operations & Development Organizations Production Validation - NCO/PMB Branch certify Operational products equivalence NCEP Operations & Development Organizations & Customers Customer Validation - NCO Director certify GO LIVE - Using the WCOSS Transition Test Checklist & Certification Document: - Part I:Approval for planned changes - Part II: Evaluation and certification - Vertical structure - Re-compile - Integrate into production scheduler - Tuning for resources and timing - Post production/ product generation - Validate output - Downstream testing for other software systems - Vertical structure - Dataflow actions - Validate network capacity - Notify customers - Receive customer feedback - Make decisions and changes based on customer feedback 21

22 WCOSS NOAA Weather and Climate Operational Supercomputing System SYSTEM DETAILS 22

23 Phase 1 and 2 Software Overview Red Hat Linux OS IBM xcat IBM Parallel Environment including runtime and developer editions Platform Computing Load Sharing Facility (LSF) ECMWF ecflow Workflow Manager IBM General Parallel Filesystem (GPFS) Intel Cluster Studio Rogue Wave TotalView Debugger IDL Community Supported Software, e.g., BUFR, GRIB, netcdf, GEMPAK, GIS, GoogleMaps, GrADS, NCAR Graphics, VIS5D, Subversion 23

24 System Architecture IBM dx360m4 idataplex server based on the Intel Sandy Bridge Processor Clustered with a Mellanox InfiniBand FDR fabric Cluster components include the compute nodes, login nodes, high-performance interconnect, and the management infrastructure Configured with hot spares for high system availability Storage based on the IBM DCS3700 disk subsystem and GPFS Includes services to address the implementation, acceptance testing, and training for the transition from the current to new systems 24

25 Phase 1 System Architecture 25

26 Phase 2 System Architecture 26

27 WCOSS NOAA Weather and Climate Operational Supercomputing System WRAP-UP 27

28 Transition Key Considerations Schedule constraint with high impact consequences Transition before bridge contract expires in September 2013 Current operational systems at peak load nearing end of life Resource dependencies external to investment Transition highly dependent on resources from the NOAA Data Assimilation & Modeling/Science investment Technical NOAA transitioning operational software systems/applications from legacy systems (Bridge) to a new/different architecture (WCOSS) Technical challenges with integration of IBM s new hardware and system software architecture 28

29 Summary System growth curve delayed but resuming and accelerating (for now) Model implementations continued at the cost of system recoverability Contract architecture allows for agile response to budget changes System architecture designed with agility to match contract capability 29

30 WCOSS NOAA Weather and Climate Operational Supercomputing System QUESTIONS 30