The Center for High Performance Computing Dell Breakfast Events 20 th June 2016 Happy Sithole
Background: The CHPC in SA CHPC
User Community: South Africa CHPC Existing Users Future Users
Introduction of CHPC CHPC is a national facility within the NICIS providing HPC resources and services to academia and industry. A broad range of Science and Engineering domains use HPC. The Center is funded by the Department of Science and Technology. Building capacity within the country and continent as part of the SKA project.
Vision An accomplished and preferred partner for HPC solutions in the country and the region Mission Provide world-class HPC that enable cuttingedge research with high impact on the South African economy
CHPC New Infrastructure System Configuration Phase 1 Dell PowerEdge C6320 Servers: Standard Compute nodes 128GB memory/ node 1 008 2 x Intel Xeon E5-2690 v3, 2.6 GHz (Haswell) processors (12 Cores Each 24 cores / node) Dell PowerEdge R930 servers: 24 192 Large Memory Compute Nodes 1024GB / node (FAT nodes) 5 4 x Intel Xeon E7-4850 v3 processors, 2.2 GHz (14 Cores Each 56 cores / node) Infiniband FDR 2:1 non-blocking (56 Gbps) Parallel Storage (Useable) PB 4 Total Number of Racks (including Compute, Login, Management and Storage Nodes) Centos 7.1 Measured Linpack Compute (Tflop/s) 782.9 280 19
LENGAU Launch
Lustre Parallel File System@CHPC Intel Lustre 2.5.4 - Intel Management for Lustre (IML) - 4PB useable IOR Results
Future Upgrades Phase 2 upgrade to be completed by end of October 2016. Initial power restriction removed, now 2 MWatts of power available to the data center. Cooling configuration for the additional servers could be considered (for example on DLC or immersion) Processor Roadmap (including accelerators) coupled with Dell Server Roadmap will assist the technology decision for the upgrade. Interconnect fabric scalable to 1PFLOP from initial design. Storage expansion to be considered in conjunction with archiving options. Optimisation of fan speed on the servers is necessary.
Fluent F1 racecar
WRF-3.7, Intel vs gcc
CASTEP Tsessebe Lengau Before optimization Tsessebe Initialisation time = 6.33 s Calculation time = 1033114.91 s Finalisation time = 72.00 s Total time = 1033193.24 s Peak Memory Use = 710728 kb it took approximately 286 hours ~ 11 days. Overall parallel efficiency rating: Terrible (16%) After optimization Email: pngoepe@ul.ac.za after optimization Lengau Initialisation time = 2.58 s Calculation time = 64277.31 s Finalisation time = 9.00 s Total time = 64288.88 s Peak Memory Use = 986236 kb It took approximately 18 hours =3/4 day. Overall parallel efficiency rating: Very good (82%)
60 km grid spacing 8 km grid spacing Ultra high-resolution climate modelling on the CHPC clusters 1 km grid spacing Computational resources determine grid spacing Subgrid-processes parameterised Parameterisations source of uncertainty
Extending CSIR climate modelling to Africa on the CHPC clusters Tailor-made projections for Mozambique, Ethiopia, Egypt, Nigeria and Namibia currently running on new CHPC cluster CCAM applied in stretchedgrid mode 8 km simulations nudged in lower resolution CORDEX simulations 1 km resolution simulations over of 200 x 200 km^2 nudged within 8 km res simulations C160 stretched-grid with 8 km horizontal resolution over a domain of about 1500 x 1500 km^2, centered over Port Said
Simulation period: 2009-2012 Multiple nudging: ERA reanalyses 75km to 8km (1300kmx1300km domain size) 1km (150kmx150km)
Industry Initiatives What are the Drivers? Highly scalable methods for modelling and simulation exploit massive parallelism and data visualisation. Heterogeneous computing environments require new programming models.. Emerging real-time, data-intensive applications require robust HPC capability. Energy requirements increase industry overheads. Tax incentive models from government. SASOL De Beers Johnson Matthey ESKOM XTRATA ECJIV NNR TOTAL MINTEK HATCH Virtual Prototypes Offerings Applications are made to scale. HPC is accessible to businesses and applications of all types. HPC is effectively utilised. Easier to migrate applications to other HPC systems.
African Data Intensive Research Cloud
Human Capital Development Initiatives Annual Winter School targeting new post-graduate researchers and focus on basics of HPC and induction into HPC facilities, hosted in conjunction with Universities around the country. Annual Introductory Scientific Programming School focusing on parallel programming and python packaged for post-graduate students. Domain focused workshops arranged in collaboration with research communities and tailored to meet the needs of the researchers. New architecture, such as the heterogeneous computing environments (GP-GPU and Xeon-Phi). System Administrator training with hands-on configuration on HPC system. Providing administrators with smaller HPC systems that they manage at their own institutions.
Student Cluster Challenge The Student Cluster Challenge is aimed at training students in HPC. The program is focused at undergraduate students, from all S&E disciplines. National selection process starts in June (Winter School) and provides teams that compete in December. Normally 10 teams of 4 students compete. The winning team in the national competition + 2 best from runner-ups form a team that represent South Africa in the International competition. The team spends a week at Dell Research Laboratory in Austin in January for further training. Dell is the major sponsor with hardware and logistics for both local competition and international team. Mellanox provide the interconnect. The students are recruited to different HPC entities in the country (For example, CHPC, SKA etc..)
The CHPC Team Wins!! (Added post ISC)
Noteworthy Projects Dell ICT Academy announced in 2015 through equity equivalence. Building already completed and the first intake of students (10 high school graduates and 5 graduates). Some of the graduates from the academy will form Dell Channel partners with strong focus on HPC system integration. Focus is on enterprise system and high performance computing. Discussions to include a team from Academy in SC program. CHPC Solution Center to focus on testing of new technologies with Dell in conjunction with other solutions centers such as Cambridge University and PISA.