Intel HPC Orchestrator System Software Stack Providing Key Building Blocks for Intel Scalable System Framework

Transcription

1 Intel HPC Orchestrator System Software Stack Providing Key Building Blocks for Intel Scalable System Framework Nivi Muthu Srinivasan Technical Support Manager, Intel Corporation 1

2 Legal Notices and Disclaimers Intel technologies features and benefits depend on system configuration and may require enabled hardware, software or service activation Learn more at intelcom, or from the OEM or retailer No computer system can be absolutely secure Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions Any change to any of those factors may cause the results to vary You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products For more complete information visit Cost reduction scenarios described are intended as examples of how a given Intel- based product, in the specified circumstances and configurations, may affect future costs and provide cost savings Circumstances will vary Intel does not guarantee any costs or cost reduction No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document Intel, the Intel logo and others are trademarks of Intel Corporation in the US and/or other countries *Other names and brands may be claimed as the property of others 2016 Intel Corporation 2

3 Agenda Background Why a community solution is needed OpenHPC* Intel HPC Orchestrator Overview Architecture Value Add Installation Summary 3

4 Current State of System Software Efforts in HPC Ecosystem Fragmented efforts across the ecosystem Everyone building their own solution A desire to get exascale performance & speed up software adoption of hardware innovation New complex workloads (ML 1, Big Data, etc) drive more complexity into the software stack THE REALITY: We will not be able to get where we want to go without a major change in system software development 1 Machine Learning (ML) 4

5 Community Effort to Realize Desired Future State A Shared Repository Stable HPC Platform Software that: Fuels a vibrant and efficient HPC software ecosystem Takes advantage of hardware innovation & drives revolutionary technologies Eases traditional HPC application development and testing at scale Extends to new workloads (ML, analytics, big data) Accommodates new environments (ie, cloud) 5

6 OpenHPC* to Intel HPC Orchestrator to Intel Scalable System Framework OpenHPC Intel HPC Orchestrator Open Source Community under Linux Foundation Ecosystem innovation building a consistent HPC SW Platform Platform agnostic Multiple distributions Intel s distribution of OpenHPC Expose best performance for Intel HW Advanced testing & premium features Product technical support & validated updates Intel Scalable System Framework 6

7 Intel Scalable System Framework A Holistic Design Solution for all HPC needs Small Clusters Through Supercomputers Compute and Data-Centric Computing Standards-Based Programmability On-Premise and Cloud-Based Intel Xeon Processors Intel Xeon Phi Processors Intel Xeon Phi Coprocessors Intel Server Boards & Platforms Intel Solutions for Lustre * Intel Optane Technology 3D XPoint Technology Intel SSDs Intel Omni-Path Architecture Intel True Scale Fabric Intel Ethernet Intel Silicon Photonics Intel HPC Orchestrator Intel Software Development Tools Intel Supported SDVis 7

8 Intel HPC Orchestrator Overview 8

9 High Performance Computing Terminology Term Head Node Compute Node Login Nodes Router Nodes Management Control Nodes Fabric Management Network Power Control / Console Network Definition Main Admin interface to the system, orchestrates the rest of the system Nodes dedicated to the batch processing of user applications Main User interface to the system, for job submission, compiles, etc Centralized connectivity to filesystems and networks external to the system Nodes that take some of the responsibilities of the Head Node and distribute and parallelize them across the system allowing for additional scaling High performance/low latency network utilized primarily by user apps, and for access to parallel filesystems (Lustre*) Network over which the Head Node controls the rest of the system Used for Provisioning, Resource Management, Time Synchronization, etc An optional additional network for security to isolate power and console functions from the user accessible parts of the system 9

10 Fabric Mgt System Diagnostics Provisioning System Management (Config, Inventory) Intel HPC Orchestrator Overview Intra-stack APIs for customization/differentiation (OEMs) Defined external APIs for consistency across versions (ISVs) Operator Interface ISV Applications Applications (not part of the stack initially) Data Collection And System Monitors DB Schema Scalable DB Workload Manager Resource Mgmnt Runtimes Optimized I/O Libraries I/O Services Scalable Debugging & Perf Analysis Tools High Performance Parallel Libraries Compiler and Programming- Model Runtimes Software Development Toolchain User Space Utilities Overlay & Pub-Sub Networks, Identity Linux Distro Runtime Libraries Node-specific OS Kernel(s) Hardware 10

11 Intel HPC Orchestrator System Architecture Head Node DB Provisioning Resource Management Fabric Manager Power Control Console Access Monitoring Login Nodes User Access NFS / LAN / Internet Privileged Access Eth Fabric Perimeter Planned Router Nodes Lustre Parallel File System 11

12 Intel HPC Orchestrator Highlights What Provisioning Method Provisioning Time (128 nodes) How Image based (vs Package based) ~4-5 Minutes Configuration Site-specific inputs (IP addresses, MAC addresses, BMC addresses, Provisioning interface, etc) Supported Distros Red Hat Enterprise Linux* - RHEL 72 SUSE Linux Enterprise Server* - SLES12 SP1 Install Procedure Fully scripted, Intel distributes iso Component Distribution Install Interface Standard YUM/Zypper RPM repository Command line 12

13 Intel HPC Orchestrator High-Level Component List Functional Areas Components Base OS compatibility RHEL 72, SLES12 SP1, CentOS 72 Administrative Tools Provisioning Resource Mgmt I/O Services Numerical/Scientific Libraries I/O Libraries Compiler Families MPI Families Development Tools Performance Tools Conman, Ganglia, Intel Cluster Checker, Lmod, Losf, Nagios, pdsh, prun Warewulf Slurm, Munge Lustre* Client (Intel Enterprise Edition for Lustre* software) Boost, GSL, FFTW, Metis, PETSc, Trilinos, Hypre, SuperLU, Mumps, Intel MKL** HDF5 (phdf5), NetCDF (incl C++ & Fortran libraries), Adios GNU (gcc, g++, gfortran), Intel Parallel Studio XE (icc, icpc, ifort)** OpenMPI, MVAPICH2, Intel MPI** Autotools (autoconf, automake, libtool), Valgrind, R, SciPy, Numpy PAPI, Intel IMB**, mpip, pdtoolkit, TAU, Intel Advisor**, Intel Trace Analyzer & Collector**, Intel Vtune Amplifier** ** Additional license required 13

14 Intel HPC Orchestrator Enhancements Advanced integration testing & extensive validation Professional support for - All Intel components - Components where Intel maintains a support contract Best effort support for all other components Enhanced documentation - Readme, Release Notes - Components Guide - Troubleshooting Guide, including Knowledge Base - Technical Update - Security Best Practices - Architecture 14

15 Intel HPC Orchestrator Enhancements Validated patches & updates Early new hardware integration with System Software Inclusion of proprietary Intel Software - Intel Parallel Studio XE 2017 Cluster Edition - Intel Enterprise Edition for Lustre* software (client) SLES 12 SP1 Base OS redistribution Integrated Test Suite Intel Cluster Checker Baseline Extensions Planned Additional components - Support for high availability - Visualization tools 15

16 Intel Cluster Checker Baseline Extensions New set of extensions to Intel Cluster Checker Baseline: system data collected when it is in a good, dependable state Collects baseline data for: RPMs Virtual Node File System Configuration files (along with whitelist/blacklist capabilities) Hardware/Firmware Compare current state of system with baseline 16

17 Intel Cluster Checker Baseline Extensions Collecting RPM baseline data - Create nodefile # cat nodefile [sms]# cat nodefile sms #role: head c1 c2 - Run rpm-baseline command # rpm-baseline f <path-to-nodefile> - Data captured in /var/tmp/rpms-baselinetxt RPM name Node name [sms]# cat /var/tmp/rpms-baselinetxt sms, libpciaccess el7 x86_64 c1, libpciaccess el7 x86_64 c2, libpciaccess el7 x86_64 Version Release Architecture 17

18 Intel Cluster Checker Baseline Extensions Collecting Files baseline data # files-baseline f <path-to-nodefile> Data captured in /var/tmp/files-baselinetxt [sms]# cat /var/tmp/files-baselinetxt sms, /etc/sysconfig/httpd, -rw-r--r--, root, root, cfc1df04aebc4df81983e1f5 c1, /etc/os-release, -rw-r--r--, root, root, 1359aa3db05a a f3 c2, /etc/sysconfig/munge, -rw-r--r--, root, root, e0505efde717144b039329a6d32a798f Permissions File Owner Group MD5 Sum 18

19 Intel Cluster Checker Baseline Extensions Collecting Hardware baseline data # hw-baseline f <path-to-nodefile> Data captured in /var/tmp/hw-baselinetxt [sms]# cat /var/tmp/hw-baselinetxt sms, 00:0d0, Intel Corporation 82801HM/HEM (ICH8M/ICH8M-E) SATA Controller [AHCI mode] c1, 00:030, Intel Corporation 82540EM Gigabit Ethernet Controller c1, 00:070, Intel Corporation 82371AB/EB/MB PIIX4 ACPI c2, 00:050, Intel Corporation 82801AA AC 97 Audio Controller Bus:DeviceFunction Hardware description 19

20 Intel Cluster Checker Baseline Extensions Comparing/Analyzing: - Collect current system state # clck-collect f <path-to-nodefile> m uname m files_head m files_comp - Analyze current system state against baseline # clck-analyze f <path-to-nodefile> I files 1 undiagnosed sign: 1 The file /etc/pamd/ppp has been added since the baseline was generated [ Id: files-added ] [ Severity: 25%; Confidence: 90% ] [ Node: RHEL2 ] This analysis took seconds FAIL: All checks did not pass in health mode 20

21 Installation 21

22 Basic Cluster Install Example Lustre* storage system Starting install guide/recipe targeted for flat hierarchy Leverages image-based provisioner (Warewulf) - PXE 1 boot (stateless) - optionally connect external Lustre * file system Obviously need hardware-specific information to support (remote) bare-metal provisioning 1 Preboot execution Environment (PXE) Data Center Network eth0 Master (SMS) eth1 high speed network tcp networking Figure 1: Overview of physical clust er architecture compute nodes to compute eth interface to compute BMC interface the local data center network and eth1 used to provision and manage t he clust er backend (not e that interface names are examples and may be di erent depending on local settings and OS conventions) logical IP interfaces are expect ed to each compute node: t he first is the standard Ethernet interfac will be used for provisioning and resource management The second is used to connect to each host s and is used for power management and remote console access Physical connectivity for t hese two IP networks is often accommodat ed via separat e cabling and switching infrastruct ure; however, an alt configurat ion can also be accommodat ed via the use of a shared NIC, which runs a packet filter to management packets between the host and BMC In addition to the IP networking, there is a high-speed network (InfiniBand in this recipe) that connected to each of t he hosts This high speed network is used for application message passing and opti for Lustre connectivity as well 13 B r ing your own license OpenHPC provides a variety of integrated, pre-packaged elements from the Int el Parallel Studio XE ware suite Portions of the runtimes provided by t he included compilers and MPI component s are usable However, in order to compile new binaries using these tools (or access other analysis t oo Intel Trace Analyzer and Collector, Intel Inspector, etc), you will need t o provide your own valid l and OpenHPC adopts a bring-your-own-license model Note that licenses are provided free of char many categories of use In particular, licenses for compilers and development s tools are provided at n to academic researchers or developers contributing t o open-source software projects More informat this program can be found at : 22

23 Installation Walks a system administrator through the various steps needed to bootstrap a cluster Ease of Use: Describes what needs to be accomplished Followed by example commands Copy & paste the commands with site-specific inputs For faster integration: Edit site specific inputs file Execute install script Site-Specific Inputs + Install Script = Cluster 23

24 Minimum Requirements Hardware & Software 1 head node, at least 2 compute nodes: can ping each other Head node and compute nodes are clustered using Ethernet (if not fabric) Support at least one network file system: NFS (or) Lustre* Mechanism to boot the compute nodes: IPMI (or) Physical access to compute nodes Pre-defined local site inputs Hostnames for head & compute nodes IP & MAC addresses for head node & all compute nodes Name of interface that will be used for provisioning Subnet netmask for internal network 24

25 Install Script Expectations 1 head node that hosts an NFS file system, 4 stateless compute nodes IPMI can talk to the compute node BMCs Head node has 2 Eth interfaces eth0 to ensure external network connectivity eth1 to provision/manage cluster 2 logical IP interfaces to each compute node 1 st for provisioning & resource management 2 nd to connect to BMCs Wherever expectations!= system setup, editing the install script will be necessary 25

26 Installation using the iso file Desktop]# mkdir /mnt/hpc_orch_iso Desktop]# mount -o loop hpc-orchestrator-rhel betaiso /mnt/hpc_orch_iso/ Desktop]# cd /mnt/hpc_orch_iso/ Desktop]# ls groupxml noarch repodata x86_64 Desktop]# rpm -ivh x86_64/intel_hpc_orchestrator_release beta-61x6_64rpm Desktop]# yum -y install docs-orch beta-581x86_64rpm Desktop]# cp -p /opt/intel/hpc-orchestrator/pub/doc/recipes/vanilla/inputlocal Desktop]# cp -p /opt/intel/hpc-orchestrator/pub/doc/recipes/vanilla/recipesh Desktop]# mkdir sections Desktop]# cp -p /opt/intel/hpc-orchestrator/pub/doc/recipes/vanilla/sections/* sections/ Desktop]# export INPUT_LOCAL=/inputlocal Desktop]# /recipesh -f 26

27 Directory Structure Desktop]# cd /opt/intel/hpc-orchestrator Desktop]# ls admin pub images lmod modulefiles clck doc examples modulefiles advisor inspector compiler mpi NFS Mounted 27

28 Integration/Test/Validation Cross-package interaction Integrated Cluster Testing Integrated Cluster Testing Each end-user test need to touch all of the supported compiler & MPI families Abstracted to repeat the tests with different compiler/mpi environments: gcc/intel compiler toolchains Intel, OpenMPI, MVAPICH2 MPI families Software Software OpenHPC OpenHPC Hardware Orchestrator Hardware + + OS Distribution OS Distribution Hardware Intel HPC OS Distribution + + System Tools System Tools Resource Manager Resource Manager Provisioner Provisioner Dev Tools Dev Tools Perf Tools Perf Tools I/O Libs I/O Libs Mini Apps Mini Apps Parallel Libs Parallel Libs Compilers Compilers User Env Serial Libs User Env Serial Libs Individual Component Validation Individual Component Validation 28

29 Post Install Integration Tests - Overview Major components have configuration options to enable/disable Suite of integration tests Installed by default by the install script Root-level tests User-level tests: short & long versions Type of test Approximate running time Root-level single digit number of minutes User-level short less than 30 minutes User-level long a few hours Provisioning OOB Mgmt O/S Config Mgmt Resource Manager User Env Fabric Node-Level Perf I/O 3rd Party Libs Apps Root Level User Space 29

30 Post Install Integration Tests Root level tests 300

31 Post Install Integration Tests User level tests 311

32 Intel HPC Orchestrator: Summary Benefits OEMs reduce R&D ISVs/Developers reduce time and resources to constantly retest apps IT Admins - reduce R&D to build and maintain a fully integrated SW stack Integrated open source and proprietary components Modular build; Customizable; Validated updates Advanced integration testing, testing at scale Level 3 technical support provided by Intel Optimization for Intel Scalable System Framework components Available through OEM & Channel Partners in Q4 16 End Users - hardware innovation reflected in SW faster on path to exascale 322

33 333