Evolving HPC Solutions Using Open Source Software & Industry-Standard Hardware

Transcription

1 CLUSTER TO CLOUD Evolving HPC Solutions Using Open Source Software & Industry-Standard Hardware Carl Trieloff Red Hat, Technical Director Lee Fisher Hewlett-Packard, WW FSI-HPC Business Development 1

2 Financial compute to cloud example Scale up Grid Internal grid Messaging Scale out Scheduler Messaging Trader Latency External public cloud Trade execution 2 Internal private cloud

3 What are some of the requirements? Cloud computing is a hot topic, but many people have important questions and challenges they need addressed before they can adopt cloud: How do I build an internal cloud? How do I avoid lock-in to a single cloud? 3 How do I deal with homogeneous & non homogeneous hardware requirements? How do I mix, match, and blend different cloud resources including internal and external clouds? How do I manage a variety of applications and groups with different SLAs, priorities, and resource requirements across clouds? How do I manage software licensing/ hardware limits? How do I abstract resource management, accounting and permissions?

4 Red Hat Enterprise MRG Integrated platform for high performance distributed computing High speed, interoperable, open standard Messaging Deterministic, low-latency Realtime kernel High performance & throughput computing Grid scheduler for distributed workloads and Cloud computing 4

5 AMQP Messaging on 8-node HP Nehalem Infiniband 40Gps > 11 M mes/s Messages/Sec Broker 2 Broker Number of Brokers per Server 5 1 Broker HP-G6 Nahlem HP-G5 Harperton Improvement ratio

6 AMQP, HP Performance, scale up. Single HP Nehalem BL460c 40G Infiniband AMQP Perftest 12M M Messages/Sec M bytes 64 Bytes 256 Bytes 1024 Byt es 6M M M Broker 4 Broker 2 Broker 1 Broker Number of Brokers on the Server two BL460c G6 with two Intel(R) Xeon(R) X5570 CPUs per blade (Nehalem 2.93 GHz, 8MB L3 cache, 95W) Memory 24GB(6x4GB), Memory Type DDR3-1333, HT, Turbo 2/2/3/3) Infiniband 4X QDR IB Dual-port Mezzanine HCAs (1 port connected) Infiniband Switch BLc 4X QDR IB Switch 6

7 KVM Performance Only 5% of bare metal AMQP Messaging Intel Nahalem 2 10Gbit Vt-D > 1 M mes/s RHEL 5.4 KVM AMQP 2-Guest M M Messages / Sec M M M M Msg Size (bytes) Msg/sec Throughput MB/sec

8 MRG scheduling resources 8

9 MRG Messaging Infiniband RDMA Latency: Under 40 Microseconds Reliably Acknowledged MRG Messaging Latency Test on HP BL460c G6 Infiniband 100K Message Rate Average Latency (ms) Bytes RDMA Nehalem 256 Bytes RDMA Nehalem 1024 Bytes RDMA Nehalem

10 Components of the Solution Stack Solutions still matter in an industry-standard, open source world FSI-HPC Solution Stack Tuning & working in labs Red Hat MRG Tuning tools Red Hat MRG & RHEV Messaging/Grid/Virt HP Voltaire / Red Hat RDMA Red Hat MRG Realtime & KVM HP reduced SMI BIOS's HP compute & storage Application Environment Workload Middleware Integrated Systems Server Interconnect L2 Fabric Operating System BIOS X86-64 Server Architecture Determinism, and performance needs to work at each layer, HP & Red Hat are partnered across the stack 10 Services Red Hat / HP Systems Users

11 Hardware matters Scale-Up Blades Scale-Out Rack-Optimized SL6000 Today s RFP Metrics: Performance/Watt Performance/BTU Performance/Rack HP Low Latency Lab with MRG + Red Hat MRG Lab with HP BL460/BL685 & IB 11

12 Dealing with SMIs HP BIOS Option for Low Latency Apps Disable frequent SMIs used for Dynamic Power Savings Mode, CPU Utilization monitoring, P-state monitoring and ECC reporting Benefits both RHEL & MRG operating environments. Latency spikes with standard BIOS settings 12 Latencies when SMIs disabled in BIOS

13 MRG Realtime RHEL on HP systems Enables applications and transactions to run predictably, with guaranteed response times. Upgrades RHEL 5 to realtime OS Provides replacement kernel for RHEL5; x86/x86_64 Preserves RHEL Application Compatibility Certified on HP hardware, see Red Hat / HP certifications Response time Time 13

14 MRG Realtime Scheduling Latency Vanilla Min: Max: Mean: Mode: 9.00 Median: 9.00 Std. Deviation: MRG RT Min: Max: 4 43 Mean: 8.34 Mode: 8.00 Median: 8.00 Std. Deviation:

15 Networking matters Voltaire DDR and QDR InfiniBand: 36 QDR QSFP ports Ethernet mngt port LEDs USB port Serial port Test Configuration: Two Nehalem-based server w/ ConnectX PCI-E HCAs, back-to-back QDR ConnectX HCA running at QDR DDR ConnectX HCA running at DDR RHEL5 UPDATE 2 Mellanox VERBs Performance Test RoEE RDMA on Enhanced Ethernet RoEE is defined to be a verbs compliant IB transport running over the emerging IEEE Converged Enhanced Ethernet standard 15

16 Building Cloud capabilities with MRG Scalable Virtualization Schedule VMs directly as jobs via libvirt Powerful Policies Provision VMs via Red Hat Enterprise Virtualization Inject jobs into VMs Resource Accounting SLA's 16 Track resources via Condor's resource accounting Apply priorities and policies Apply security Authentication (e.g. SSL, ), Integrity, Encryption VMs run multiple concurrent instances, start on Black Friday or semi-monthly, re-run after fault Machines only run VMs from owner s group between 9 and 5, everyone else has a low priority shot from 5 to 9 Global control limiters (e.g. NFS mount users, licenses), Various Cloud Services IaaS clouds: run all workloads as VMs PaaS clouds: leverage job scheduling with VM scheduling

17 Aggregating & Bridging Clouds MRG includes the ability to schedule jobs and applications to multiple clouds, based on policy 17 MRG has the ability to send VMs to other resource managers MRG becomes the unified interface to many types of resources internal VM resources and multiple external clouds MRG's life-cycle management, accounting and policy benefits still available Use cases include Manage overflow/spillover Access to specialized resource managers Transformation between VM types/systems Allow a single app/stack to bridge multiple clouds

18 MRG Cloud Aggregation Architecture Schedd accepts jobs over SOAP, AMQP, CLI GAHP: Grid ASCII Helper Protocol 18 An adapter to an external resource manager Exist for many batch systems Exists for EC2-like resource managers Extensible to new resource managers Job Router transforms types, e.g. stack to VM to EC2 AMI

19 Durable Messaging Throughput comparison MRG Durable Messaging Throughput Across Different Storage Types Intel 16 CPU Harpertown 12GB memory 667 Memory speed Intel 82571EB Gigabit Ethernet HP IO Accelerator (Fusion-io) 32-byte messages Message Rate NIC 1 NIC Dura ble IO Fusio n Ca rd 1 NIC Dura ble Fibe r Disk 1 NIC Dura ble In te rn a l SCSI d rive HP IO Accelerator

20 Durable Messaging Latency Comparison Latencytest with Durable Store Different Storage Types Intel 16 CPU Hapertown 12GB memory 667 Memory speed Intel 82571EB Gigabit Ethernet HP IO Fusion 32-byte messages Average Latency (ms) NIC No Durable 1 NIC Iofusion Durable 1 NIC Fiber on durable 1 NIC Sata Durable HP IO Accelerator

21 MRG template for HP Matrix HP BladeSystem Matrix enables: Automated provisioning to speed deployment Capacity planning to optimize workloads dynamically Disaster recovery simplified Red Hat + HP developing MRG template for Matrix: to quickly stand up 'internal cloud' deployments with workflows, scripts, and best practice templates 21

22 Testing and developing solutions working together Delivered in reference papers & certifications Throughput Memory Usage Red Hat / HP White Paper: cache buff free GigE GigE IPoIB IB SDP IB RDMA

23 Additional Information

24 THANK YOU 24