InfiniBand-based HPC Clusters

Transcription

1 Boosting Scalability of InfiniBand-based HPC Clusters Asaf Wachtel, Senior Product Manager 2010 Voltaire Inc.

2 InfiniBand-based HPC Clusters Scalability Challenges Cluster TCO Scalability Hardware costs Software License costs Space, Power & Cooling Communication Scalability Handle increasing compute power Multi-core core, GPUs Utilization Scalability Many jobs & users Varying sizes, traffic patterns & QoS Application Scalability Home-grown or ISVs MPI Collectives 2

3 Voltaire 40Gb/s InfiniBand Portfolio Fabric provisioning and performance monitoring Application Acceleration 40Gb/s InfiniBand Switching Platforms HSSM SSI Blade Switch x IB ports 4036E 34 x IB ports + 2 x 1/10GbE x IB ports /648 x IB ports 3

4 Scalable Architectures Fat Tree Full bi-sectional bandwidth at any node count Uniform oversubscription options HyperScale Scale to thousands of nodes with linear performance Large non-blocking islands (more than 2,000 cores) 4-hops maximum latency to any port Lowest number of switches and cables Torus Lowest cost solution Built entirely with edge switches and copper cables Optimized support by Voltaire software, including Torus2QoS routing 4

5 HyperScale in the Top500 Large, low-latency, non-blocking Islands Lowest number of switches & cables Scales to thousands of nodes with linear performance 8:1 Oversubscribed Core 1,200-node Interconnect in only 2 Racks 13 x non-blocking HyperScale Islands 1.05PFLOPs 83.7% Efficiency 5

6 The Challenge: Static Routing Inefficiency The Challenge: One Size Routing does not Fit All Static routing assumes uniform traffic across entire fabric Real life is different Most jobs use small portion of the clusters Different traffic patterns for different jobs Different requirements for different traffic types (e.g. storage) The Solution: Voltaire TARA (Traffic Aware Routing Algorithm) A new routing algorithm on top of OpenSM Dynamically optimizes routing according to defined traffic patterns: Fabric topology Job-specific communication patterns Symmetric/Asymmetric communication Traffic load/qos Fully integrated t with leading job schedulers 6

7 TARA Traffic Aware Routing Algorithm Maximizing Cluster Utilization OpenSM without UFM TARA UFM TARA is ON port weight port weight switch.port Internal ports on the line cards switch.port

8 The Challenge: Collective Operations Scalability Grouping algorithms are unaware of the topology and inefficient Network congestion due to all-to-all communication Slow nodes & OS involvement impair scalability and predictability The more powerful servers get (GPUs, more cores), the more poorly collectives scale in the fabric % collectives out of total run time Total run time Run time variance # Ranks # Ranks # Ranks Significant Inhibitor to MPI Application Scalability 8

9 Introducing: Voltaire Fabric Collective Accelerator Grid Director Switches: Switches: Fabric Collective Processing operations Power offloaded to switch CPUs Grid Director FCA Manager: Unified Fabric Topology-based p collective Manager tree (UFM): + Separate Virtual + network Topology Aware IB multicast for result Orchestrator distribution. + FCA Agent: Inter-core processing localized & optimized. + Breakthrough performance with no additional hardware 9

10 FCA Fabric Collective Accelerator Unmatched Application Scalability First and only system-wide solution for offloading MPI collectives Accelerates MPI collective computation by as much as 100X 10-40% improvement in application runtime Integrated with leading MPI implementations Fluent truck_111m 192 cores PMPI PMPI + FCA PMPI PMPI + FCA 10

11 Summary Reduced total cost of ownership via scalable topologies (HyperScale) Increase cluster utilization via Traffic Aware Routing g( (TARA) Boost application scalability using Fabric Collective Acceleration (FCA) More Performance for each $ Spent 11

12 Thank You Asaf Wachtel Senior Product Manager, InfiniBand Solutions 2010 Voltaire Inc.