The Cray XD1. Technical Overview. Amar Shan, Senior Product Marketing Manager. Cray XD1. Cray Proprietary

Transcription

1 The Cray XD1 Cray XD1 Technical Overview Amar Shan, Senior Product Marketing Manager Cray Proprietary

2 The Cray XD1 Cray XD1 Built for price performance 30 times interconnect performance 2 times the density High availability Single system command & control Purpose-built and optimized for high performance workloads May 04 Cray Proprietary 2

3 Cray XD1 System Architecture Compute 12 AMD Opteron 32/64 bit, x86 processors High Performance Linux RapidArray Interconnect 12 communications processors 1 Tb/s switch fabric Active Management Dedicated processor Application Acceleration 6 co-processors Processors directly connected via integrated switch fabric May 04 Cray Proprietary 3

4 Under the Covers Six SATA Hard Drives 4 Fans Chassis Front Six 2-way SMP Blades 500 Gb/s crossbar switch Four independent PCI-X Slots 12-port Interchassis connector Connector to 2 nd 500 Gb/s crossbar switch and 12-port inter-chassis connector Chassis Rear May 04 Cray Proprietary 4

5 Compute Blade 4 DIMM Sockets for DDR 400 Registered ECC Memory RapidArray Communications Processor AMD Opteron 2XX Processor AMD Opteron 2XX Processor Connector to Main Board 4 DIMM Sockets for DDR 400 Registered ECC Memory May 04 Cray Proprietary 5

6 The AMD Opteron Processor Dedicated Memory Bus Native 32 & 64 bit x86 compatibility 64KB 64KB 1 MB Up to 19.2 GB/s I/O May 04 Cray Proprietary 6

7 Cray Innovations Balanced Interconnect Active Management Cray XD1 Application Acceleration Performance and Usability May 04 Cray Proprietary 7

8 Interconnect Cray Proprietary

9 Cray XD1 Interconnect System RapidArray Interconnect processors Switch fabric Communication s software May 04 Cray Proprietary 9

10 Typical HPC Application Compute Communicate Compute Communicate Compute. HPC applications exhibit intense compute/ communicate cycles 20% - 60% of time, CPUs sit idle, stalled by communications Application performance is very sensitive to latency and bandwidth Interconnect Drives System Performance May 04 Cray Proprietary 10

11 Removing the Communications Bottleneck GigaBytes Memory GFLOPS GigaBytes per Second Processor I/O Interconnect Xeon Server 5.3 GB/s DDR GB/s PCI-X 0.25 GB/s GigE Cray XD1 6.4GB/s DDR 400 RA 8 GB/s Cray RS/Strider 6.4GB/s DDR GB/s DDR GB/s SS 31 GB/s Cray X1 31 GB/s 34.1 GB/s 102 GB/s May 04 Cray Proprietary 11

12 HPC Communications Optimizations Cray Communications Libraries MPI 1.2 library TCP/IP PVM Shmem Global Arrays System-wide process & time synchronization RapidArray Communications Processor HT/RA tunnelling with bonding Routing with route redundancy Reliable transport Short message latency optimization DMA operations System-wide clock synchronization AMD Opteron 2XX Processor RapidArray Communications Processor 2 GB/s 3.2 GB/s RA 2 GB/s Direct Connected Processor Architecture May 04 Cray Proprietary 12

13 Interconnect Benchmarks (MPI Latency) MPI Latency versus Message Size Latency (M icrosec.) Message Length (Bytes) Cray XD1 RapidArray MVAPICH (MPI/VAPI/IB) MPICH-GM (Myrinet D card) MPICH-QsNet (Quadrics Elan3) 4 times lower latency than Myrinet (small message). The The Cray XD1 XD1 has has sent sent 1 KB KB before others have sent sent a single byte. May 04 Cray Proprietary 13

14 Interconnect Benchmarks (MPI Throughput) Bandwidth versus Message Size Bandwidth (M B/s) Cray XD1 RapidArray MVAPICH (IB) MPICH-GM (Myrinet) MPICH-QsNet (Quadrics) Data length (Bytes) The The Cray XD1 XD1 is is 5X 5X throughput of of Quadrics at at 1KB 1KB messages May 04 Cray Proprietary 14

15 Processor Interaction How to Double Application Performance May 04 Cray Proprietary 15

16 Synchronized Linux Scheduler May 04 Cray Proprietary 16

17 Direct Connect Topology 1 Cray XD1 Chassis 12 AMD Opteron Processors 53 GFLOPS 8 GB/s between SMPs 1.6 µsec interconnect Integrated switching 3 Cray XD1 Chassis 36 AMD Opteron Processors 158 GFLOPS 8 GB/s between SMPs 1.8 µsec interconnect Integrated switching 25 Cray XD1 Chassis, two racks 300 AMD Opteron Processors 1.3 TFLOPS 2-8 GB/s between SMPs 1.8 µsec interconnect Integrated switching May 04 Cray Proprietary 17

18 Fat Tree Topology Spine switch Spine switch Spine switch 12 Cray XD1 chassis 144 AMD Opteron Processors 633 GFLOPS 4/8 GB/s between SMPs 1.9 µsec interconnect Fat tree switching, integrated first & third order 6/12 RapidArray spine switches (24-ports) May 04 Cray Proprietary 18

19 Management Cray Proprietary

20 Causes of System Outages Most problems occur during: Upgrades, Problem diagnosis, Configuration Power Supply 2% Other 1% Disks 7% Complex systems Patchwork software Inadequate tools Fans 5% Hardware 15% Multivendor h/w, s/w Driver & software compatibility Corrupt, stale software & configs Other (Software & Operations) 85% Source: UC Berkeley May 04 Cray Proprietary 20

21 Active Manager System CLI and Web Access Active Management Software Usability Single System Command and Control Resiliency Dedicated management processors, real-time OS and communications fabric. Proactive background diagnostics with selfhealing. Automated management for exceptional reliability, availability, serviceability May 04 Cray Proprietary 21

22 FCAPS Fault Configuration Accounting Performance Security FUNCTION Maintain health of the system Monitor, report on and automatically heal or Allow operator intervention to resolve problems Define, monitor and change operational parameters of the system Provide administrators with a real-time view of total system configuration and Automate configuration tasks to minimize effort and inconsistencies Monitor system resources and usage Support cost accounting or bill back Allow administrators to fine tune system operation to improve job and system performance Control access to the system system resources and to specific functions within the system Active Manager Features System monitoring Hardware management Alarm management Commission, upgrade, and expand system Software management Network configuration management Manage users Partition Management Quotas Usage tracking per job, user, department Reporting Resource and Queue Management Job management File system management System performance analysis User privilege management Data backups May 04 Cray Proprietary 22

23 Active Manager GUI: SysAdmin Portal GUI GUI provides quick quick access to to status status info info and and system functions May 04 Cray Proprietary 23

24 Active Manager Command Line Interface Administrators can can access Active Active Manager software and and Linux Linux through the the CLI CLI May 04 Cray Proprietary 24

25 XML Integration Point Simplifies data transfer May 04 Cray Proprietary 25

26 System Partitions Users & Administrators Compute Partition 1 File Services Partition Front End Partition Compute Partition Service Partition File Services Database DNS Front End Partition Compute Partition 2 Compute Partition 1 Manage multiple processors and copies of of Linux as as single, unified system May 04 Cray Proprietary 26

27 Single System Command and Control Users & Administrators Compute Partition 1 File Services Partition Partition management Linux configuration Hardware monitoring Software upgrades File system management Data backups Network configuration Accounting & user management Security Performance analysis Resource & queue management Front End Partition Compute Partition 2 Compute Partition 1 May 04 Cray Proprietary 27

28 Active Manager System/Partition View Managing virtual computers instead of of individual processors May 04 Cray Proprietary 28

29 Active Manager Task Wizard Simplifies complex tasks to to increase efficiency and and reduce downtime May 04 Cray Proprietary 29

30 Active Manager Job Scheduler Job Job management is is integrated with with self-healing features to to increase job job completion rates May 04 Cray Proprietary 30

31 Active Manager User Portal Intuitive user access for for job job submission and and status lets lets users focus on on applications, not not computing May 04 Cray Proprietary 31

32 Self-Monitoring Parity Heartbeat Temperature Fan speed Diagnostics Air Velocity Voltage Current Hard Drive Processors Memory Thermals Power supply Fans Active Manager Interconnect PCI-X Power Supply Dedicated Management Processor, OS, Fabric May 04 Cray Proprietary 32

33 Self-Healing Users & Administrators Compute Partition 1 File Services Partition Front End Partition Continuous Monitoring Detect (Future) Failure Attempt reset Isolate failed component Re-allocate resources (N+1 sparing or policy-based) Compute Partition 2 Compute Partition 1 Automated Recovery Reduces MTTR from hours to to minutes May 04 Cray Proprietary 33

34 Active Manager Thermal Management May 04 Cray Proprietary 34

35 Active Manager Alarm Management May 04 Cray Proprietary 35

36 Roll back Undo command: Journaling rlsemaster apprlse otherappmaster v1.3 v1.4 v2.0 gamess nwchem RPM Files RPM Files RPM Files v4.6 r2.3a r2.4 Install Files Install Files Install Files Quick Rollback Reduces MTTR from hours to to minutes May 04 Cray Proprietary 36

37 Application Acceleration FPGA Cray Proprietary

38 Application Acceleration P Application Accelerator RAP Application Acceleration Reconfigurable Computing Tightly coupled to Opteron FPGA acts like a programmable coprocessor Well-suited for: Searching, sorting, signal processing, audio/video/image manipulation, error correction, coding/decoding, packet processing, random number generation. P RAP SuperLinear speedup for key algorithms May 04 Cray Proprietary 38

39 Application Acceleration FPGA... do for each array element... end DataSet Compute Processor Application Acceleration FPGA Fine-grained parallelism applied for 100x potential speedup May 04 Cray Proprietary 39

40 FPGA and Vector Processors Vector Processors SIMD instruction level parallelism Mature Development Environment (C, Fortran) Integer or Floating Point FPGAs MIMD code fragments are replicated Emerging Development Tools (mostly HW tools VHDL, Verilog) Integer or Fixed Point Suited to matrix operations: BLAS, LAPACK, Suited to pre-processing (signal processing, sorting/searching, error correction, coding/decoding, packet processing) May 04 Cray Proprietary 40

41 Cray XD1: Built for Performance Faster Faster interconnect throughput Lower Lower interconnect latency System-wide process synchronization Ideal Speedup Myrinet Gigabit Ethernet Fast Ethernet Cray Greater Efficiency Greater Scalability Faster Performance Processors Increasing application efficiency and scalability for breakthrough performance gains May 04 Cray Proprietary 41

42 The Cray XD1 Cray XD1 Built for price/performance Interconnect bandwidth/latency System-wide process synchronization Application Acceleration FPGAs Standards-based 32/64-bit X86, Linux, MPI High resiliency Self-configuring, self-monitoring, selfhealing Single system command & control Intuitive, tightly integrated management software Purpose-built and and optimized for for high performance workloads May 04 Cray Proprietary 42

43 Questions? Amar Shan, Senior Product Marketing Manager Cray Proprietary

44 Additional Slides Cray Proprietary

45 Cray System Portfolio Cray X1 1 to 50+ TFLOPS $3 M - $10 M+ Vectorized apps Cray MSP/UNICOS/mp RS/Strider 1 to 50+ TFLOPS ,000+ processors $1 M - $5 M+ Opteron/Linux/Catamount Cray XD1 48 GFLOPS TFLOPS processors $50 K - $2 M+ AMD Opteron/Linux Purpose-Built High Performance Computers May 04 Cray Proprietary 45

46 Communications Protocols May 04 Cray Proprietary 46

47 Switchless Toroid Topology 27 chassis, 324 Processors 1.4 TFLOP 3x3x3 Toroid Nearest Neighbor 4-8 GB/s bandwidth per SMP 1.8 µsec latency 3D torus Worst case: 6 Hops, 2.8 µsec Well-Suited for Nearest Neighbor Problems May 04 Cray Proprietary 47

48 Active Manager Self Healing Policies May 04 Cray Proprietary 48

49 Application Acceleration Co-Processor AMD Opteron HyperTransport 3.2 GB/s 3.2 GB/s 3.2 GB/s 3.2 GB/s RAP RapidArray 3.2 GB/s 3.2 GB/s Application Acceleration FPGA Xilinx Virtex II Pro QDR SRAM 2 GB/s 2 GB/s Cray RapidArray Interconnect May 04 Cray Proprietary 49

50 Application Acceleration Interface RapidArray Transport Core User Logic QDR RAM Interface Core ADDR(20:0) D(35:0) Q(35:0) RAP TX RX ADDR(20:0) D(35:0) Q(35:0) ADDR(20:0) D(35:0) Q(35:0) QDR SRAM RapidArray ADDR(20:0) D(35:0) Q(35:0) XC2VP30 running at 200 MHz. 4 QDR II RAM with over 400 HSTL-I I/O at 200 MHz DDR (400 MTransfers/s). 16 bit simplified HyperTransport I/F at 400 MHz DDR (800 MTransfers/s.) QDR and HT I/F take up <20 % of XC2VP30. The rest is available for user applications. May 04 Cray Proprietary 50

51 Application Acceleration Variants A variety of Application Acceleration variants can be manufactured by populating different pin compatible FPGAs and QDR II RAMs. FPGA Speed Logic Elements PowerPC 18x18 Multipliers XC2VP , XC2VP , RAMs Speed Dimensions Quantity Total Size K7R MHz 1M x MByte May 04 Cray Proprietary 51

52 FPGA Development Flow Cores HDL RAP I/F, Metadata QDR RAM I/F Synthesize Implement Download Binary File VHDL, Verilog, C Synplicity, Leonardo, Precision, Xilinx ISE Simulate Xilinx ISE From Command line or Application Verify Modelsim Xilinx ChipScope May 04 Cray Proprietary 52

53 FPGA Linux API Admininstration Commands fpga_open allocate and open fpga fpga_close close allocated fpga fpga_load load binary into fpga Control Commands fpga_start start fpga (release from reset) fpga_stop stop fpga Status Commands fpga_status get status of fpga Data Programmer Commands sees get/put and message fpga_put passing programming put model data to fpga ram May 04 Cray Proprietary 53

54 Storage Strategy Cray Proprietary

55 File Systems: Local Disks One S-ATA HD per SMP; Local Linux directory per HD EXT2/3 EXT2/3 EXT2/3 RapidArray EXT2/3 EXT2/3 EXT2/3 Cray XD1 May 04 Cray Proprietary 55

56 File Systems: Diskless compute nodes Single Boot Disk for System, External NAS for User Data Front End Partition EXT2/3 system NFS NFS NAS Compute Partition Cray XD1 May 04 Cray Proprietary 56

57 File Systems: SAN SMP acting as a File Server for the SAN File Server EXT2/3 FC HBA FC SAN NFS Compute Cray XD1 May 04 Cray Proprietary 57

58 File Systems: Local Disk per Compute Node One disk per SMP; NFS Cross-mounting; External NAS Optional EXT2/3 EXT2/3 EXT2/3 NFS EXT2/3 EXT2/3 EXT2/3 Cray XD1 May 04 Cray Proprietary 58

59 Designed for High Availability Fewer Failures Stateless Hardware Reduced Variability Self-Configuring Today 99% Availability 100 minutes downtime / week 1 failure / week Faster Recovery Self-Monitoring Self-Healing Software Rollbacks Target 99.99% Availability 53 minutes downtime / year 1 failure of 5 minutes/month No incremental cost One vendor One phone call Fully integrated and tested May 04 Cray Proprietary 59