Quadrics QsNet II : A network for Supercomputing Applications

Size: px

Start display at page:

Download "Quadrics QsNet II : A network for Supercomputing Applications"

Brent Hall
5 years ago
Views:

1 Quadrics QsNet II : A network for Supercomputing Applications David Addison, Jon Beecroft, David Hewson, Moray McLaren (Quadrics Ltd.), Fabrizio Petrini (LANL)

2 You ve bought your super computer You ve constructed your building for it! So w hat do you w ant from the netw ork? Any process to any other process communication in zero time. That s it! Sim ple Can t have that so you need the next best thing Image Courtesy of LANL

3 A Quadrics Supercomputer Network Ultra low user process to user process latency Highest possible (affordable?) compute communications ratio Seamless scaling to many 1000s of nodes High availability Reliable dat a t ransfer Mixed system and m ultiple user traffic on one network

4 Supercomputers Los Alam os National Laboratory ASCI Q Gflops Law rence Liverm ore National Laboratory - MCR Gflops Law rence Liverm ore National Laboratory - ALC Gflops Pacific Northw est National Laboratory Gflops Pittsburgh Supercomputer Centre -LeMieux4463Gflops CEA - Tera Gflops Images Courtesy of LANL, LLNL,PNNL, PSC, CEA

5 QsNet II Com ponent s Elan 4 net work interface card Elit e 4 swit ch component QsNet I I Switch

6 Topology Fat trees give great performance Scales to the number of nodes Fault tolerance st ruct ure Uniform connectivity Supports global operations

7 A Process Communication Cable or fibre up t o 150m Cable up t o 13m Elit e4 Swit ch Chip

8 8 byte write latency on a 4000 node machine with 50m of cable Latency ( ns) Elit e Swit ch Cable Elan CPU ChipSet Elan3 ( QsNet ) Elan4 ( QsNet II )

Elan 4 funct ional unit s 64-bit virtual addressing Short Transaction ENgine Pipelined ( R) DMA engine 64-bit RISC processor 16Kbyte On-chip I-cache Mem ory System 32Kbyte On-chip

9 Elan 4 funct ional unit s 64-bit virtual addressing Short Transaction ENgine Pipelined ( R) DMA engine 64-bit RISC processor 16Kbyte On-chip I-cache Mem ory System 32Kbyte On-chip D-cache, pipelined fills, multi-port. 64-bit MMU 128-TLB entries, hash walk engine, mixed page sizes, 16 bit context. 64-bit/133MHz PCI-X 64Mbytes ECC DDR RAM Link. 2.6 Gbytes/ sec total

10 Elan4 Com m and Queues Enables a user process to send packet s into the network with very low latency Used to start all operations. (RDMA, STEN, RI SC t hreads et c) Up to 8K com m and queues can be allocated Com m and queues can by used by m any processes simultaneously from multiple CPUs

11 Com m and Program m ing Model

12 Elan4 Com m and Queues The Com m and Processor can execute com m ands directly as they are written from the PCI-X bus 80ns from PCI-X bus to network link Pr ovides aut o r et r y of STEN packet s DDR-SDRAM used as backing store for Queues Copes wit h a m ain CPU process t im eslice in a command stream and concurrent access by multiple CPUs Copes wit h occasional out of order PIO writes

13 Com m and Proc I m plem ent at ion

14 Elan4 RDMA processor Addresses are 64 bit Pipelined operation t o hide lar ge PCI - X read latency Pr ocesses t w o RDMA descr ipt or s concur r ent ly to achieve peak bandwidth with multiple small RDMAs Two run queues for high and low priority scheduling Tim eslices between m ultiple RDMAs

15 RI SC processor 64 bit word Instruction set optimised for low latency scheduling 16 Kbyte I-Cache Regist ers loaded direct ly from the network Block load/ store instructions for copy bandwidth

16 Elan4 im plem entat ion LSI G12 process 0.18um 4 + R layer metal process 7.5 mm x 7.5 mm ~800,000 gates ~283 signal pins 512 ball BGA 3 watts

17 QsNet II Physical Link 1.333Ghz design speed 4b5b coding for DC balance on cables and fiber ~920 Mbytes/s after protocol Internal switch links deliver 1.18 Gbytes/ s after protocol Copper Optics 2 virtual channels 10 bit LVDS total 40 wires 12m range 12 bit parallel optical fiber 150m range

18 QsNet II Elit e4 Swit ch Com ponent 8 QsNet II links 2 virtual channels Broadcast to range of outputs Full autom at ic error det ect ion / recovery Arbitration based on age of packet Two levels of priority Adaptive routing support Unblocked latency of ~ 20ns Traceroute transaction for interrogating the network

19 Elit e 4 im plem entat ion LSI G12 process 0.18um layer metal process. 8.67mm x 8.67mm ~ 1 million gates ~348 signal pins 608 ball BGA 6.5 watts

20 Perform ance Short message latency RDMA bandwidt h Perfor m ance in applications MPI m essage passing Global operations

21 Short message latency on a 4000 node machine with 50m cable Latency ( us) Number of bytes in message

22 RDMA bandwidt h Batched RDMAs Bandw idth ( MB/ s) Size in bytes

23 MPI short m essage latency useconds Infiniband QsNet I I (elan4) Bytes

24 MPI Bandwidth - Elan MBytes/ sec MPI bandwidth MPI ping Size in bytes

25 Global operat ions - Hardware barrier scaling lateccy ( us) Elan3 Elan Nodes

26 Acknowledgements

1/5/2012. Overview of Interconnects. Presentation Outline. Myrinet and Quadrics. Interconnects. Switch-Based Interconnects

1/5/2012. Overview of Interconnects. Presentation Outline. Myrinet and Quadrics. Interconnects. Switch-Based Interconnects Overview of Interconnects Myrinet and Quadrics Leading Modern Interconnects Presentation Outline General Concepts of Interconnects Myrinet Latest Products Quadrics Latest Release Our Research Interconnects