CS 498 Hot Topics in High Performance Computing. Networks and Fault Tolerance. 10. Routing and Flow Control

Transcription

1 CS 498 Hot Topics in High Performance Computing Networks and Fault Tolerance 10. Routing and Flow Control

2 Intro What did we learn in the last lecture Some more topologies Routing (schemes and metrics) What will we learn today Routing practical examples Flow control Blue Waters topology and routing (if time) Torsten Hoefler: CS 498 Hot Topics in HPC 267

3 Deterministic Routing: InfiniBand Full bisection bandwidth fat-tree Communications 1 6 and 4 14 Torsten Hoefler: CS 498 Hot Topics in HPC 268

4 Deterministic Routing: InfiniBand Bisection (band)width is not a good metric! It is only an upper bound on worst-case performance! Routing can lower effective bandwidth Deterministic routing *WILL* (see Valiant s proof) Introduce Effective Bisection Bandwidth Expected bandwidth for a random permutation pattern (observable empirically) Optimal: full bandwidth, check impact by simulation Torsten Hoefler: CS 498 Hot Topics in HPC 269

5 Deterministic Routing: InfiniBand Sandia 4096 compute nodes dual Xeon EM64T 3.6 Ghz CPUs 6 GiB RAM ½ bisection bandwidth fat tree 4390 active LIDs while queried Torsten Hoefler: CS 498 Hot Topics in HPC 270

6 Deterministic Routing: InfiniBand LLNL 1152 compute nodes dual 4-core 2.4 GHz Opteron 16 GiB RAM full bisection bandwidth fat tree 1142 active LIDs while queried Torsten Hoefler: CS 498 Hot Topics in HPC 271

7 Deterministic Routing: InfiniBand TACC 3936 compute nodes quad 4-core 2.3 GHz Opteron 32 GiB RAM full bisection bandwidth fat tree 3908 active LIDs while queried Torsten Hoefler: CS 498 Hot Topics in HPC 272

8 Deterministic Routing: InfiniBand TUC 542 compute nodes dual 2-core 2.6 GHz Opteron 4 GiB RAM full bisection bandwidth fat tree 566 active LIDs while queried Torsten Hoefler: CS 498 Hot Topics in HPC 273

9 Deterministic Routing: InfiniBand Impact of backend congestion on bandwidth Provoked congestion 24-port switches Fat-tree Max. cong: 12! Torsten Hoefler: CS 498 Hot Topics in HPC 274

10 Deterministic Routing: InfiniBand Effective bisection bandwidth: Ranger: 57.6% Atlas: 55.6% Thunderbird: 40.6% (1/2 bisection bandwidth) Obvious questions: Does it make sense to build networks with full bisection bandwidth? What is the tradeoff between cost and bandwidth? Torsten Hoefler: CS 498 Hot Topics in HPC 275

11 Adaptive Routing: Myrinet Implemented and benchmarked optimal deterministic routing (will be defined later), random routing, and local and global adaptive routing in Firmware 512 Myrinet MX nodes connected as fullbisection bandwidth Clos (similar to fat-tree) Running effective bisection bandwidth benchmark, reporting min, avg, max bandwidth Torsten Hoefler: CS 498 Hot Topics in HPC 276

12 Deterministic Routing: Myrinet Torsten Hoefler: CS 498 Hot Topics in HPC 277

13 Random Routing: Myrinet Torsten Hoefler: CS 498 Hot Topics in HPC 278

14 Local Adaptive Routing: Myrinet Torsten Hoefler: CS 498 Hot Topics in HPC 279

15 Global (Probing) Adaptive Routing: Myrinet Torsten Hoefler: CS 498 Hot Topics in HPC 280

16 Overview of Techniques: Myrinet Torsten Hoefler: CS 498 Hot Topics in HPC 281

17 Routing Implementations Source routing All routing decisions are made entirely at the source node and encoded in packet Distributed (table) routing Routes are distributed across the switches (each switch only stores the parts it needs) Algorithmic routing Routes can be determined algorithmically (e.g., butterfly, DOR, ) Torsten Hoefler: CS 498 Hot Topics in HPC 282

18 Routing on Arbitrary Topologies General routing schemes needed for general topologies (e.g., InfiniBand, Ethernet) Standard Ethernet uses spanning tree (bad) What metric to optimize for? Application-optimized routing (possible if communication topology is known) Arbitrary permutations seem reasonable for general case! effective bisection bandwidth Torsten Hoefler: CS 498 Hot Topics in HPC 283

19 Routing Metrics Model network as G=(V P [V C, E) Path r(u,v) is a path between u,v 2 V P Routing R consists of P(P-1) paths Edge load l(e) = number of paths on e 2 E Edge forwarding index ¼(G,R)=max e2e l(e) ¼(G,R) is a upper bound to congestion of permutation routing! Goal is to find R that minimizes ¼(G,R) shown to be NP-hard in the general case Torsten Hoefler: CS 498 Hot Topics in HPC 284

20 A Greedy Heuristic P-SSSP routing starts a SSSP run at each node finds paths with minimal edge-load l(e) updates routing tables in reverse essentially SDSP updates l(e) between runs let s discuss an example Torsten Hoefler: CS 498 Hot Topics in HPC 285

21 P-SSSP Routing (1/3) Step 1: Source-node 0: 286

22 P-SSSP Routing (2/3) Step 2: Source-node 1: 287

23 P-SSSP Routing (3/3) Step 3: Source-node 2: ¼(G,R)=2 288

24 Routing Example: InfiniBand Example: InfiniBand InfiniBand uses deterministic distributed routing OpenSM configures the local network (subnet) implements different routing schemes Linear forwarding table (LFT) at each switch Lid mask control (LMC) enables multiple addresses per port Torsten Hoefler: CS 498 Hot Topics in HPC 289

25 Routing Example: InfiniBand Different routing algorithms: MINHOP (finds minimal paths, balances number of routes local at each switch) UPDN (uses Up*/Down* turn-control, very similar to MINHOP) FTREE (fat-tree optimized routing) DOR (dimension order routing for k-ary n-cubes) LASH (uses DOR and breaks credit-loops with virtual lanes) Torsten Hoefler: CS 498 Hot Topics in HPC 290

26 Benchmark Results Odin Simulation Benchmark (Netgauge Pattern ebb) Simulation predicts 5% improvement Benchmark shows 18% improvement! 291

27 Benchmark Results Deimos Simulation Benchmark (Netgauge Pattern ebb) Simulation predicts 23% improvement Benchmark shows 40% improvement! 292

28 Flow Control Determines how network resources are allocated Can either be viewed as resource allocation or contention resolution problem Mechanisms: bufferless: drop or misroute if channel is blocked circuit switching: only headers are buffered and reserves path, waits if path is not available buffered: decouples channel allocation in time Torsten Hoefler: CS 498 Hot Topics in HPC 293

29 Flits Packets are MTU-sized Typically several 1000 bytes Switch buffering and forwarding often works at smaller granularity (several bytes) Flit FLow Control UnIT Packets are divided into flits by the hardware Typically no extra headers Torsten Hoefler: CS 498 Hot Topics in HPC 294

30 Buffered Flow Control Store-and-forward Receives full packets into buffer and forwards them after they have been received High latency (each switch waits for full packet) Cut-through Forwards packet as soon as first (header) flit arrives and outgoing resources are available Low latency but blocks a channel for the duration of a whole packet transmission Torsten Hoefler: CS 498 Hot Topics in HPC 295

31 Buffered Flow Control Wormhole Like cut-through but channels and buffers are allocated on a per-flit basis Header flit allocates virtual channel, data and tail flits follow, tail flit frees channel VC might have pointers to a packet s flits Most flexible, high priority packets can intercept lower-priority packets in the middle Most efficient use of buffer space Also allows for buffers much smaller than MTU! Torsten Hoefler: CS 498 Hot Topics in HPC 296

32 Buffer Management and Backpressure Flow control methods need to communicate availability of buffers to downstream nodes Common types in use today: Credit-based Router keeps count of number of free flit buffers of peer routers, decrements when sending, stops when zero reached, peers send credits back On/Off Binary decision, neighbor routers send on or off flag to start or stop incoming flit streams Torsten Hoefler: CS 498 Hot Topics in HPC 297

33 Buffer Management and Backpressure Common types in use today (continued): Ack/Nack No state at routers, each flit will be ack d if it fits and nack d if not (resend) Drop No real flow control, packets are dropped if they do not fit and upper layer will re-send (common in Internet) Torsten Hoefler: CS 498 Hot Topics in HPC 298

34 Livelock If packets can take non-minimal routes, they could be routed in circles forever Good example: Internet Simple workaround: TTL Is not typical for HPC settings though Will not be further discussed here Torsten Hoefler: CS 498 Hot Topics in HPC 299

35 Deadlock A group of packets cannot progress because they wait for each other to free resources Cyclic dependency (deadlock criterion) Deadlock is catastrophic (networks clogs up) Deadlock avoidance E.g., loop-free routing Deadlock resolution E.g., packet timeouts Torsten Hoefler: CS 498 Hot Topics in HPC 300

36 Complex Deadlock Example Source Network and Routes Buffer Dependency Graph 301

37 Deadlock Avoidance Limit route selection function Up*/Down* routing Identify a tree (root) in the topology Allow only a single turn (relative to root) Lower bandwidth due to limited routes Virtual Channels VCs are independent, different block each other Can be used to create cycle-free layers Used in OpenSM s LASH algorithm Torsten Hoefler: CS 498 Hot Topics in HPC 302

38 Layered Routing Each VL requires physical resources Only 16 in InfiniBand (8 in practice) Layered routing needs to minimize needed VLs Minimization is NP-complete for arbitrary channel dependency graphs [see Domke, Hoefler, 11] Torsten Hoefler: CS 498 Hot Topics in HPC 303

39 Topology Example: Blue Waters P7-IH Quad Block Diagram 32w SMP w/ 2 Tier SMP Fabric, 4 chip Processor MCM + Hub SCM w/ On-Board Optics Four POWER 7 chips One IBM Hup Chip per node DIMM 2 DIMM 3 DIMM 11 DIMM 10 DIMM 6 DIMM 7 DIMM 0 DIMM 1 A Clk Grp B Clk Grp C Clk Grp D Clk Grp A Clk Grp B Clk Grp C Clk Grp D Clk Grp A Clk Grp B Clk Grp C Clk Grp D Clk Grp A Clk Grp B Clk Grp C Clk Grp D Clk Grp MC 0 MC11 W Z MC0 0 MC1 1 8c up 8c up P7-0 P7-1 B A Y X C Z C Y A B X W B A W X C Y C Z A B X Y P7-3 P7-2 8c up 8c up MC00 MC11 Z W MC0 0 MC1 1 A Clk Grp B Clk Grp C Clk Grp D Clk Grp A Clk Grp B Clk Grp C Clk Grp D Clk Grp A Clk Grp B Clk Grp C Clk Grp D Clk Grp A Clk Grp B Clk Grp C Clk Grp D Clk Grp DIMM 5 DIMM 4 DIMM 12 DIMM 13 DIMM 14 DIMM 15 DIMM 8 DIMM 9 28x XMIT/RCV 10 Gb/s 832 Z W 12x 12x 12x 12x 12x 12x 12x 12x 12x 12x 12x 12x 12x 12x 12x 12x D0-D x 12x 12x 12x 12x 12x 12x 12x 12x 12x 12x 12x Lr0-Lr GB/s 164 Ll GB/s 164 Ll1 Hub Chip Module 22+22GB/s 164 Ll GB/s 164 Ll GB/s 164 Ll GB/s 164 Ll GB/s 164 Ll6 7+7GB/s 7+7GB/s 7+7GB/s EG2 EG1 EG2 Y X 10+10GB/s (12x=10+2) 5+5GB/s (6x=5+1) 320 GB/s 240 GB/s 7 Inter-Hub Board Level L-Buses 8B+8B, 90% sus. peak PCIe 61x PCIe 16x PCIe 8x 304 Torsten Hoefler: CS 498 Hot Topics in HPC Source: IBM

40 1.1 TB/s HUB 192 GB/s Host Connection PX0 Bus Hot Plug Ctl PX1 Bus Hot Plug Ctl PX2 Bus Hot Plug Ctl FSP1-A FSP1-B TPMD-A, TMPD-B MDC-A MDC-B SEEPROM 1 SEEPROM 2 I2C I2C SVIC SVIC I2C FSI FSI 8x 8x 16x 16x 16x 16x 336 GB/s to 7 other local nodes 240 GB/s to local-remote nodes 320 GB/s to remote nodes 40 GB/s to general purpose I/O L local HUB To HUB Copper Board Wiring cf. The PERCS 10 LL0 Bus Copper LL1 Bus Copper LL2 Bus Copper LL3 Bus Copper LL4 Bus Copper LL5 Bus Copper LL6 Bus Copper 8B 8B 8B 8B 8B 8B 8B 8B 8B 8B 8B 8B 8B 8B EI-3 PHYs 6x PCI-E IO PHY 6x PCI-E IO PHY Diff PHYs PCI-E IO PHY Hub Chip Torrent 6x 6x 8B W-Bus 8B W-Bus TOD Sync 8B X-Bus 8B X-Bus TOD Sync 8B Y-Bus 8B Y-Bus TOD Sync Diff PHYs 8B Z-Bus 8B Z-Bus TOD Sync 28 I2C 12x 12x 12x 12x 16 D Buses I2C_0 + Int I2C_27 + Int D0 Bus Optical D15 Bus Optical I2C To Optical Modules D Bus Interconnect of Supernodes Optical LR0 Bus Optical 24 L remote Buses LR23 Bus Optical HUB to QCM Connections Address/Data L remote 4 Drawer Interconnect to Create a Supernode Optical 305 Torsten Hoefler: CS 498 Hot Topics in HPC Source: IBM

41 Drawer 8 nodes 64/40 Optical 'D-Link' P C I e 17 P1-C17-C1 P C I e 16 P1-C16-C1 P C I e 15 P1-C15-C1 P C I e 14 P1-C14-C1 P C I e 13 P1-C13-C1 P C I e 12 P1-C12-C1 P C I e 11 P1-C11-C1 P C I e 10 P1-C10-C1 P C I e 9 P1-C9-C1 Optical Fan-out from HUB Modules 2,304 Fiber 'L-Link' P C I e 8 P1-C8-C1 P C I e 7 P1-C7-C1 P C I e 6 P1-C6-C1 P C I e 5 P1-C5-C1 P C I e 4 P1-C4-C1 P C I e 3 P1-C3-C1 P C I e 2 P1-C2-C1 P C I e 1 P1-C1-C1 64/40 Optical 'D-Link' 32 chips 256 cores HUB 0 HUB 1 HUB 2 HUB 3 HUB 4 HUB 5 HUB 6 HUB 7 First Level Interconnect L-Local HUB to HUB Copper Wiring 256 Cores N0-DIMM15 N0-DIMM14 N0-DIMM13 N0-DIMM12 N0-DIMM11 N0-DIMM10 N0-DIMM09 N0-DIMM08 P7-1 QCM 0 P7-2 P7-0 P7-3 U-P1-M1 N0-DIMM07 N0-DIMM06 N0-DIMM05 N0-DIMM04 N0-DIMM03 N0-DIMM02 N0-DIMM01 N0-DIMM00 N1-DIMM15 N1-DIMM14 N1-DIMM13 N1-DIMM12 N1-DIMM11 N1-DIMM10 N1-DIMM09 N1-DIMM08 P7-1 QCM 1 P7-2 P7-0 P7-3 U-P1-M2 N1-DIMM07 N1-DIMM06 N1-DIMM05 N1-DIMM04 N1-DIMM03 N1-DIMM02 N1-DIMM01 N1-DIMM00 N2-DIMM15 N2-DIMM14 N2-DIMM13 N2-DIMM12 N2-DIMM11 N2-DIMM10 N2-DIMM09 N2-DIMM08 P7-1 QCM 2 P7-2 P7-0 P7-3 U-P1-M3 N2-DIMM07 N2-DIMM06 N2-DIMM05 N2-DIMM04 N2-DIMM03 N2-DIMM02 N2-DIMM01 N2-DIMM00 N3-DIMM15 N3-DIMM14 N3-DIMM13 N3-DIMM12 N3-DIMM11 N3-DIMM10 N3-DIMM09 P7-1 QCM 3 P7-2 P7-0 P7-3 U-P1-M4 N3-DIMM07 N3-DIMM06 N3-DIMM05 N3-DIMM04 N3-DIMM03 N3-DIMM02 N3-DIMM01 N3-DIMM08 N4-DIMM15 N4-DIMM14 N4-DIMM13 N4-DIMM12 N4-DIMM11 N4-DIMM10 N4-DIMM09 N4-DIMM08 P7-1 QCM 4 P7-2 P7-0 P7-3 U-P1-M5 N3-DIMM00 N4-DIMM07 N4-DIMM06 N4-DIMM05 N4-DIMM04 N4-DIMM03 N4-DIMM02 N4-DIMM01 N4-DIMM00 N5-DIMM15 N5-DIMM14 N5-DIMM13 N5-DIMM12 N5-DIMM11 N5-DIMM10 N5-DIMM09 N5-DIMM08 P7-1 QCM 5 P7-2 P7-0 P7-3 U-P1-M6 N5-DIMM07 N5-DIMM06 N5-DIMM05 N5-DIMM04 N5-DIMM03 N5-DIMM02 N5-DIMM01 N5-DIMM00 N6-DIMM15 N6-DIMM14 N6-DIMM13 N6-DIMM12 N6-DIMM11 N6-DIMM10 N6-DIMM09 N6-DIMM08 P7-1 QCM 6 P7-2 P7-0 P7-3 U-P1-M7 N6-DIMM07 N6-DIMM06 N6-DIMM05 N6-DIMM04 N6-DIMM03 N6-DIMM02 N6-DIMM01 N6-DIMM00 N7-DIMM15 N7-DIMM14 N7-DIMM13 N7-DIMM12 N7-DIMM11 N7-DIMM10 N7-DIMM09 N7-DIMM08 P7-1 QCM 7 P7-2 P7-0 P7-3 U-P1-M8 N7-DIMM07 N7-DIMM06 N7-DIMM05 N7-DIMM04 N7-DIMM03 N7-DIMM02 N7-DIMM01 N7-DIMM00 DCA-0 Connector (Top DCA) DCA-1 Connector (Bottom DCA) Torsten Hoefler: CS 498 Hot Topics in HPC Source: IBM 1st Level Local Interconnect (256 cores) 306

42 Topology Example: Blue Waters Torsten Hoefler: CS 498 Hot Topics in HPC 307

43 LL Topology 24 GB/s 7 links/hub Fully connected 8 Hubs Torsten Hoefler: CS 498 Hot Topics in HPC Source: B. Arimilli et al. The PERCS High- Performance Interconnect 308

44 LR Topology 5 GB/s 24 links/hub Fully connected 4 Drawers 32 Hubs Torsten Hoefler: CS 498 Hot Topics in HPC Source: B. Arimilli et al. The PERCS High- Performance Interconnect 309

45 D Topology 10 GB/s 16 links/hub Fully connected 512 SNs 2048 Drawers Hubs Torsten Hoefler: CS 498 Hot Topics in HPC Source: B. Arimilli et al. The PERCS High- Performance Interconnect 310

46 Routing Deterministic routing Hits Borodin-Hopcroft bound ( ) Bad worst-case performance Indirect (random) routing Increases latency Effectively halves (global) bandwidth Worst-case becomes very unlikely 311 Torsten Hoefler: CS 498 Hot Topics in HPC

47 Direct Routing SuperNode A LR12 SN-SN Direct Routing L-Hops: 2 D-Hops: 1 Total: 3 hops SuperNode B D3 LR21 Torsten Hoefler: CS 498 Hot Source: Topics B. in Arimilli HPC et al. The PERCS High- Performance Interconnect 312

48 SuperNode A SuperNode B SN-SN Indirect Routing LL7 L-Hops: 3 D-Hops: 2 Total: 5 hops Total paths = # SNs 2 LR21 D5 SuperNode x D12 LR30 Torsten Hoefler: CS 498 Hot Topics in HPC 313 Source: B. Arimilli et al. The PERCS High- Performance Interconnect

49 Example Model: BW Global Alltoall Bandwidth Assume all communications happen simultaneously Derive upper (expected) bound for direct routing Simple counting argument Each CN can be reached through series of LL, LR, D Not more than one D link or LL-LR, LR-LL, LL-LL, LR-LR Denote e(p) as number of nodes reachable through path P e(ll) =7, e(lr)=24, e(d)=d (variable number of D-links) # nodes reachable in two hops: e(ll-d)=e(d-ll)=7d e(lr-d)=e(d-lr) = 24d 314 Torsten Hoefler: CS 498 Hot Topics in HPC Source: B. Arimilli et al. The PERCS High- Performance Interconnect

50 Example: Alltoall Bandwidth Continued # nodes reachable in three hops: e(ll-d-ll) = 49d e(ll-d-lr)=e(lr-d-ll) = 168d e(lr-d-lr)=576d Number of paths from each source: d Now count the number of paths (i.e., congestion) through each LL, LR, D link c(l): (omitted uninteresting parts of summation) c(ll) = 1+64d c(lr) = 1+64d c(d) = Torsten Hoefler: CS 498 Hot Topics in HPC Source: B. Arimilli et al. The PERCS High- Performance Interconnect

51 Example: Alltoall Bandwidth Continued Effective (expected) bandwidths for each link-type: bandwidth = link bandwidth / congestion b(ll) = 24 GB/s / (1+64d) b(lr) = 5 GB/s / (1+64d) b(d) = 10 GB/s / 1024 If d<8, D is the bottleneck, otherwise LR Bandwidth per PE: 5 GB/s / 1+64d * total # paths d=9 ( cores): GB/s d=10 ( cores): GB/s total ~ 0.8 PB/s Connection to P7 chips: 4*24 GB/s = 96 GB/s ~ 83% 316 Torsten Hoefler: CS 498 Hot Topics in HPC Source: B. Arimilli et al. The PERCS High- Performance Interconnect