EE 382C Final Project Presentation. Ted Jiang Curt Harting 5/24/11

Transcription

1 EE 382C Final Project Presentation Ted Jiang Curt Harting 5/24/11

2 Overview Dragonfly Network 15 End Points

3 Overview 22 s per Group 330 Endpoints per Group

4 15 End Points Overview 331 Groups Endpoints Total

5 Board & Rack Design 15 nodes RACK Each 32U Rack has 30 endpoints and 2 router nodes (15 nodes per router) Each router has 51 QSFP connectors 15 on front for endpoint to router 36 on the back for intra/inter group

6 Group Layout 11 Racks per group in 2 rows 3.0m 2.73m 1m Hot Aisle 2.5m Cold Aisle Hot Aisle Cold Aisle Data Center will be 18 groups (49.5m) by 19 groups (57m)

7 Cost Estimation - Optical Item Amount Cost For 1 Total Cost Optical Cables Electrical Cables to end point Electrical Cables Intra-Group PCB Boards 1U 330*330/2 = $100 $5,445, ,000 $0.10 $10,000 21*22*331/ 2=76,461 22*331 = 7282 Connectors 51*22* ,000 = 471,382 $0.10 $7,461 $250 $1,820,500 $0.10 $47,138 E Cables - Intragroup 0% E Cables - Endpoint 0% Connectors 1% PCB Boards 1U 25% Chips 1% Optical Cables 73% Chips 22*331 = $12.25 $89, TOTAL $7,419,304

8 Cost Estimation - Electrical Item Amount Cost For 1 Total Cost Electrical Inter Electrical Endpoint 330*330 * 5/2 = 272,250 $0.10 $27, ,000 $0.10 $10,000 E Cables - Intergroup 0% E Cables - Endpoint 0% E Cables Connectors - Intragroup 1% 0% Electrical Intra 21*22*331/ 2=76,461 $0.10 $7,461 PCB Boards 1U 25% PCB 22*331 = 7282 Connectors 51*22* ,000 = 471,382 Chips 22*331 = 7282 Repeaters 330*330*5 /2 = 272,250 $250 $1,820,500 $0.10 $47,138 $12.25 $89,205 $0.5*1mm 2 + $0.1*4 + $0.1*10cm 2 = $1.9 $517,275 TOTAL $2,518,804 Savings 66% Repeaters 7% Chips 1%

9 Latency Estimation Component Amount w/ Optical All Electrical Longest intergroup cable (80m) 400ns 464ns Average intergroup cable (25m) 125ns 145ns Connectors (total) 4ns 2ns Board (1/4m) + SERDES 2.25ns 2.25ns Longest Endpoint to (2/3m) 3.33ns 3.33ns Longest Intra-Group Wires (3m) 15ns 15ns Chip to Chip Intergroup WC 410ns 472ns Chip to Chip Intergroup AC 135ns 153ns Chip to Chip Intragroup 22ns 22ns Chip to Chip Endpoint to 10ns 10ns Chip Latency (2Ghz clock) 20ns 20ns

10 Chip Area Component Area (um2) Crossbar (51x51x20) 52,020 Inter-group Buffers, each: 17,000 (850ns*40Gbps = 34000b) Inter-group Buffers, Total 255,000 Intra-group buffers, each: 500 (25ns*40Gbps = 1000b) Intra-group buffers, total 10,500 Endpoint buffers, total 4,500 Total XBAR, Buffer 322,020 Other stuff 177,980 Chip Area 500,000um 2 Perimeter I/O Needed 2,040 Tbps Perimeter Needed 20.4mm Die Size (2:1 aspect) 7mm x 3.5mm

11 Power Estimation Item Power - Optical Power - Electrical Optical Cables Inter 108,900W NA Electrical Cables Inter NA 0W Electrical Repeaters NA 45,738W End Point Wires 0W 0W Intra-Group Wires 0W 0W Chip Serdes 29,711W 29,711W Data crossing Chip (10mm) 14,855W 14,855W Reading/Writing Memory 1.486W 1.486W TOTAL 153kW 90.3kW 63kw*24hr*365*$0.1 = $55188/year

12 Simulation Setup Simulate a single group 330 node generators 330 group generators Group Traffic Generator Simulated Group R R R R R R R R Node traffic generator

13 Simulated Group 22 routers with 51 ports each Fully connected 20-bit flits 10 flits per packet Group Traffic Generator 300 cycles Progressive adaptive routing 4 virtual channels R 44 cycles Simulated Group R 10 cycles N All 10 hotspots reside in the simulated group

14 Node Traffic Generator Hotspot RDMA UR RPC X500 WC RPC X500 Request To network Reply From network

15 Group Traffic Generator Respond to requests from the simulated group Generate fake requests based on feedback from the simulated group The four traffic types Adaptive cross traffic Added effects Simulated delay on messages Sending a portion of the traffic adaptively

16 Results Throughput 30 traffic iterations in 6.8 million cycles ~17% endpoint throughput ~24% global channel utilization Latency (simulated group) UR RPC transaction: 1250 cycles b WC RPC transaction: 2300 cycles 36 RDMA transaction: Gbps cycles Hotspot transaction: 1400 cycles %

17 Simulation Stability Generation Rate (Packets/cycle) Feedback Generation Variables Cross Traffic UR Traffic WC Traffic Simulation time (10000 cycles)

18 Simulation Stability 1 Feedback Adaptivity Variables Fraction of total packets WC Adaptive UR Adaptive Simulation time (10000 cycles)

19 Throughput Network Throughput Global channel Node Throughput (flits/cycle) Simulation time (10000 cycles)

20 Throughput Instantanious node composition Hotspot RDMA UR RPC WC RPC

21 Latency Packet Network Latency Min Nonmin Source Nonmin 1-hop Latency (cycles) Simulation time (10000 cycles)

22 Progressive Adaptive Routing Adaptive routing success Source 1-hop Fraction of total packets Simulation time (10000 cycles)

23 Hotspot Packet Network Latency Min Nonmin Source Nonmin 1-hop Latency (cycles) Simulation time (10000 cycles)

24 Hotspot Network Throughput Global channel Node Throughput (flits/cycle) Simulation time (10000 cycles)

25 Other Considerations Outstanding requests and latency ~28% network throughput with 8 outstanding request Use concentration to increase global channel utilization Higher global channel utilization Higher latency Lower endpoint throughput

26 1 0.8 Instantanious node composition Hotspot RDMA UR RPC WC RPC