Building blocks for custom HyperTransport solutions

Transcription

1 Building blocks for custom HyperTransport solutions Holger Fröning 2 nd Symposium of the HyperTransport Center of Excellence Feb th 2009, Mannheim, Germany

2 Motivation Back in 2005: Quite some experience with PCI(-X) Convinced of HyperTransport protocol Lean and efficient, low latency, possibility of coherency, direct connection Highly suitable for networking and FPGA technologies Learned about HTX at ISC05 OK, now let s do this 2

3 Outline HT 1.x/2.x HT 3.x HT-based on-chip network Conclusion 3

4 HyperTransport 2.x Established Chip-to-chip and board-to-board interconnect technology AMD s Direct Connect Architecture Lean packet protocol Parallel links Dedicated source clocking No 8b/10b coding No clock recovery Variable link width and frequency Compared to PCIe Higher bandwidth Much lower latency Less protocol overhead 4

5 HT Core ncht version HT Cave unit 8 or 16bit HT links Efficient queue interface HT AMD Opteron HT HT Core NP CMD & DATA P CMD & DATA R CMD & DATA NP CMD & DATA P CMD & DATA User Module Each 96bit CMD & 64 bit DATA Resource usage HT R CMD & DATA HT Unit V4-FX60: 11% of slices V4-FX100: 7% of slices FPGA & ASIC technologies Xilinx, Altera, Lattice IBM 130nm Open source 5

6 cht version Same features as ncht Core Not open source Available from AMD under NDA Introduces 4 th queue for probes & broadcasts Some additional special packets Different device header HT Core Coherent Version Coherent Memory Controller/Cache Not part of cht Core Must be implemented separately Partial implementation of coherency protocol possible 6

7 HT Core Features Fully synchronous design Efficient pipelined structure Synthesizable Verilog HDL code Ultra low latency Incoming: 12 cycles Outgoing: 6 cycles Deseralizer Sy nc nii Decode RX Buffer Init Config CRC Gen Credit Serializer Mux Outputgen TX Buffer Xilinx V4 tech HT200 & HT400 mode 200MHz 8 or 16bit HT interface Xilinx V5 tech HT800 mode 200MHz 8bit HT interface (preliminary) IBM 130nm tech HT800 mode 400MHz 8 or 16bit HT interface 7

8 HT Core Status & Users Status HT Core FPGA proven ASIC ports ongoing 8

9 HTX-Board First available FPGA-based test bed with HTX connection Reference design of HTC Validated with HP DL145 G3 HP DL165 G5 Tyan N3600R / S2912E Supermicro H8QME-2+ (Iwill DK8-HTX) Features Xilinx Virtex-4 FX100 16bit wide HT link Six high speed serial links Flash memory Embedded DDR2-DRAM 9

10 HTX-Board Block Diagram Programming options JTAG PROM USB HT PHY Based on Parallel I/O Lowest latency method possible 10

11 HTX-Board Current Status Mature product, successfully used in Academia and Industry 4 th generation currently in production Demonstrator for developments of the Computer Architecture Group Ultra low latency interconnect networks Fine grain communication HT Core AMD, Dresden, Germany AMD, Bellevue, USA Cadence Design Systems, San Jose, USA Dialogic Corporation, Parsippany, NJ, USA Dolphinics, Oslo, Norway Google IBM, Yorktown Heights, NY, USA IBM, Research Center, China SUN Microsystems, CA, USA Technical University Chemnitz, Germany University of Karlsruhe, Germany Universidad Politecnica de Valencia, Spain Georgia Tech, USA TU Delft, Netherlands 11

12 HTX-Board as HyperTransport Verification Platform Use as Verification Platform Rapid Prototyping Station Bring up & test of new designs 12

13 HTX-Board as Interconnection Network Use as fully-featured development Six links enable direct networks No external switching required Special Special Cluster Cluster Features Features SPD SPD for for non-volatile non-volatile data data Programming Programming using using USB USB Examples EXTOLL Fine grain communication Valencia Cluster HT native extension Particle Physics Experiments Clock distribution & synchronization Data & control transfer Start-up Start-up latency latency ~ ~ 1us 1us See See paper paper presentation presentation at at WHTRA WHTRA tomorrow tomorrow Courtesy of Universidad Politècnica de Valencia 13

14 HT Core & HTX Board Performance Bandwidth [MB/s] DMA Bandwidth Xilinx V4 HT400 16bit Latency PIO Read Latency 0,400 0,350 0,300 0,250 0,200 0,150 0,100 0,050 0,000 Iwill [µs] HP [µs] HT200 0,371 0,330 HT400 0,195 0, HT-400 HP DMA write HT-400 HP DMA read Transfer Size [Byte] 180ns 14

15 Outline HT 1.x/2.x HT 3.x HT-based on-chip network Conclusion 15

16 HyperTransport 3.x HT3 supersedes HT1 with Higher frequencies Improved error handling Link splitting AC mode HTX3 specification published 2008 Support for additional CTL pair Improvements for FPGAs HT 1.x/2.x 400MT/s 2.8GT/s Opteron: up up to to 1GT/s HT 3.x 3.x 2.4GT/s 6.4GT/s Opteron: not not yet yet available 16

17 HTX3 Link splitting scenarios HT3 8bit HT3 8bit HT3 HT3 16bit 17

18 HT3 Core cht & ncht versions HT Cave unit Efficient queue interface Posted, Non-Posted, Request Each 96bit CMD & 128 bit DATA Resource usage V5-LXT110: 70% of slices Xilinx XC5V technology 8bit 150MHz core clock 16bit 300MHz core clock 2.4GB/s Current implementation 4.8GB/s Future plan, frequency challenging for XC5V See See paper paper presentation presentation at at WHTRA WHTRA tomorrow! tomorrow! 18

19 HTX3 Xilinx Board First available FPGAbased test bed with HTX3 connection HT3 PHY design Based on GTPs (up to 3.75GBit/s) Supports link splitting Supported by Features Features XC5VLX110T XC5VLX110T & 2x 2x LX50T LX50T 16bit 16bit wide wide HT3 HT3 link link 2 2 CX-4 CX-4 connectors connectors (IB, (IB, 10GigEth) 10GigEth) SO-DIMM SO-DIMM connector connector 19

20 Altera Stratix-IV based Single FPGA solution HT3 PHY design Based on GXs (up to 8.5GBit/s) HTX3 Altera Board (preliminary) 20

21 Outline HT 1.x/2.x HT 3.x HT-based on-chip network Conclusion 21

22 HT-based on-chip network Need to interconnect multiple modules FUs, RFs, TLBs, Caches, MCs Don t introduce protocol conversion Solution: HTAX Non-blocking crossbar Protocol derived from HT Support for more endpoints & source tags Extended packet length Fully configurable, scalable, low latency 6/12 cycles 3 cycles 22 2 cycles

23 Outline HT 1.x/2.x HT 3.x HT-based on-chip network Conclusion 23

24 Conclusion (1) Availability ncht Core Fully verified Sponsored by AMD Open source Downloadable for free after short registration cht Core Fully verified Sponsored by AMD Available for free, NDA required Contact AMD HTX-Board Fully verified Available from University of Heidelberg Contact us for pricing ncht3 Core Under development cht3 Core Design finished Extended verification phase Contact us Xilinx HTX3 Board Prototyping phase finished Contact us Altera HTX3 Board Board design finished Prototypes expected Q2/2008 Small volume expected Q3/

25 Conclusion (2) HTX-Board / HT Core PIO access: 180ns DMA BW: ~1.4GB/s USER Complete framework Turnkey solution for accelerator applications (c/nc) HT/HT3 Core HTAX crossbar RF TLB Caches Upcoming cmc HTX3 Board(s) / HT3 Core Design of cht3 Core finished! COMM 25