Joe Pelissier InfiniBand * Architect

Transcription

1 ,QILQL%DQG $UFKLWHFWXUH 2YHUYLHZ Joe Pelissier InfiniBand * Architect Fabric Components Division Corporation August 22-24, 2000 Copyright 2000 Corporation. * Other names and brands are property of their respective owners

2 Agenda z Focus and Value Proposition z Architecture Model and Layers z Switches & Links z Host Architecture z I/O Architecture z Software Architecture z Management z Electro/Mechanical Copyright 2000 Corporation. *Other names and brands are property of their respective owners.

3 InfiniBand * Architecture Focus z Unified Intra-System Fabric Intra-box / Inter-box solution Minimize cabling Network diameters of <300 meters Server application architecture support RAS features necessary for 24x7 systems InfiniBand Architecture Focused on Server Interconnects Copyright 2000 Corporation. *Other names and brands are property of their respective owners.

4 Clustering with an InfiniBand * Fabric,QILQL%DQGOLQNV,QILQL%DQGRU)&,QILQL%DQG6ZLWFK,QILQL%DQGRU(WKHUQHW )& (WKHUQHW,3QHWV,QWHUQHW,QWUDQHW Copyright 2000 Corporation. *Other names and brands are property of their respective owners.

5 Value Proposition for InfiniBand * Architecture z Ease of clustering One fabric connection for all host I/O IPC, Storage I/O, Network I/O Improved upgrade/growth strategy z Scalability Specialized fabrics upgraded at InfiniBand fabric edge accrues to all cluster components Thousands of nodes per subnet / multiple subnets Cost-effective intra-subnet switches Multiple link widths for high bandwidth Copyright 2000 Corporation. *Other names and brands are property of their respective owners.

6 Value Proposition for InfiniBand * Architecture z Performance CPU-offloaded hardware support for message queuing, memory protection, fabric protocol processing z Reliability, Availability, Serviceability Redundant paths and/or fabrics In-band management Connector, backplane, baseboard, chassis, power Layered error management architecture Copyright 2000 Corporation. *Other names and brands are property of their respective owners.

7 Value Proposition for InfiniBand * Architecture z Flexibility Topology independent Multiple link widths / Cu and Optical Multiple units of transfer Auto-negotiation Multiple form factors / standardized connectors InfiniBand Technology is is the Complete Architecture for Server and Cluster Applications Copyright 2000 Corporation. *Other names and brands are property of their respective owners.

8 The InfiniBand * Architecture Model +RVW&KDQQHO$GDSWHUV IRUFRPSXWLQJSODWIRUPV Target TCA 7DUJHW&KDQQHO$GDSWHUV IRU6SHFLDOL]HG6XEV\VWHPV CPU IB Link IB Link CPU Mem Cntlr Sys Mem HCA IB Link Switch Router IB Link TCA Target Host Interconnect Host Interconnect 6XEQHWVFRQVLVWRI /LQNV 6ZLWFKHV Network or IB Router IB Link 5RXWHUVHQDEOH LQWHUVXEQHWFRPPXQLFDWLRQV ZKLOHSURYLGLQJVXEQHWLVRODWLRQ Copyright 2000 Corporation. *Other names and brands are property of their respective owners.

9 InfiniBand * Architecture Layers 8SSHU/HYHO 3URWRFROV &OLHQW 7UDQVDFWLRQV &OLHQW 7UDQVSRUW /D\HU,% 2SHUDWLRQV 0HVVDJHV,% 2SHUDWLRQV 1HWZRUN /D\HU 1HWZRUN,QWHU6XEQHW5RXWLQJ*,' 3DFNHW 5HOD\ 1HWZRUN /LQN /D\HU /LQN (QFRGLQJ 0HGLD $FFHVV &RQWURO 3K\VLFDO /D\HU (QG1RGH 3DFNHW 5HOD\ Copyright 2000 Corporation. *Other names and brands are property of their respective owners. /LQN (QFRGLQJ 0HGLD $FFHVV &RQWURO (QG1RGH 6ZLWFK 5RXWHU (QG1RGH 0$& 0$& 0$& /LQN /LQN 6XEQHW 5RXWLQJ /,' 0$&

10 Switches & Links Switch Forwards packets based on destination Local ID, service level Link Link Switch Link Link Switch Links Copyright 2000 Corporation. *Other names and brands are property of their respective owners. 2.5 Gbit/s signaling rate Dual Simplex Cu & Optical Multiple link widths: 1x, 4x, 12x Auto-negotiation to mutually acceptable width and signaling rate width and signaling rate Common backplane Common backplane connector(s)

11 InfiniBand * Link Protocol ([DPSOH,22SHUDWLRQ 0HVVDJH 5RXWDEOHXQLW RIWUDQVIHU 3DFNHW 0HVVDJH 3DFNHW 7UDQVDFWLRQ 0HVVDJH 3DFNHW 0HVVDJH 8SWR*% $XWRPDWLF 6HJPHQWDWLRQ 5HDVVHPEO\ Copyright 2000 Corporation. *Other names and brands are property of their respective owners. 0LQLPXP%078,%$DOVRVXSSRUWV %078ZLWK DXWRQHJRWLDWLRQRQSDWK

12 InfiniBand * Virtual Lanes 6HUYLFH/HYHOLV PDSSHGWRD9LUWXDO/DQH 9LUWXDO/DQHV9/ V 3DFNHWVVHQWVSHFLI\LQJ 6HUYLFH/HYHO (DFKOLQNLQDIDEULF PD\VXSSRUWDGLIIHUHQW QXPEHURI9/ V 3DFNHW 3DFNHW 3DFNHW 3DFNHW 3DFNHW 3DFNHW 6/ 6/ WR WR 9/ 9/ 0DS 0DS Mux 3K\VLFDO/LQN De- Mux 9LUWXDO/DQHVDUH GHGLFDWHGVHWVRIEXIIHUV 3DFNHWVDUHVHQWRQHDWDWLPH E\WHVWULSHGDFURVVZLGWK DQGELWVHULDORQHDFKFRQGXFWRU Copyright 2000 Corporation. *Other names and brands are property of their respective owners.

13 InfiniBand * Switch Attributes '/,' 6/ 3.H\ 3D\ORDG /RFDO,'WR 3RUWWDEOH 3DUWLWLRQ 7DEOH 6/9/ 7DEOH 3DUWLWLRQ 7DEOH,QILQL%DQG6ZLWFK Copyright 2000 Corporation. *Other names and brands are property of their respective owners.,qerxqg3dfnhwrqd 9LUWXDO/DQH 3RUW 3RUW 3RUW 3RUW 2XWERXQG 3DFNHWRQD 9LUWXDO/DQH 3RUW 3RUW 3RUW 3RUW

14 Partitions 3DUWLWLRQ +RVW$ LQWHUKRVW +RVW% 3DUWLWLRQ 3ULYDWHWR$,QILQL%DQG )DEULF 3DUWLWLRQ 3ULYDWHWR%,2% 3DUWLWLRQ 6KDUHG,2$,2&,2' 7KHSULPDU\IXQFWLRQRI3DUWLWLRQLQJ LVWRSDUWLWLRQWKH$GDSWHUVSDFH 3DUWLWLRQLQJFDQDOVREHXVHGWRDVVRFLDWH KRVWVRUGHYLFHVZLWKFRPPRQDWWULEXWHV Copyright 2000 Corporation. *Other names and brands are property of their respective owners.

15 InfiniBand * Host Architecture CPU CPU Mem Cntlr Sys Mem HCA Link Link Host Interconnect Host Interconnect Copyright 2000 Corporation. *Other names and brands are property of their respective owners. Host Channel Adapter Connects memory controller to fabric through one or more links Provides work queues for posting work requests Provides RDMA engines Manages transport functions Supports memory translation and protection

16 InfiniBand * I/O Architecture I/O Controller I/O Unit One Target Channel Adapter One or more I/O Controllers IOC TCA Target of I/O request messages May be storage, networking, or other Request protocols may be standard or proprietary, at low levels or high Link Link Target Channel Adapter Provides link and transport services to I/O Controllers Capabilities scale to match needs Switch Link TCA Link IOC IOC Copyright 2000 Corporation. *Other names and brands are property of their respective owners.

17 InfiniBand * Verbs Architecture 9HUEVDUHDQDEVWUDFWXSSHUOHYHODUFKLWHFWXUDOLQWHUIDFH 1RWLQWHQGHGWREHDQ$3,EXWWRLQVSLUH269$3,VDWWKH HTXLYDOHQWVHPDQWLFOHYHO $SSV $SSV,%0JPW$SSV 269/D\HU 8VHU6SDFH$3,V 8VHU6SDFH$3,V 269,%6XSSRUW,+99HUEV/D\HU,%+: $OO7%' 3HU269,%+&$ 3URYLGHU,%+&$,%3URYLGHU$3,V,%&RQVXPHU$3,V,%6HUYLFHV,%%RRW6XSSRUW 6WRUDJH 'ULYHUV /$1 /$1 'ULYHUV Copyright 2000 Corporation. *Other names and brands are property of their respective owners.

18 Verb Concepts z Queue Pair Represents a duplex communications endpoint z Work Request Requests communications operation z Completion Queue Provides completed operation status z Memory Region Source/sink area for communications data Copyright 2000 Corporation. *Other names and brands are property of their respective owners.

19 InfiniBand * Software Transport 6HQG4XHXH :RUN 5HTXHVWV &RQVXPHU :RUN &RPSOHWLRQV 9HUEV &KDQQHO,QWHUIDFH 4XHXH3DLUV 0HPRU\ 7DEOHV &RPSOHWLRQ 4XHXHV 5HFHLYH4XHXH Copyright 2000 Corporation. *Other names and brands are property of their respective owners.

20 InfiniBand * Memory Model z Memory Regions Virtual to Physical mapping QP consumers use virtual addressing, HCA performs VtoP mapping Explicit registration by consumer with the OS QP access to Regions managed through Protection Domains Similar to VI Architecture Copyright 2000 Corporation. *Other names and brands are property of their respective owners. 5HJLRQ 5HJLVWHUHG 9LUWXDO $GGUHVV 6SDFH 26 0DQDJHG 3DJH 3DJH 3DJH 3DJH

21 InfiniBand* Memory Model z Memory Windows Flexible & efficient RDMA access control to Memory Regions Consumer binds a preallocated Window to a specified portion of an existing Region :LQGRZ &RQVXPHU 0DQDJHG 5HJLRQ 5HJLVWHUHG 9LUWXDO $GGUHVV 6SDFH 26 0DQDJHG Copyright 2000 Corporation. *Other names and brands are property of their respective owners. 3DJH 3DJH 3DJH 3DJH

22 Transport Service Types X = allowed operation Reliable Connected Reliable Datagram Unreliable Connected Unreliable Datagram Raw Datagram Send X X X X X RDMA Read X X Send Queue Operation RDMA Write X X X Atomic Op X X Bind Window X X X Multi- Cast X 6HUYLFH7\SH Copyright 2000 Corporation. *Other names and brands are property of their respective owners.

23 InfiniBand * Management Model IB Mgmt Console Management Framework Mgmt. Applications Management Datagrams IB Managed Subnet End Node Subnet Manager Switch Switch Switch End Node Copyright 2000 Corporation. *Other names and brands are property of their respective owners. Data Repository SM Repository End Node To Peer Subnet Router Subnet Mgmt. Controller Controller Sensor LED VPD

24 InfiniBand * Management Model z One node/switch must emerge as a subnet manager Controls fabric topology and behavior Additional standby SM s supported z All nodes/switches must process management datagrams (MADs) Lightweight messages for fabric discovery, control, etc. Copyright 2000 Corporation. *Other names and brands are property of their respective owners.

25 InfiniBand * Management Model z Proxy agent defined for traditional management (e.g. SNMP, CIM) z Other services may emerge as needed e.g. Bandwidth Reservation Copyright 2000 Corporation. *Other names and brands are property of their respective owners.

26 Electrical Mechanical Architecture Objectives z Enable the industry z Promote interoperability Strict specification in these areas z Flexibility & choices Multiple links & form factors z Comprehensive management model z Satisfy the industry s RAS demands Copyright 2000 Corporation. *Other names and brands are property of their respective owners.

27 Form Factors )RXUDGDSWHUIRUPIDFWRUV VXSSRUWHG 6LQJOHZLGHKHLJKW [PP6LQJOHZLGH GRXEOHKHLJKW 'RXEOHZLGHVLQJOH KHLJKW'RXEOHZLGH GRXEOHKHLJKW /DUJHYDULHW\RIDGDSWHU VROXWLRQVSRVVLEOHHJDUUD\ FRQWUROOHUV )DFHSODWHFDQVXSSRUW TXDG6&6,+66'& FRQQHFWRUV 2SWLRQDOUHGXQGDQW EDFNSODQHFRQQHFWRU Copyright 2000 Corporation. *Other names and brands are property of their respective owners.

28 Mechanical View Module Guide Air Vents EMI Contacts Cover Carrier Module Ejector EMI Contacts Copyright 2000 Corporation. *Other names and brands are property of their respective owners. Air Vents (Front to Back Airflow)

29 InfiniBand* Physical Link z Interoperability between different suppliers z Interoperability among widths z Hot Pluggable z Allow either external or integrated SERDES z Drive 20 inches of printed wiring or 17 meters of cable z Retimers supported z Signal cross over in cable or backplane z Wake on event or in-band request z Achieve Worst Case BER of 1*10-12 Copyright 2000 Corporation. *Other names and brands are property of their respective owners.

30 Connections %DFNSODQH &RSSHU&DEOH 2SWLFDO&DEOH Copyright 2000 Corporation. *Other names and brands are property of their respective owners.

31 Actively Managed Chassis Provides Increased Functionality Detect presence Control power Control Hot-swap In-band view of: Chassis state Slot population 3ULYDWH &KDVVLV 'HYLFHV 6ZLWFK,QILQL%DQG,QWHUFKDVVLV/LQNV 0RGXOH 0RGXOH 0RGXOH 3ULYDWH 0JPW )XQFV &KDVVLV 0JPW (QWLW\ Copyright 2000 Corporation. *Other names and brands are property of their respective owners.,qilql%dqg0//lqnv

32 Power Management Model Power Domain - Host Power Domain - Switch Power Domain - I/O X3 X3 +&$ 3RUW 3RUW, 8 ; 3RUW 6ZLWFK 3RUW 3RUW 8 ;,2& 7&$ 3RUW,2&, 8 ;,2&,2& 7&$ 3RUW, 8 ; 0 0 0,28QLW 0RGXOH 3 3 'RXEOH:LGHRU7DOO0RGXOH6KRZQ 3 M* - indicates a virtual module which may or may not be physically realized in implementations. PM State Classes: I - I/O Controller, U - Unit, M - Module, X - Transceiver, P - Power Copyright 2000 Corporation. *Other names and brands are property of their respective owners.

33 Call to Action z Join the success of InfiniBand* Architecture Develop product to bring this new level of functionality to the market place. Learn more about InfiniBand * Attend the IDF InfiniBand * classes and visit the showcase Join the InfiniBand * Trade Association For more information visit: Copyright 2000 Corporation. *Other names and brands are property of their respective owners.