InfiniBand* Architecture

Transcription

1 InfiniBand* Architecture Irv Robinson - Intel Mike Krause - Hewlett Packard Dennis Miller - Intel Arland Kunz - Intel Intel Labs

2 Server Labs Wednesday 10:15-12:15 12:15 IPMI Part 1 Platform Management Technologies Technical details on the IPMI 1.5 LAN and Modem platform management specifications Plus related initiatives such as DMTF Pre-OS Working Group, Metolious-2, and a proposed PCI management bus 2:15-4:15 IPMI Part 2 IPMI v1.0 Conformance Test Lab. Hands-on with the latest IPMI v1.0 Conformance Test Tool Plus labs on IPMI and IPMB messaging, information on test customization, and application of the tool for IPMI-based systems development Intel Labs

3 Agenda Architecture Overview Enabling of Flexible Solutions High Speed Serial Links Intel s Industry Enabling Efforts Intel Labs

4 InfiniBand* Architecture Overview Irv Robinson Intel Corporation Co-Chair Chair - Technical Working Group InfiniBand* Trade Association

5 Agenda Why InfiniBand* Architecture is Important InfiniBand Architecture and Components The Fabric as the Core Concept InfiniBand Hosts, Targets & Management The Fundamentals of Workflow Specification Milestones

6 InfiniBand* Architecture and Components InfiniBand Architecture Describes a System Area Network Unified Fabric for use between elements of computer systems Support for multiple concurrent workloads 1st Order Fabric (memory interface semantics described in the specifications) Abstracted usage model for sophisticated computing hardware & software Message Passing foundation Peer-to to-peer capabilities Virtual memory and multi-threading threading aware

7 InfiniBand* Architecture and Components So What? The application world is moving to distributed message passing applications The demand for clustered & networked systems is increasing rapidly Everything needs to be connected Scalability has got to move out of the box Downtime is out - Hot Plug is in Flexibility is required to accommodate technologies evolving at different rates & in different ways I/O infrastructure separation from processor complex Fabric interface closer to memory & processors Increased processing capabilities in all units

8 InfiniBand* Architecture and Components Scalable Systems are a Pain Today Computing Node Computing Node Computing Node All fabrics are connected to every node System Area Spaghetti IPC Interconnect IP Network Storage Area Network Multiple, function-specific specific,, fabric types are required, with: Multiple technology evolution trends Multiple administration strategies Multiple interfaces And they all funnel into...

9 InfiniBand* Architecture and Components Legacy Host Architecture Slot-based I/O connection Interface to memory via an I/O Bus Low level of abstraction (load/store) CPU CPU Host Interconnect Mem Cntlr Sys Mem I/O Bus Adapter Scaling strategy is adding buses Function-specific fabric

10 InfiniBand* Architecture and Components An InfiniBand Unified Fabric System Computing Node Computing Node Computing Node Eliminates the need for slot-based computing nodes Hot Plug scalability can be practical IP Network InfiniBand Switched Fabric Storage Area Network Much reduced intra-system spaghetti Function-specific fabrics at edges of system. Less intrusive evolution model.

11 InfiniBand* Architecture and Components The InfiniBand Architecture Model Target TCA CPU CPU Host Interconnect Mem Cntlr Sys Mem HCA Link Link Switch Link Link TCA Target xca Router Network Router Link

12 InfiniBand* Architecture and Components The Fabric is the Core Concept Switch Routes packets based on destination Local ID, service level Link Link Link Link Link Switch Link Switch Link Link Link Link Link Link Link Link Links 2.5 Gbits/s signaling rate Dual Simplex Multiple link widths: 1x, 4x, 12x Auto-negotiation to mutually acceptable width and signaling rate Common backplane connector(s)

13 InfiniBand* Architecture and Components Transport Concepts Transaction Message Message Message Packet Packet Packet Packet Packet Packet Packet Packet Packets Routable unit of transfer Messages consist of multiple packets Automatic segmentation & re-assembly Virtual Lanes & Service Levels for QoS & Traffic Shaping

14 InfiniBand* Architecture Fabric Partitions Host A Partition 3 I/O D Host B Partition 1 InfiniBand Fabric Partition 2 I/O A I/O B I/O C

15 InfiniBand* Architecture Subnets & Routers Link Link Link Link Link Link Link Link Router Transports packets between subnets Intra-Router links may be InfiniBand links or other

16 InfiniBand* Architecture and Components InfiniBand Host Architecture Host Channel Adapter CPU CPU Host Interconnect Mem Cntlr Sys Mem HCA Link Link Connects memory controller to fabric through one or more links Provides work queues for posting work requests Provides RDMA engines Manages transport functions Supports memory translation and protection

17 InfiniBand* Architecture and Components I/O Concepts I/O Unit One Target Channel Adapter One or more I/O Controllers IOC TCA Link Link Target Channel Adapter Provides link and transport services to I/O Controllers Peer-to-peer enabled Switch Link Link TCA IOC IOC I/O Controller Target of I/O request messages May be storage or network Request protocols may be standard or proprietary, at low levels or high

18 InfiniBand* Architecture and Components Management Enabling IPMI SNMP CIM ACPI DMI & many more Industry Standards for Network & Sys Management Configuration Management Determination of end node attributes System diagnostics Discovery Switch provisioning Fabric operations Fabric Management Partition Management Creation & Administration of Partitions

19 InfiniBand* Architecture and Components Work Queues The basic mechanism for inter-endpoint endpoint communications Dual Simplex model Work Queues come in pairs - Queue Pairs (QPs) Work is scheduled by posting the request to the queue Poster notified when request completes Outbound Send 5 RDMA Wr 4 RDMA Rd 3 Send 2 Send 1 HCA or TCA Memory Recv 1 Recv 3 Recv 2 Inbound Link

20 Work Queues, Channels, and Connections Connections A connection is a logical association of work request producer / consumer entities Over one or more channels Host Driver Connection IO Controller I O I O Channel

21 Work Queues, Channels, and Connections Channels and Service Types QP - 1 QP - 2 I O I O Channel A channel is an association of QPs InfiniBand* architecture supports several communication service types: Connected Reliable Connected Unreliable Datagram Reliable Datagram Raw Datagram

22 InfiniBand* Architecture Partitioning & Software Work Flow Verbs enqueue Work Requests Verbs abstract the HCA from the application, and the API from the HCA Not directly visible to end-user Work Request Verbs WQE WQE WQE Work Queue HCA Hardware Work Completion CQE CQE Completion Queue

23 InfiniBand* Architecture The Verbs Model An InfiniBand architecture-aware aware O/S assumes capabilities described by Verbs, but assumes its own abstracted view O/S O/S Interface (Driver) HCA Interface HCA Hardware (Vendor Specific) An O/S specific driver interacts with HCA, and abstracts HCA functions and parameters to O/S view The HCA provides a vendor-specific interface with functions and parameters. Non-abstracted view of commands & descriptors. The HCA hardware must provide a specific set of functional capabilities dictated by Verbs

24 Summary InfiniBand* Fabrics are System Area Networks for interconnecting elements of computing systems A message-based, highly abstracted architecture with hardware assists and direct memory interface Host and Target interface capabilities specified independent of hardware and software

25 Specification Milestones Q Working Draft Available to all members via web site Q Final Draft Functionally complete Release for comments Mid-2000 Release

26 Agenda Architectural Review Enabling of Flexible Solutions High Speed Serial Links Intel s Industry Enabling Efforts Intel Labs

27 Flexible Interconnect - Flexible Solutions Michael Krause Hewlett Packard Company Link Working Group Co-Chair Chair InfiniBand SM Trade Association

28 What is a flexible interconnect? A technology inflection point Changes the way we think about problems Delivers innovative solutions to customer problems Creates new customer and industry opportunities Built with an eye on the future Well-defined, layered architecture Architecture evolves at the rate of technology Open multi-vendor inter-operability from day one Strong customer investment protection Can be tailored to fit the customer requirements

29 Paradigm Shift

30 Paradigm Shift Paradigm shift is: Single, cohesive, open paradigm: RAS Management Application Communication Point-to to-point, switch-based Message-based communications Applications direct map to H/W Standardized hardware semantics H/W Semantics == Application paradigm H/W implements standard comms ops H/W implements Frees software to focus on application requirements True peer-to to-peer communication End Node End Node End Node End Node Switch End Node Subnet A Switch Switch End Node End Node Switch End Node End Node End Node

31 Reduce Application Design Impact Map Application paradigm to hardware transport services Reliable Connection (RC) Unreliable Connection (UC) Reliable Datagram (RD) Unreliable Datagram (UD) Multicast (optional) Raw Packet (optional) Off-load application operation with Reliable communication: HW generates acknowledgments for every packet. HW generates / checks packet sequence numbers HW rejects duplicates, detects missing packets Client transparent recovery from most fabric level errors. Client transparent recovery from most fabric level errors.

32 Innovative Fabric-level Services: Virtual Lane (VL) Packets Mux De- Mux Packets Multiplex multiple independent data streams onto a single physical link which provides: Differentiated services on a packet-boundary basis Increase fabric utilization in the face of head-of of-line blocking on a given VL and via VL-based routing across multiple paths Support for up to 16 VLs with 1 VL reserved for fabric management Implementations shall support a minimum of 1 VL for application usage and 1 VL for fabric management IBA defines a VL mapping algorithm to ensure inter-operability between endnodes which support different numbers of VLs.

33 Flexible Topology Building Blocks Switch Routes packets within a subnet. May support VLs Multiple link width - 1 / 4 / 12 Partitioning / Multicast (optional) Router Routes packets between subnets IBA subnet-to to-iba subnet IBA subnet - to disparate fabric to - IBA subnet e.g. 10 GbE joining two InfiniBand subnets IBA subnet - to disparate (multi-protocol) e.g. 10 GbE / OC 192 subnet End Node End Node End Node End Node Subnet A Switch Switch Switch End Node Subnet B End Node Router Switch End Node End Node

34 Flexible Topologies Match topology to customer reqs Single-board Integrated multi-module module Active or passive backplanes Active or passive chassis Single-subnet Multiple subnets Disparate fabric linking IBA subnets Variety of network topologies Trees Loops K-ary cubes CPU CPU MEM CTL HCA Switch Switch Enet FC FC GigE GigE SCSI SCSI SCSI SCSI SCSI Enet GigE GigE

35 Innovative Fabric-level Protection: Partitioning H/W enforced protection Enumeration Only see what you re allowed while transparently sharing common fabric / endnodes Per QP Packet-filtering P_Key (16-bit) Endnode / Switch / Router Silent discard - opt. Alarm gen Partition Manager Within / across subnets Single-point management End Node Subnet B End Node End Node Switch End Node Router Switch End Node Router End Node Switch Switch Switch Switch End Node End Node End Node End Node End Node Subnet A End Node

36 I/O Sharing Increased resource utilization via I/O sharing For example, shared console, boot device, 10 GbE, etc. I/O Sharing resource requirements function of: Transport service (e.g. QP / connection) Partition - QP per supported partition Each QP may be assigned a different QoS Bandwidth, priority of service, etc. Customer benefits: Reduced total cost of ownership Reduced number of endnodes to manage Improved load-sharing for high-speed devices / backbones

37 RAS Benefits Leverages industry s Five 9 s Experience Fault-zone isolation Redundant components / fabrics Multi-port channel adapters Multipath support Within and across subnets Mirroring Load-balancing Active-Active paradigm Delivers customer with maximum value while providing transparent fail-over solutions Hot-plug / removal of all IBA components CPU / Mem Complexes HCA Switch TCA ENET HCA Switch Chassis

38 Paradigm Shift - Customer Benefits Enables widest application / management base Protects customer investment Supports new application paradigms Tailor solutions to meet customer environment Improved latency tolerance Improved system efficiency / performance New hardware topologies Semantics remain constant independent of distance Subnets to improve access control, management, QoS, performance Shared I/O endnodes to reduce total cost of ownership Single-point and distributed management support Price / Performance / RAS / Management Trade-off capabilities to meet operational requirements

39 Hardware Model & Benefits

40 Component Communication Component communication via: ASIC-to to-asic Board-to to-board Chassis-to to-chassis via cables Standardized: Signaling Cables - Copper, Optical Connectors - Backplane, face- plate Customer benefits: Multiple solution sources Open standard - inter-operability Flexible solution deployment CPU CPU CPU CPU TCA FC MEM CTL HCA Switch TCA SCSI Router TCA ENET

41 Multiple Form Factors Four adapter form factors supported: Single wide / height (100 x 240 mm), Single-wide / double-height Double-wide / single-height, Double-wide / double-height Large variety of adapter solutions possible, e.g. array controllers Face plate can support quad SCSI / HSSDC connectors Optional redundant backplane connector Front to Back Airflow Cover Carrier Board Face-plate

42 Flexible H/W Management Built-in in In-band management Defined management operations transmitted via InfiniBand* link Optional I 2 C bridge in InfiniBand* ASIC On-board power regulation Eliminates power mgmt issues as technology evolves Aux power for system management / low-power devices Board bus; Link ASIC local I 2 C backplane I 2 C Board I 2 C access to VPD, LED s, sensors, etc. Backplane access to chassis VPD, slot In-band ACPI & wake-on LAN / Link Out-of of-band board & chassis power control

43 Actively Managed Chassis Provides Increased Functionality Detect presence Control power Control Hot-swap In-band view of: Chassis state Slot population Private Chassis Devices InfiniBand* Inter-chassis Links Switch Module Module Module Chassis Private Mgmt Mgmt Entity Funcs IB-ML Links

44 Hardware Model Benefits Open, inter-operable hardware components Multiple sources - Improved competition Reduced customer / vendor costs Development, manufacturing, support, etc. Variety of h/w communication paradigms Innovative and varied designs / packaging Multiple adapter form factors Tailor design to optimal form factors Switch / router may be implemented in an adapter Passive backplane for improved RAS Integrated management for adapters, chassis, backplanes Complete solution from day one

45 Software Model & Benefits

46 Verbs / Channel Interface Work Request Upper Layer Protocol Work Completion Data Segment 1 Verbs Data Segment 2 Work Queue Element Channel Interface Completion Queue Element Data Segment 3 Message Work Queue Element Work Queue Element Send or Receive Queue Completion Queue Element Completion Queue Element Completion Queue Verbs define abstract h/w semantics - do not define an API Channel interface is implementation-specific - combination of HCA Driver / HCA H/W

47 Memory Management Controls HCA access to user and system memory Provides memory access using virtually contiguous ranges Allows both drivers and user-level applications to initiate and control access to memory Local memory protection Prevents applications from disturbing each other s data Remote memory access control Controls access from other hosts and devices to local memory Memory management provides OS-bypass operation support.

48 Memory Mgmt Verb Categories Protection Domains ( PDs ) PDs provide a fundamental basis for access control by associating Queue Pairs, Regions, and Windows Memory Regions ( Regions ) Regions enable and control HCA access to host memory, both for local and remote operations Memory Windows ( Windows ) Windows enable enhanced control over remote access to host memory Fine-grain memory protection (byte-level) Refined remote key (R_Key) semantics

49 Memory Region Registration & Local Operation Access System Memory Consumer Process Consumer Process Virtual Address Space Data Buffer Register VA range L_Key Work request (VA, L_Key) Data (PAs) Operating System Service Register (VA range, PAs, L_Key) HCA

50 Memory Region Registration & Remote Operation Access System Memory Consumer Process Data Buffer Register VA range L_Key & R_Key Data (PAs) Operating System Service Register (VA range, PAs, L_Key, R_Key) HCA RDMA op (VA, R_Key) Remote Agent

51 Memory Window Allocation, Binding, & Remote Access System Memory Consumer Process Data Buffer Allocate Window Windo w R_Key Bind (VA, R_Key) Data (PA) Operating System Service Allocate Window HCA RDMA op (VA, R_Key) Remote Agent

52 Software Model Benefits Verbs define s/w view of abstract InfiniBand* h/w semantics Used with HCAs to provide complete architecture application access Allows ISVs and OSVs to implement API over common set of h/w semantics Improved application portability ISVs / OSVs free to implement APIs to meet their needs Verbs may be implemented under legacy APIs Transparent legacy application support Two-levels of memory protection Allow vendors to tailor solution to meet customer price / performance / protection requirements

53 Management Model & Benefits

54 Comprehensive Management Infrastructure Partition InfiniBand TM TM Management Mgmt Mgmt Connection Fabric Fabric Mgmt Baseboard Mgmt Mgmt Mgmt Mgmt Mgmt Device Device Mgmt Mgmt S N M P Fabric Management Fabric Administration Partition Management Connection Management Device Management Baseboard Mgmt Subnet Services SNMP Tunneling Fabric Topology & Fabric Health Configuration Query & Update Communication policy enforcement Connection / RD establishment primitives Device Enumeration, diagnostics In-line baseboard instrumentation Layer 2 to Layer 3 servicess Native Support of SNMP

55 Defined Fabric Management Operations Notices are polled via FabricGet(Notice) operation Traps are asynchronously sent to well known target Endnode FabricGet() FabricGetResp() Endnode Notice Polling Model Subscribed Endnode FabricReport( Macid 3 ) FabricReport( ACK ) FabricTrap ( Error ) Trap forwarding model Consistent management operations across all endnodes Fabric Manager

56 Management: RAS Features Diagnostics Fault isolation Trap and Event forwarding Access Control Partitions Management keys Dynamic configuration Hot-plug / removal Channel Migration Redundant fabric support Fabric fail-over support Transparent client fail- over Managed / Automatic

57 Communications are well known Fabric Mgr Fabric Mgmt Agent NODE VERBS LAYER Perform Connect Mgr Mgr General Svc Interface Redirector QP 0 is Fabric Mgmt VL 15, QP 0 is reserved for fabric management QP 1 is General Service Q Interface P GSI can forward packets to 1 interested requester Can also send redirect to LID n requester to refer to appropriate QP Q P 0

58 Management Benefits Management designed in - not an afterthought Supports current and future management technologies (e.g. SNMP, CIM) Reduced development costs Reduce total cost of ownership Complete infrastructure solution Status, control, hardware, software, chassis, etc.

59 InfiniBand TM Architecture Benefits Summary

60 Benefits - Fabric Level Hardware-implemented Transport services Link, Network, and transport layers implemented in HW for faster data movement with lower overhead. Supports recovery from permanent errors when redundant paths configured. Dual routing mechanisms (switches & routers) Fast, efficient local routing based on small local headers. Global routing uses standard RFC2460 IPv6 header Large address space Eliminates having to implement complex protocols to preserve addresses Can address trillions of I/O devices Robust, high-performance link layer. Provides link-level level flow-control and QoS End-to to-end dynamic rate control and congestion management in next release

61 Benefits - Application Level Addressing uses IPv6 Smooth integration into Internet world Hides address / fabric complexity from applications Wide range of transport services in hardware Reliable / Unreliable Connection, Reliable / Unreliable Datagram Multicast, Raw packet Wide range of Message Types Sends, RDMAs, Atomics Multiple memory management paradigms / protection Memory regions and Memory windows Region-level and byte-level Efficient work element post model Matches current application / driver base Encourages and allows innovation

62 Benefits - Solution Level I/O, IPC and raw protocol support are all part of fundamental architecture. Don t re-architect Device Driver Architecture. Only providing I/O bus replacement for data transport services Hides fabric complexity from endnodes End-node node apps know other endnodes only by addresses - paths and fabric-level local addressing is hidden from drivers / adapters Central fabric management Error recovery handled in fabric management / channel adapters Reduced total cost of ownership Single, cohesive RAS architecture Architecture built for the long-haul Forward and backward compatibility

63 Agenda Architectural Review Architectural Key Topics High Speed Serial Links Intel s Industry Enabling Efforts Intel Labs

64 InfiniBand* Interconnect Architecture Overview Dennis Miller Sr. Signal Integrity Engineer Intel Corp. Server Architecture Lab Intel Labs

65 Agenda Interconnect Architectural Overview Interconnect Key Topics Boards Cable/Connectors Serial/De-serializer Component Interconnect Considerations Intel Labs

66 InfiniBand* Architecture Interconnect 1X, 4X, 12X Focus CPU CPU Host Interconnect ASIC Mem Cntlr Sys Mem H P SerDes Bd/Cable/Connectors Target TCA Link HCA Link Switch Link Link xca HCA = Host Channel Adapter TCA = Target Channel Adapter TCA Target xca Link Router Network Router Src: InfiniBand* Trade Association Intel Labs

67 Interconnect Goals Competitive cost for every server market tier 1X, 4X, 12X backplane/cable connections Interconnect to support optical and copper Expandability to future 2.5 Gbit/sec+ serial technologies Interconnect electricals support integrated or external serial/de-serializers serializers Intel Labs

68 InfiniBand* Interconnect Concept Server Router Storage I B L i n k IB Link InfiniBand Link RAID Disks FC Disk Storage Subsystem Remote Client Rack Optic Link I/O Chassis Power Supply IB Link IB SCSI Fiber Channel Optic Link 2.5Gb IB Link Disk Array IB Link Intel Labs

69 Board Focus + - Eye Crosstalk Tx Rx Jitter Blocking Capacitor Vias Reflection Z 0 Z Due to mismatch impedance Diagram Differential Tx - Loss + Rx - Trace Corner Trace EMI EMI Equalization Resistor Capacitor To ensure loss is freq indep InfiniBand* ASIC Clock 250 MHz SerDes Rx Tx Tx + Tx - Rx + Rx - Connector Cable 10-bit parallel bus Cable Gauge/Length Radius Bend EMI EMI Intel Labs

70 Cable/Connector Focus 2.5 Gb/s common backplane connectors and 1X, 4X, 12X cable connector(s) High volume manufacturing Connector type and size Board design rules and assembly Co-axial cable-type considerations Quad, twin lead, spectra-strip strip Cable mechanical considerations Multiple lengths, bends, bundling Cable/connector environmental performance Intel Labs

71 SerDes Component Focus Gb/s 2.5+ Ext./Internal SerDes Multi-Port Optic/Cu 2.5 External SerDes Single Port Optic/Cu Integrated SerDes Multi-Port Optic/Cu External SerDes Quad Port Optic/Cu SerDes Considerations: Silicon technology Voltage Package type/size Port size External/integrated 1.25 External SerDes Single Port Optic/Cu Internal SerDes Multi-Port Optic/Cu Solid Lines: Committed Product Dotted Lines: Potential Product year Intel Labs

72 Interconnect Considerations Interconnect Single-ended ended to differential Equivalent circuit to S-parameters S characterization of discontinuities Simulation models and tools Behavioral to transistor model 2D to 2.5D and 3D tools Measurements Microwave engineering set of measurements Differential measurements Intel Gb/s Physical Link Design Rules IDF IDF InfiniBand* Physical Link Workshop February from 4:30 4:30 to to 6:30 6:30 pm, pm, Mesquite A Intel Labs

73 Agenda Architectural Review Architectural Key Topics High Speed Serial Links Intel s Industry Enabling Efforts Intel Labs

74 Intel s Enabling Plans Arland Kunz Sr. Technical Marketing Engineer Fabric Component Division Intel Corporation Intel Labs

75 IHV H/W Enabling Strategy Focus on two objectives: Short Term - Enable for launch in 01 Storage Networking Long Term - Enable for cost and performance in 02/03 Storage Networking Vendors Can Make or Buy TCA Logic "All dates are provided for planning purposes only and are subject to change." Intel Labs

76 Today SCSI JBOD Array Ultra 2 SCSI FC-AL JBOD Array Crossroads* Storage Router GHz FC-AL World s First 2.5Gbit/Sec I/O Solution! LSI Logic* FC-AL JBOD Array LSI Logic* Controller Dell* Server GHz FC-AL InfiniBand* Architecture prototype links 2.5 Gbit/sec Compaq* Server Intel Labs

77 From Prototypes to Products Specs Development Kits (Host & Switch SDK, SDV) Samples (Host & Switch) Solutions H1 00 H2 00 H1 01 H2 01 Kits & Samples are Limited - Contact your Intel Representative Delivered by the InfiniBand* Trade Association "All dates are provided for planning purposes only and are subject to change." Intel Labs

78 Call To Action Pick up serial link & demo whitepapers Attend the link lab Visit the demo pavilion Speak to your Intel representative about SDK, SDV & sample availability Get InfiniBand* products on your roadmap for 01 Intel Labs

79 Intel Labs