InfiniBand * Access Layer Programming Interface

Transcription

1 InfiniBand * Access Layer Programming Interface April

2 Agenda Objectives Feature Summary Design Overview Kernel-Level Interface Operations Current Status 2

3 Agenda Objectives Feature Summary Design Overview Kernel-Level Interface Operations Current Status 3

4 Objectives Programming interface defined and accepted in the Linux * environment Support multiple vendor HCAs Simplify accessing the InfiniBand fabric Decrease time to market for IBA solutions Minimize client code requirements to permit focused optimization efforts Provide features and abstractions common to most users 4

5 Agenda Objectives Feature Feature Summary Design Overview Kernel-Level Interface Operations Current Status 5

6 Feature Summary Provide transport level access to an InfiniBand fabric Expose all InfiniBand features Collection of verbs and management services Speed path for IO operations Provide support for common functions E.g. SAR, retries, timeouts, etc. 6

7 Agenda Objectives Feature Summary Design Design Overview Kernel-Level Interface Operations Current Status 7

8 Design Overview User-Level Access Layer Layer Kernel-Level Access Services Management Services Resource Management Work Processing Memory Manager User-Level Verbs Verbs Library Kernel-Level Access Layer Layer User-Level Support Services Management Services Resource Management Work Processing Memory Manager Verbs Verbs Provider Driver Driver 8

9 Agenda Objectives Feature Summary Design Overview Kernel-Level Interface Operations Current Status 9

10 Kernel-Level Interface Consists of 77 function calls 39 are Verbs programming interfaces 38 are Access Layer extensions APIs in italic may not be needed by most clients Most Access Layer extensions perform MAD transactions on behalf of the client With SA, CFM, and Device Management Agents Most MAD transactions are asynchronous A client callback function is invoked on completion A client can specify synchronous operation on many MAD transactions 10

11 Kernel-Level Interface Functional Groups: Access Layer Instance CA Services Domain Services Memory Registration QP Services CQ Services EE Context Services Address Vector Services Multicast Services Datagram Services Service ID Resolution Connection Management Device Management Service Record Management Information Query Notification Services 11

12 Kernel-Level Interface Access Layer Instance Open and close an instance of AL Defines resource domain boundaries Simplifies resource cleanup Function calls: ib_open_al ib_close_al 12

13 Kernel-Level Interface CA Services Obtain CA GUIDs and access to a CA List all channel adapters in the local system Required to access resources on a given channel adapter Function calls: ib_get_ca_guids ib_open_ca ib_query_ca ib_modify_ca ib_close_ca 13

14 Kernel-Level Interface Domain Services Create and destroy Protection and Reliable Datagram domains Function calls: ib_alloc_pd ib_dealloc_pd ib_alloc_rdd ib_dealloc_rdd 14

15 Kernel-Level Interface Memory Registration Register and deregister memory Function calls: ib_reg_mem ib_reg_phys ib_reg_shared ib_query_mr ib_modify_mr ib_modify_phys ib_dereg_mr ib_create_mw ib_query_mw ib_bind_mw ib_destroy_mw 15

16 Kernel-Level Interface QP Services Manage QPs and post work requests Function calls: ib_create_qp ib_query_qp ib_modify_qp ib_destroy_qp ib_post_send ib_post_recv 16

17 Kernel-Level Interface CQ Services Manage CQs and process completions Function calls: ib_create_cq ib_modify_cq ib_query_cq ib_destroy_cq ib_poll_cq ib_rearm_cq 17

18 Kernel-Level Interface EE Context Services Manage End-to to-end Contexts Function calls: ib_create_eec ib_query_eec ib_modify_eec ib_destroy_eec 18

19 Kernel-Level Interface Address Vector Services Manage Address Vectors Function calls: ib_create_av ib_query_av ib_modify_av ib_destroy_av 19

20 Kernel-Level Interface Multicast Services Join and leave a multicast group Function calls: ib_join_mcast ib_leave_mcast 20

21 Kernel-Level Interface Datagram Services Initialize and bind a QP for datagram service, including MADs Function calls: ib_init_dgrm_svc ib_reg_mad_svc ib_coalesce_mad ib_local_mad 21

22 Kernel-Level Interface Service ID Resolution Register a service and perform SIDR Function calls: ib_reg_sidr ib_dreg_sidr ib_sidr_req ib_sidr_rep 22

23 Kernel-Level Interface Connection Management Manage RC, UC, or RD connections Function calls: ib_cm_listen ib_cm_cancel ib_cm_req ib_cm_rep ib_cm_rtu ib_cm_rej ib_cm_mra ib_cm_lap ib_cm_apr ib_cm_dreq ib_cm_drep ib_force_apm 23

24 Kernel-Level Interface Device Management Manage IO controllers Used by target systems to export IOCs to the fabric Function calls: ib_create_ioc ib_destroy_ioc ib_reg_ioc ib_add_svc_entry ib_remove_svc_entry ib_reject_ioc 24

25 Kernel-Level Interface Service Record Management Register and deregister a service record Registers a service with the SA Function calls: ib_reg_svc ib_dereg_svc 25

26 Kernel-Level Interface Information Query Obtain information from a class manager Built-in in queries (others through client MADs): Service Records by Service Name Service Records by well-known Service ID IOC Records by local port GUID Service Entries by IOC GUID Node Record by node GUID Port Record by port GUID Path Records by port GUID pair Path Records by GID pair Function call: ib_query 26

27 Kernel-Level Interface Notification Services Notify client of subscribed events Register for local and remote event notification Register for IOC assignments Function calls: ib_reg_notify ib_dreg_notify ib_subscribe ib_unsubscribe 27

28 Agenda Objectives Feature Summary Design Overview Kernel-Level Interface Operations Current Status 28

29 Operations Initialization Required Steps: 1. Open an instance of AL 2. Locate and open a CA 3. Allocate a Protection Domain Reliable Datagrams QPs also require: Allocate a Reliable Datagram Domain Create an End-to to-end Context Passive side of RD connection may do these steps in the connection callback function 29

30 Operations Service Records Service provider registers a Service Record with the SA Service consumer queries the SA for a list of Service Records Query methods: By a Service Name, e.g., CFM.IBTA By a well-known Service ID Clients can implement their own SA query methods using MAD services 30

31 Operations Service Records Service Provider Service Consumer AL AL ib_reg_svc ib_query SA SA Service Record SA Database Service Record Service Record Service Records 31

32 Operations Obtaining Path Records Client issues a query for path records Based on port GUID pair Based on port GID pair AL performs MAD transactions with SA AL invokes client callback function with a list of path records 32

33 Operations Obtaining Path Records Client Client AL AL ib_query SA SA Path Record Query SA Database Path Records Path Records Path Records 33

34 Operations Service ID Resolution UD service provider registers a Service with the local CM CM invokes client callback with SIDR request Client may accept or reject UD service consumer requests SIDR with the local CM CM invokes callback with SIDR response 34

35 Operations Service ID Resolution UD UD Service Provider UD UD Service Consumer ib_sidr_reg_t ib_sidr_rep_t ib_sidr_req_t AL AL ib_reg_sidr ib_sidr_rep AL AL ib_sidr_req CM CM UD Service Provider CM CM 35 InfiniBand Fabric

36 Operations Registering a Service Provider A ib_cm_listen() Listen handle ib_reg_svc() status Register callback AL Provider A Registering a connected service Svc Record response SA Provider B Registering a datagram service Svc Record response AL ib_create_cq() CQ handle ib_create_qp() QP handle ib_init_dgrm_svc() status ib_reg_mem() Mem handle ib_post_recv() status ib_reg_sidr() status ib_reg_svc() status Register callback Provider B 36

37 Operations Locating a Service Consumer ib_query(name) AL SA AL Provider status Query callback ib_query(gids) status Query callback Svc Query(name) Svc Records Path Query(GIDs) Path Records Transactions below this line are required for datagram services ib_sidr_req() status SIDR callback SIDR Req SIDR Rep SIDR callback ib_sidr_rep() status 37

38 Operations Establishing a Connection Client submits a connection request Request indicates client/server or peer-to to-peer protocol for an RC, UC, or RD connection Client provides QP / ECC, CQs, primary path, and optional alternate path CM invokes callbacks when the next connection state is reached Client calls CM to transition the connection state 38

39 Operations Establishing a Connection Active Active Client Client Passive Client Client ib_cm_req_t ib_cm_rtu_t ib_cm_listen_t ib_cm_rep_t AL AL AL AL ib_cm_req ib_cm_rtu ib_cm_listen ib_cm_rep CM CM CM CM 39 InfiniBand Fabric

40 Operations Registering for Event Notification Client registers for a event Events include CA / port / IOC add, remove, or error, and port LID or Pkey change Multiple registrations allow different callbacks AL invokes client callback with a device event record Event record includes client context, reason for callback, and CA / port / IOC GUIDs per reason 40

41 Operations Registering for Event Notification Client Client ib_notify_req_t ib_dev_rec_t AL AL ib_reg_notify PnP PnP Mgr Mgr Event Event 41

42 Operations Subscribing for Reports Client AL SA Class Mgr Class Agt ib_subscribe() status Svc Query(name) Svc Records Path Query(GIDs) Path Records Subscribe callback Report callback SIDR Req SIDR Rep Set(InformInfo) Get resp Report Report Resp trap Trap repress 42

43 Operation Plug and Play Manager AL PnP Manager is built upon the event notification and subscription mechanisms A PnP Device File is used to manually register for a callback PnP Manger will load the device driver if needed Device driver registers with AL AL PnP Manager invokes client callback with event notification or report 43

44 Operation Plug and Play Manager Client Client modprobe modprobe ib_al_open_t ib_dev_rec_t AL AL ib_open_al PnP Device File PnP PnP Mgr Mgr Event Event 44

45 Operations Creating an IO Controller Target channel driver registers an IOC with AL providing the IoControllerProfile AL assigns the IOC to an IO Unit slot AL sets IsDeviceManagementSupported capability bit on all ports of the given CA Target channel driver may add Service Entries to the registered IOC CFM is responsible for IOC management 45

46 Operations Creating an IO Controller Client Client ib_ioc_profile_t ib_svc_entry_t ib_ioc_handle_t AL AL ib_create_ioc ib_add_svc_entry ib_reg_ioc Io Controller Profile Service Entry IoUnitInfo CFM CFM 46

47 Agenda Objectives Feature Summary Design Overview Kernel-Level Interface Operations Current Current Status 47

48 Current Status IBA Specification Deviations When adding an IOC, the reserved field at bit offset 184 in the IoControllerProfile is set to the assigned IOC SlotID When querying Service Entries, check the IoControllerProfile reserved field. If non- zero assume this value is the IOC SlotID A workaround is possible by searching the IO Unit slots to locate the desired IOC 48

49 Current Status Design and Documentation Obtaining stakeholder requirements and incorporating into design Focus is on currently on kernel mode User mode will follow and be very similar Documentation Software Architecture Specification - Draft 2 Header file Draft 0.1 High Level Design Specification - Draft 0.1 (just started) 49

50 Current Status Opens / Risks InfiniBand Architecture Multicast groups are poorly defined Service Record format will likely change Most IB Annexes are not approved Possible additional requirements Schedule constraints 50

51 Backup 51

52 Operations UD Communication Consumer AL SA AL Provider ib_create_av() status ib_post_send() status Comp callback Datagram Comp callback ib_create_av() status Comp callback Datagram ib_post_send() status Comp callback 52