Writing RDMA applications on Linux

Transcription

1 Writing RDMA applications on Linux Roland Dreier 1

2 RDMA? 2

3 RDMA: Remote DMA 3

4 RDMA: Remote Direct Memory Access 4

5 RDMA: Remote Direct Memory Access one sided operations 5

6 RDMA: Remote Direct Memory Access one sided operations get/put semantics 6

7 RDMA: Remote Direct Memory Access one sided operations get/put semantics direct data placement 7

8 RDMA: Remote Direct Memory Access...but wait, there's more... 8

9 RDMA: Remote Direct Memory Access Asynch work/completion queues 9

10 RDMA: Remote Direct Memory Access Asynch work/completion queues Kernel bypass 10

11 RDMA: InfiniBand 11

12 RDMA: InfiniBand iwarp 12

13 RDMA Verbs Subtitle 13

14 Verbs? 14

15 Verbs: not quite an API 15

16 Verbs: not quite an API; abstract definition of functionality 16

17 Verbs: resources (objects) operated on by verbs (functions) 17

18 Verbs: create object 18

19 Verbs: create object destroy object 19

20 Verbs: create object destroy object more interesting things... 20

21 Objects: 21

22 Objects: device context 22

23 Objects: queue pair (QP) 23

24 Objects: queue pair (QP) send queue & receive queue 24

25 Objects: queue pair (QP) post send post receive modify state 25

26 Objects: completion queue (CQ) 26

27 Objects: completion queue (CQ) work request completions reported as CQ entries 27

28 Objects: completion queue (CQ) poll CQ request notification 28

29 Objects: completion channel 29

30 Objects: memory region (MR) 30

31 Objects: protection domain (PD) 31

32 Objects: shared receive queue (SRQ) 32

33 Objects: shared receive queue (SRQ) multiple QPs can share a receive queue cleaning up is a little tricky 33

34 Objects: shared receive queue (SRQ) post receive 34

35 Objects: address handle (AH) memory window (MW) 35

36 Work processing: requests from WQs get executed completions are reported to CQs mostly things stay in order 36

37 Linux & RDMA 37

38 librdmacm 38

39 librdmacm Linux library to abstract connection setup 39

40 librdmacm Linux library to abstract connection setup same code runs on IB and iwarp 40

41 librdmacm mimics TCP socket model 41

42 librdmacm mimics TCP socket model cm_id is socket analog 42

43 librdmacm mimics TCP socket model cm_id is socket analog IP addressing used even on InfiniBand 43

44 librdmacm mimics TCP socket model cm_id is socket analog IP addressing used even on InfiniBand additional address/route resolution steps 44

45 librdmacm events reported through channels 45

46 librdmacm events reported through channels rdma_create_event_channel() rdma_get_cm_event() rdma_ack_cm_event() 46

47 librdmacm active connection steps 47

48 librdmacm active connection steps rdma_resolve_addr() rdma_resolve_route() rdma_connect() 48

49 librdmacm passive connection steps 49

50 librdmacm passive connection steps rdma_bind_addr() rdma_listen() rdma_accept() 50

51 libibverbs 51

52 libibverbs Linux implementation of RDMA verbs 52

53 libibverbs Loads device specific drivers for hardware support 53

54 libibverbs Loads device specific drivers for hardware support IB: libmthca, libmlx4, libipathverbs, libehca iwarp: libcxgb3, libamso 54

55 libibverbs Loads device specific drivers for hardware support IB: libmthca, libmlx4, libipathverbs, libehca iwarp: libcxgb3, libamso 55

56 libibverbs Creating QP can be confusing rdma_create_qp() vs. ibv_create_qp() all those parameters! 56

57 libibverbs Posting work requests is tricky too send opcodes iwarp doesn't have immed data or atomics signaled and unsignaled completions 57

58 Q and A 58

59 59