MOVING FORWARD WITH FABRIC INTERFACES

Transcription

1 14th ANNUAL WORKSHOP 2018 MOVING FORWARD WITH FABRIC INTERFACES Sean Hefty, OFIWG co-chair Intel Corporation April, 2018

2 USING THE PAST TO PREDICT THE FUTURE OFI Provider Infrastructure OFI API Exploration Companion APIs (Bonus!) 1.5 API Updates RxM provider SOCK endpoint types Memory registration API optimizations 2017 v v v v1.6.1 v1.6.2 v Provider Enhancements PSM2 native RxM performance SHM shared memory support Persistent memory 1.7 Predictions New providers RxD, multi-rail, new vendors SHM xpmem support API enhancements 2 OpenFabrics Alliance Workshop 2018

3 PROVIDER INFRASTRUCTURE 3 OpenFabrics Alliance Workshop 2018

4 ARUN AND DMITRY S AMAZING RXM PROVIDER High-priority OFI OFI RxM MPI / SHMEM MSG MSG MSG MSG Primary path for HPC apps accessing verbs hardware Verbs NetworkDirect TCP Optimizes for hardware features TCP to replace sockets MSG MSG MSG MSG Connection multiplexing Strong MPI performance Evaluating tighter provider coupling 4 OpenFabrics Alliance Workshop 2018

5 RXD Focus for v1.7 OFI OFI RxD MPI / SHMEM DGRAM DGRAM Offload large transfers Future path for HPC scalability Verbs UD usnic UDP Raw Ethernet Other..? Fast development path for hardware support Extend features of simple provider DGRAM DGRAM DGRAM Reliability, segmentation, and reassembly DGRAM Re-designing for performance and scalability Analyzing provider specific optimizations 5 OpenFabrics Alliance Workshop 2018

6 ALEXIA S FANTASTIC SHARED MEMORY PROVIDER Now available in stores near you! SMR SHM Provider Shared Memory Region SMR One-sided and two-sided transfers SMR CMA (cross-memory attach) for large transfers Version Flags PID Region Size Lock Shared memory primitives Command Queue Response Queue Inject Buffers Peer Address Map Single command queue xpmem support under development 6 OpenFabrics Alliance Workshop 2018

7 MEMORY MONITOR AND REGISTRATION CACHE Provider Driver notification, hook alloc/free, provider specific Internal API Get/put MRs Notification Notification Queue Queue subscribe events Memory Monitor Core Monitor Plug-in A generic solution is desired here Callbacks to add/delete MRs Registration Cache LRU List Custom Limits Usage Stats Merges overlapping regions MR Map MR MR MR Tracks active usage 7 OpenFabrics Alliance Workshop 2018

8 PERFORMANCE MONITORING Ex: Sample CPU instructions for various code paths Performance Data Set Event Data Event Data Count Event Data Count Sum Count Sum Sum Performance Management Unit CPU Cache Linux RDPMC Cycles Instructions Hits Misses Inline performance tracking NIC Performance domains? 8 OpenFabrics Alliance Workshop 2018

9 HOOKING PROVIDER Framework done, needs core integration OFI User OFI Core Always available release and debug builds Zero-impact unless enabled Hook Core/Util Provider Intercept calls to any provider Debugging, performance analysis, feature enhancements, testing 9 OpenFabrics Alliance Workshop 2018

10 API EXPLORATION 10 OpenFabrics Alliance Workshop 2018

11 VARIABLE LENGTH MESSAGES User User send Size unknown until sent receive size =? size = X X > transport msg size X Software layers duplicate feature Eager message rendezvous RMA read or tagged message MTU ack remaining transfer RMA write, tagged send, send RTS CLS transfer Similar wire protocols different implementations 11 OpenFabrics Alliance Workshop 2018

12 VARIABLE LENGTH MESSAGES No change at sender maybe send User Only lowest layer developer needs to figure out how to spell rendezvous! Report ready to receive completion User Claim/ Discard ID size = X ID + X Modeled after tagged message feature Opt-in impacts protocol Provider optimizes around hardware abilities Opportunity: report discard to sender Application flow control and load balancing Dynamically disable receive processing (e.g. EBUSY) 12 OpenFabrics Alliance Workshop 2018

13 MULTI-RAIL PROVIDER Focus for v1.7 Application or admin configured Multiple EPs, ports, NICs, fabrics One fi_info structure per rail OFI OFI mrail EP 1 EP User EP 1 EP 2 EP 2 Increase bandwidth and message rate Failover Rail selection plug-in EP 1 EP 2 Isolate rail selection algorithm Require variable message support TBD: recovery fallback Active Standby 13 OpenFabrics Alliance Workshop 2018

14 PERSISTENT MEMORY High-availability model (v1.6) Documentation limits use case RMA Write User PMEM MR New completion semantic User Register PMEM Persistent Memory Commit complete Work with SNIA (Storage Networking Industry Association) Evolve APIs to support other usage models Exploration Byte addressable or object aware Single or multi-transfer commit Advanced operations (e.g. atomics) Keep implementation agnostic Handle offload and on-load models Support multi-rail Minimize state footprint 14 OpenFabrics Alliance Workshop 2018

15 DATA DOMAINS APIs assume memory mapped regions Same coherency domain CPU load/stores CPU Memory PMEM May need to sync results with CPU Peer Device Device Memory (Smart) NIC Device FPGA Memory May not want to write data through CPU caches Memory regions may not be mapped Programmable offload capabilities and flow processing Results may be cached by NIC for long transactions 15 OpenFabrics Alliance Workshop 2018

16 COMPANION APIS 16 OpenFabrics Alliance Workshop 2018

17 C++ STANDARDIZATION Feedback from C++ community Implement proposal Detail alternatives Justify extensions Add support for fabrics directly to the C++ language Proposal Extend ASIO Implement over libfabric ASIO Model User Program Async Handler e.g. connect Async Handler e.g. transmits IO Object e.g. resolver IO Object e.g. socket Callback driven IO Service (tracks and progresses requests) Maps to all OFI asynchronous reporting objects

18 NOTIFICATION QUEUE Async Handler e.g. connect Extend to allow separation of control and data events Callback completion model Async Handler e.g. transmits IO Service (tracks and progresses requests) Notification Queue Event Handler Transmit Handler Receive Handler Error Handler Concurrency Wait Object Queue Size Signaling Vector Tx Format Rx Format dispatch() poll() post() run() stop() reset() Interfaces modeled after IO service Event Queue(s) Completion Queue(s) Wait Set Poll Set

19 RSOCKETS rsockets (librdmacm) A CM Verbs RC QP UD QP Increase OS & fabric portability Significantly boosts performance versus sockets with HW acceleration Pursuing OpenJDK integration OFI Maintain verbs protocol Verbs SOCK DGRAM EP SOCK STREAM EP Omni Path SOCK DGRAM EP SOCK STREAM EP rsockets (librsockets) TCP SOCK STREAM EP UDP SOCK DGRAM EP Network Direct SOCK STREAM EP Always available 19 OpenFabrics Alliance Workshop 2018

20 14th ANNUAL WORKSHOP 2018 THANK YOU Sean Hefty, President and CEO My Own Little World