PASTE: A Networking API for Non-Volatile Main Memory

Transcription

1 PASTE: A Networking API for Non-Volatile Main Memory Michio Honda (NEC Laboratories Europe) Lars Eggert (NetApp) Douglas Santry (NetApp) TSVAREA@IETF 99, Prague May 22th 2017 More details at our HotNets paper:

2 Motivation Non-Volatile Main Memories (NVMMs) Persistent Byte-addressable Low latency 10s-1000s of ns /persistent-memory.html A lot of work in databases and file systems NVHeap [ASPLOS'11], NOVA [FAST'16], NVWal [ASPLOS'16], HiKV [ATC'17] etc What are implications for networking?

3 Review: Today's Stack with NVMM

4 Review: Today's Stack with NVMM How data move from the network to the storage

5 Case Study: Careful Data Transfer Server persists client s request prior to acknowledgment e.g., 1KB commit:

6 Case Study: Careful Data Transfer Server persists client s request prior to acknowledgment e.g., 1KB commit: client Network stack NIC App DRAM 2030 us Networking (w/o step (4)) takes 40 us (5) (1) read() (3) (2) write()/fsync() or memcpy()/msync() (4) Storage stack SSD/ DIsk

7 Case Study: Careful Data Transfer Server persists client s request prior to acknowledgment e.g., 1KB commit: client (5) (1) us Networking takes 40 us This 2 us is not small read() (3) Network stack (2) DRAM NIC App write()/fsync() or memcpy()/msync() (4) Storage stack NVMM Emulated using a reserved region of DRAM

8 Case Study: Careful Data Transfer Parallel requests are serialized on each core 33 % throughput decrease, 50 % latency increase

9 Data Copies Matter Cache Misses Persisting data (e.g., to a log) always happens to a different destination read() memcpy() kernel buffer app buffer log file (mmap()-ed) Overall cache misses Major Contributor Networking only % net_rx_action() (84%) Networking + NVMM (read() + memcpy() + msync()) % memcpy() (98%) Reported by Linux perf We must avoid data copy!

10 Packet Store (PASTE) Overview Packet buffers on a named NVMM region DMA to NVMM Zero-copy APIs client (1) NVMM (3) Network stack App / mnt/nvmm/ pktbufs (2) NIC Storage stack metadata only (e.g., buffer index) (4) / mnt/pmem/ appmd

11 Fast Persistent Write with PASTE mmap() Application (1) Read data (zero copy) (2) Write metadata entry (3) Flush (buffer and metadata) User Kernel metadata header buf_ofs: 123 metadata entries buf_idx off len /mnt/nvmm/myapp_metadata (static) TCP/IP input and output NIC ring

12 Fast Persistent Write with PASTE Unread Read or written mmap() Application (1) Read data (zero copy) (2) Write metadata entry (3) Flush (buffer and metadata) User Kernel metadata header buf_ofs: 123 metadata entries buf_idx off len /mnt/nvmm/myapp_metadata (static) TCP/IP input and output NIC ring

13 Fast Persistent Write with PASTE Unread Read or written mmap() Application (1) Read data (zero copy) (2) Write metadata entry (3) Flush (buffer and metadata) User Kernel metadata header buf_ofs: 123 metadata entries buf_idx off len /mnt/nvmm/myapp_metadata (static) TCP/IP input and output NIC ring

14 Fast Persistent Write with PASTE Unread Read or written mmap() Application (1) Read data (zero copy) (2) Write metadata entry (3) Flush (buffer and metadata) User Kernel metadata header buf_ofs: 123 metadata entries buf_idx off len /mnt/nvmm/myapp_metadata (static) TCP/IP input and output NIC ring

15 Fast Persistent Write with PASTE Unread Read or written Flushed mmap() Application (1) Read data (zero copy) (2) Write metadata entry (3) Flush (buffer and metadata) User Kernel metadata header buf_ofs: 123 metadata entries buf_idx off len /mnt/nvmm/myapp_metadata (static) DMA is performed to L3 cache (DDIO) TCP/IP input and output NIC ring

16 Fast Persistent Write with PASTE Unread Read or written Flushed mmap() Application (1) Read data (zero copy) (2) Write metadata entry (3) Flush (buffer and metadata) User Kernel metadata header buf_ofs: 123 metadata entries buf_idx off len /mnt/nvmm/myapp_metadata (static) DMA is performed to L3 cache (DDIO) TCP/IP input and output NIC ring

17 Fast Persistent Write with PASTE Unread Read or written Flushed mmap() Application (1) Read data (zero copy) (2) Write metadata entry (3) Flush (buffer and metadata) User Kernel metadata header buf_ofs: 123 metadata entries buf_idx off len /mnt/nvmm/myapp_metadata (static) DMA is performed to L3 cache (DDIO) TCP/IP input and output NIC ring

18 Fast Persistent Write with PASTE Unread Read or written Flushed mmap() Application (1) Read data (zero copy) (2) Write metadata entry (3) Flush (buffer and metadata) User Kernel metadata header buf_ofs: 123 metadata entries buf_idx off len /mnt/nvmm/myapp_metadata (static) DMA is performed to L3 cache (DDIO) TCP/IP input and output NIC ring

19 Fast Persistent Write with PASTE Unread Read or written Flushed mmap() Application (1) Read data (zero copy) (2) Write metadata entry (3) Flush (buffer and metadata) User Kernel metadata header buf_ofs: 123 metadata entries buf_idx off len /mnt/nvmm/myapp_metadata (static) DMA is performed to L3 cache (DDIO) TCP/IP input and output NIC ring

20 Fast and Selective Persistent Write with PASTE Unread Read or written Flushed mmap() Application Idempotent (1) Read data (zero copy) request (2) Write metadata entry (3) Flush (buffer and metadata) User Kernel metadata header buf_ofs: 123 metadata entries buf_idx off len /mnt/nvmm/myapp_metadata (static) DMA is performed to L3 cache (DDIO) Unnecessary data is not flushed to DIMM TCP/IP input and output NIC ring

21 Fast and Selective Persistent Write with PASTE Unread Read or written Flushed mmap() Application (1) Read data (zero copy) (2) Write metadata entry (3) Flush (buffer and metadata) User Kernel metadata header buf_ofs: 123 metadata entries buf_idx off len /mnt/nvmm/myapp_metadata (static) DMA is performed to L3 cache (DDIO) Unnecessary data is not flushed to DIMM TCP/IP input and output NIC ring

22 Fast and Selective Persistent Write with PASTE Unread Read or written Flushed mmap() Application (1) Read data (zero copy) (2) Write metadata entry (3) Flush (buffer and metadata) User Kernel metadata header buf_ofs: 123 metadata entries buf_idx off len /mnt/nvmm/myapp_metadata (static) DMA is performed to L3 cache (DDIO) Unnecessary data is not flushed to DIMM TCP/IP input and output NIC ring

23 Fast and Selective Persistent Write with PASTE Unread Read or written Flushed mmap() Application (1) Read data (zero copy) (2) Write metadata entry (3) Flush (buffer and metadata) User Kernel metadata header buf_ofs: 123 metadata entries buf_idx off len /mnt/nvmm/myapp_metadata (static) DMA is performed to L3 cache (DDIO) Unnecessary data is not flushed to DIMM TCP/IP input and output NIC ring

24 Preliminary Results Implementation Extend the netmap framework TCP/IP is also supported % throughput increase, 9-46 % latency reduction

25 Related Work Enhanced network stacks MegaPipe (OSDI 12), Stackmap (ATC 16), Fastsocket (ASPLOS 16) IX and Arrakis (OSDI 14), mtcp (NSDI 13), Sandstorm (SIGCOMM 14), MICA (NSDI 14) No NVMM aware NVMM filesystems BPFS (SOSP 09), NOVA (FAST 15) NVMM databases NVWAL (ASPLOS 15), REWIND (VLDB 15), NV-Tree (FAST 15) No networking aware

26 Conclusion Networking APIs are now a bottleneck for durably storing data Ongoing work Brushing up APIs Smooth integration with Forming a filesystem with Fast data movement between files Use case applications Network Stack NIC App Netmap API NVMM Filesystem