IME Infinite Memory Engine Technical Overview

Transcription

1 1 1 IME Infinite Memory Engine Technical Overview

2 2 Bandwidth, IOPs single NVMe drive

3 3 What does Flash mean for Storage? It's a new fundamental device for storing bits. We must treat it different from HDD We have to manage data placement across tiers at a larger scale + New opportunities for Novel Developments around scaling, data security, performance protection

4 4 DDN IME Application I/O Workflow Compute Diverse, high concurrency applications Fast Data NVM & SSD Persistent Data (Disk) Lightweight IME client intercepts application I/O. Places fragments into buffers + parity IME client sends fragments to IME servers IME servers write buffers to NVM and manage internal metadata IME servers write aligned sequential I/O to SFA backend Parallel File system operates at maximum efficiency

5 5 Distributed Hash Table + Log Structure Filesystem Data Key Distributed Network File1 File4 File3 File6 Hash Function DFCD ED789E 46042D43 DC355CE peers DHT provides foundation for Network parallelism Node-level fault tolerance Distributed metadata Log Tail Space reclaimed here data empty space data data data data Log Head data New data added here Log (time) wrap Log Structured Filesystem used at the storage device level High performance device throughput (NAND Flash Maintaining device longevity

6 6 DDN IME DataFlow in the Client COMPUTE NODE APPLICATION POSIX or MPIIO I/O is issued by the application IME places fragments into buffers 2 IME LUSTRE 1 data buffers accumulator parity buffers are simultaneously built parity 3 file open, file close, stat 4 metadata requests are passed through to the PFS client 5 Once full, buffers sent to IME server layer

7 7 DDN IME Erasure Coding application COMPUTE Data protection against IME server or SSD Failure is optional IME data buffers parity buffers (the lost data is "just cache ) LUSTRE Erasure Coding calculated at the Client Great scaling with extremely high client count Servers don't get clogged up IME Erasure coding does reduce useable Client bandwidth and useable IME capacity: 3+1: 56Gb 42Gb 5+1: 56Gb 47Gb 7+1: 56Gb 49Gb 8+1: 56Gb 50Gb PFS

8 8 DDN IME Data Residency Control COMPUTE maximum percentage of dirty data resident in IME before the data is automatically synchronized to the PFS: flush_threshold_ratio [0%.. 100%] Once Synchronised, the data is marked clean The clean data is kept in IME until the min_free_space_ratio is reached. min_free_space_ratio [0%.. 100%] I M E DIRTY CLEAN purge clean data until min_free_space_ratio sync dirty data until flush_threshold_ratio PFS

9 9 Use of Log Structuring in IME Consider two different application I/O patterns (write) Sequential Non-sequential Burst buffer blocks (BBB) are really just buffers generated at the client Note the contents of the BBB can be aligned or not. The same storage method is used for both blocks (despite the qualitative difference of their contents)!

10 10 Use of Log Structuring in IME What does this give us? Near line rate performance regardless of output pattern. 3 IOR Checkpoints to (4k strided, shared-file)

11 11 Non-Deterministic Data Placement Deterministic Approach: IME Clients use hash function to place data on rank0 host Non-deterministic approach: IME clients learn and observe the load of IME servers and route write requests to avoid highlyloaded servers COMPUTE Client Pending Request Queue Lengths IME

12 12 Aggregate IME Adaptive vs. Non-Adaptive WRITE Performance Ideal, healthy system One degraded IME server, Adaptive Amdahl s Law in action! One degraded IME server, Non-adaptive

13 13 IME v1.0 Mount Points FUSE client provides IME POSIX mount point df -h /ime/gsfs Filesystem Size Used Avail Use% Mounted on imefs 26T 3.9T 22T 16% /ime/gsfs Filesystem Mount Point # df -h /dev/gsfs/ Filesystem Size Used Avail Use% Mounted on /dev/gsfs 26T 3.9T 22T 16% /gsfs

14 14

15 15

16 16

17 17

18 18

19 19 Parallel File System: Shared File Performance Filesystem locking

20 20 IME vs Parallel File System: Shared File Performance

21 21 Rack Performance: IME IOR File-per-Process (GB/s) 4k Random IOPS ,000, ,000,000 1,000, , , , Write Read 1 Write Read ~550GB/s Read, Write ~50 Million IOPs

22 22 Benchmark Data POSIX Single Shared File IOR with Segments GB/sec 600 IME 20 Nodes (GB/sec) IOPS 100,000,000 IME 20 Nodes (IOPS) ,000,000 1,000, , , , k 8k 16k 32k 64k 128k 256k 512k 1024k 1 4k 8k 16k 32k 64k 128k 256k 512k 1024k Write(MB/sec) Read(MB/sec) Write IOPS Read IOPS

23 23 IME - Burst Buffer Productizing many years of research and development Yields a huge percentage of peak BW No server side erasure code overhead Fewer memory copies / visits Flexible data placement Allows server to avoid slow or oversubscribed components Log structured writes Utilize NVM devices in the most performant manner Completely declustered RAID Rebuilds of devices in MINUTES not days

24 24 Thank You! Keep in touch with us 9351 Deering Avenue Chatsworth, CA company/datadirect-networks