SAY-Go: Towards Transparent and Seamless Storage-As-You-Go with Persistent Memory

Transcription

1 SAY-Go: Towards Transparent and Seamless Storage-As-You-Go with Persistent Memory Hyeonho Song, Sam H. Noh UNIST HotStorage 2018

2 Contents Persistent Memory Motivation SAY-Go Design Implementation Evaluation Summary Future work 1

3 Persistent Persistent Memory Features Non-volatility Byte-level random access Fast access time (nanoseconds) PRAM Storage Memory 3D XPoint STT-MRAM 2

4 Persistent 1H H 2017 Coming soon 2H 2018 Intel Optane SSD DC P4800X Intel Optane Memory DIMM Based Optane Block device Block device Character device 3

5 Persistent 1H H 2017 Coming soon 2H 2018 Our target Intel Optane SSD DC P4800X Intel Optane Memory DIMM Based Optane Block device Block device Character device 4

6 Persistent 1H H 2017 Coming soon 2H 2018 Intel Optane SSD DC P4800X Intel Optane Memory DIMM Based Optane CPU CPU CPU DRAM DRAM DRAM PM (cache) PM traditional HDD, SSD PM traditional HDD, SSD 5

7 Motivation Applications demand more and more Virtual manager VFS SFS manager Device driver DRAM storage 6

8 Motivation DRAM capacity never enough! Virtual manager manager VFS SFS Device driver storage is relatively roomy due to high density fully used free used 7

9 Traditional solution Swap DRAM mechanism capacity is with not enough! storage Virtual manager manager VFS SFS Device driver fully used free used 8

10 Swap mechanism When DRAM runs out User application allocation is not possible Main fully used swap area Storage free blocks DRAM traditional HDD 9

11 Swap mechanism: swap-out Secure free space in DRAM User application data save (copy) to swap area Main fully used swap area Storage free blocks DRAM traditional HDD 10

12 Swap mechanism: swap-out Secure free space in DRAM User application allocation is possible Main not full swap area Storage free blocks DRAM traditional HDD 11

13 Swap mechanism: swap-in When application wants to access data in swap area User application Main not full swap area Storage free blocks DRAM traditional HDD 12

14 Swap mechanism: swap-in When application wants to access data in swap area User application data restore(copy) to free space in DRAM Main not full swap area Storage free blocks DRAM traditional HDD block device is not directly accessible by the CPU 13

15 Swap mechanism: swap-in When application wants to access data in swap area User application access to data Main not full swap area Storage free blocks DRAM traditional HDD 14

16 Weakness of traditional swap mechanism Large overhead with slow storage Data copy is essential User application swap-in Selecting s to be swapped out is a complex process Main not full swap area Storage free blocks Swap area cannot be used as storage space DRAM swap-out traditional HDD Static partition of file system 15

17 Simple solution: swap with PM storage PM performance is better than traditional storage Reduced data copy time Overhead of the swap mechanism is also reduced Virtual manager VFS SFS Does this suffice? manager Device driver fully used DIMM Based PM 16

18 There should be a better way? PM storage can be accessed directly from the CPU Virtual manager VFS SFS PM as both main and storage Dynamically use PM Increase capacity of the working when needed manager Device driver fully used DIMM Based PM 17

19 Our Goal Dynamic adjustment of the and storage boundary Virtual manager User VFS SFS Virtual manager User VFS SFS SAY-Go system (Storage-As-You-Go) dynamic usage of PM manager Device driver manager Device driver DRAM traditional HDD, SSD DRAM PM 18

20 Our Goal Dynamic adjustment of the and storage boundary Virtual manager User VFS SFS Virtual manager User VFS SFS SAY-Go system (Storage-As-You-Go) dynamic usage of PM manager Device driver manager Device driver similar attempts were proposed DRAM traditional HDD, SSD DRAM PM 19

21 Related work Dynamic usage of PM Virtual manager User VFS SFS Memorage 1) ICS 2013 : Working expansion pvm 2) EuroSys 2016 : Direct allocation of PM to working manager DRAM Device driver PM 1) Jung, J. Y. and Cho, S. Memorage: Emerging persistent ram based malleable main and storage architecture. In Proceedings of the International ACM Conference on International Conference on Supercomputing (ICS) (2013). 2) Kannan, S., Ada, G., and Karsten, S. pvm: persistent virtual for efficient capacity scaling and object storage. In Proceedings of the European Conference on Computer Systems (EuroSys) (2016). 20

22 Related work Dynamic usage of PM Virtual manager User VFS SFS Memorage 1) ICS 2013 : Working expansion pvm 2) EuroSys 2016 : Direct allocation of PM to working manager DRAM Device driver PM 1) Jung, J. Y. and Cho, S. Memorage: Emerging persistent ram based malleable main and storage architecture. In Proceedings of the International ACM Conference on International Conference on Supercomputing (ICS) (2013). 2) Kannan, S., Ada, G., and Karsten, S. pvm: persistent virtual for efficient capacity scaling and object storage. In Proceedings of the European Conference on Computer Systems (EuroSys) (2016). 21

23 Memorage When DRAM runs out User application allocation is not possible Main fully used Storage free blocks DRAM PM 22

24 Memorage Working expansion User application 1. select free blocks to be used in file system Main fully used Storage free blocks DRAM PM 23

25 Memorage Working expansion User application Main fully used DRAM Storage free blocks PM 1. select free blocks to be used in file system 2. blocks are transformed into allocatable free s by reorganizing them as data structures used in the OS management layer 24

26 Memorage Working expansion User application 3. set that region to additional using hot-plug feature of OS 1. select free blocks to be used in file system Main fully used expand Storage free blocks 2. blocks are transformed into allocatable free s by reorganizing them as data structures used in the OS management layer DRAM PM 25

27 Memorage Working expansion User application Main fully used expand allocation becomes possible Storage free blocks DRAM PM 26

28 Weakness of Memorage Large runtime overhead Process of changing free blocks into s proceeds at runtime Memory hot-plug feature is required blocks -> s Overlooks the issue of consistency Consistency of allocator is not considered Memory leak can occur with faults 27

29 Related work Dynamic usage of PM Virtual manager User VFS SFS Memorage 1) ICS 2013 : Working expansion pvm 2) EuroSys 2016 : Direct allocation of PM to working manager DRAM Device driver PM 1) Jung, J. Y. and Cho, S. Memorage: Emerging persistent ram based malleable main and storage architecture. In Proceedings of the International ACM Conference on International Conference on Supercomputing (ICS) (2013). 2) Kannan, S., Ada, G., and Karsten, S. pvm: persistent virtual for efficient capacity scaling and object storage. In Proceedings of the European Conference on Computer Systems (EuroSys) (2016). 28

30 pvm When DRAM runs out User application allocation is not possible Main fully used Persistent store free PM s DRAM PM 29

31 pvm When DRAM runs out User application Main fully used Persistent store free PM s direct allocation - npmalloc() - nvmmap(persist or NOPERSIST) DRAM PM 30

32 Weakness of pvm Requires application modification Explicit library function calls need to be made npmalloc(), nvmmap() direct allocation Similar approach to previous work NV-Heap Mnemosyne NVM Duet 31

33 Comparison of related works strength weakness Goal Memory-Storage division Swap Memorage pvm free up working space expansion persistent store fixed fixed fixed Consistency not considered not considered yes Transparency yes yes no Runtime overhead yes yes no Direct access to data no yes yes 32

34 SAY-Go design goals Virtual manager manager for working DRAM User VFS SFS Device driver for storage PM Seamless integration - PM role is dynamically adjusted as need be Transparency - Automatic scaling of working - Without modification of the application Consistency - Ensure consistency of the PM allocator - Prevent leaks Dynamic partition 33

35 SAY-Go components Consists of two parts Virtual manager File system with dynamic partitioning 1. File system with dynamic partition support - Support dynamic layout - Region managed in unit Integrated manager for working DRAM for storage PM 2. Integrated manager - Page allocator for both working and storage - Ensure consistency of allocator 34

36 Layout Consists of two parts Virtual manager File system with dynamic partitioning 1. File system with dynamic partition support - Support dynamic layout - Region managed in unit Integrated manager for working for storage Dynamic Partition File System (DPFS), in progress DRAM PM 2. Integrated manager - Page allocator for both working and storage - Ensure consistency of allocator Persistent Memory Buddy (PMB), HotStorage 18 35

37 Layout Consists of two parts Virtual manager File system with dynamic partitioning 1. File system with dynamic partition support - Support dynamic layout - Region managed in unit Integrated manager for working for storage Dynamic Partition File System (DPFS), in progress DRAM PM 2. Integrated manager - Page allocator for both working and storage - Ensure consistency of allocator Persistent Memory Buddy (PMB), HotStorage 18 Implemented in Linux 36

38 PMB implementation features Virtual manager for working PMB File system with dynamic partitioning for storage Persistent Memory Buddy (PMB) Seamless integration - PM role is dynamically adjusted as need be Transparency - Automatic scaling of working - Without modification of the application DRAM PM Consistency - Ensure consistency of the PM allocator - Prevent leaks PMB designed with SAY-Go goal in mind 37

39 PMB Layout in Linux Processes VM manager for working NORMAL PMB MIGRATE VFS DPFS Device driver for storage STRG DRAM and PM are managed in units - _NORMAL - plus 2 new s * _MIGRATE and _STRG Each has free s - buddy allocator DRAM s PM s 38

40 Buddy allocator Characteristics in Linux Unsorted Circular linked list Push and pop operation occurs only at the end of the list Push and pop operations are implemented by atomic operations Free_list order(10). order(2) order(1) order(0) 4MB. 4KB 4MB. 4KB 4MB. PMB based on buddy allocator 39

41 PMB Layout in Linux Processes VM manager for working NORMAL PMB MIGRATE VFS DPFS Device driver for storage STRG DRAM and PM are managed in units - _NORMAL - plus 2 new s * _MIGRATE and _STRG Each has free s - buddy allocator DRAM s PM s Each has different role 40

42 Role of each Processes VM manager for working NORMAL DRAM s PMB MIGRATE PM s VFS DPFS Device driver for storage STRG _NORMAL - DRAM is covered by this _STRG - Minimum storage area - Persistent metadata saved to this area - Persistent descriptor - Bitmap that maintains the allocation state - Log table for PMB consistency 41

43 Role of each Processes VM manager for working NORMAL PMB MIGRATE VFS DPFS Device driver for storage STRG _MIGRATE - Key component in PMB - Either allocated depending on its use * as persistent (storage) * as non-persistent () DRAM s PM s 42

44 Page allocation Processes VM manager for working NORMAL DRAM s PMB MIGRATE PM s VFS DPFS Device driver for storage STRG _NORMAL - Allocate working Storage allocated from _STRG - Allocate storage _MIGRATE - Allocate PM s to both zones - Allocation done using the existing interface - Allocation done by kernel thread in the background 43

45 capacity Processes VM manager for working NORMAL DRAM s PMB MIGRATE PM s VFS DPFS Device driver for storage STRG _NORMAL - Initially, only DRAM s - Will grow with demand _STRG - Initially, essential storage area - Will grow with demand 44

46 Page movement NORMAL expansion Processes VM manager for working NORMAL DRAM s PMB MIGRATE PM s VFS DPFS Device driver for storage STRG _NORMAL - Initially, only DRAM s - Will grow with demand - PM migration PM s 45

47 Page movement NORMAL expansion Processes VM manager for working NORMAL DRAM s PMB MIGRATE PM s VFS DPFS Device driver for storage STRG _NORMAL - Initially, only DRAM s - Will grow with demand - PM migration - Increased _NORMAL size 46

48 Page movement NORMAL retraction Processes VM manager for working NORMAL DRAM s PMB MIGRATE PM s VFS DPFS Device driver for storage STRG _NORMAL - Initially, only DRAM s - Will grow with demand - PM retrieval 47

49 Page movement NORMAL retraction Processes VM manager for working NORMAL PMB MIGRATE VFS DPFS Device driver for storage STRG _NORMAL - Initially, only DRAM s - Will grow with demand - PM retrieval - Reduced _NORMAL size DRAM s PM s 48

50 Page migration _MIGRATE to _NORMAL Both zones managed with Buddy allocator When number of free s falls below watermark Transfers 4MB contiguous s simple pointer manipulation PMB for working for storage NORMAL MIGRATE STRG DRAM s PM s NORMAL MIGRATE STRG Free_list order(10). order(2) order(1) order(0) Free_list order(10). order(2) order(1) order(0) 4MB 2GB. 2. link pointer 4MB 1. un-link pointer 49

51 Page retrieval _NORMAL to _MIGRATE No watermark Retrieved released by application simple pointer manipulation for working NORMAL DRAM s PMB MIGRATE PM s for storage STRG NORMAL MIGRATE STRG Free_list order(10). order(2) order(1) order(0) Free_list order(10). order(2) order(1) order(0). 2GB. 1. un-link pointer. 4MB 4MB.. 2. link pointer 50

52 Management unit NORMAL MIGRATE STRG Free_list order(10). order(2) order(1) order(0) Free_list order(10). order(2) order(1) order(0) 4MB PMB takes the 2MB huge size 4KB 2GB 4MB 2MB Contiguous space (2 0 ~ 2 10 s) - _NORMAL: 4KB ~ 4MB - _MIGRATE, _STRG: 2MB ~ 2GB Page movement unit is 4MB contiguous - Performed between _NORMAL (order 10) and _MIGRATE (order 1) 51

53 Consistency Memory leak may occur in moving process NORMAL MIGRATE STRG Free_list order(10). order(2) order(1) order(0) Free_list order(10). order(2) order(1) order(0) 4MB 2GB. 2. link pointer 4MB 1. un-link pointer upon crash 52

54 Consistency Memory leak may occur in moving process NORMAL MIGRATE STRG Free_list order(10). order(2) order(1) order(0) Free_list order(10). order(2) order(1) order(0) 4MB 2GB. 4MB 1. un-link pointer these s are leaked 53

55 Consistency Use logging to prevent leak NORMAL MIGRATE STRG Free_list order(10). order(2) order(1) order(0) Free_list order(10). order(2) order(1) order(0) 4MB 2GB. 3. link pointer 4. commit log 4MB 1. Write log about movement or allocation 2. un-link pointer 54

56 Consistency Use logging to prevent leak NORMAL MIGRATE STRG Free_list order(10). order(2) order(1) order(0) Free_list order(10). order(2) order(1) order(0) 4MB 2GB. 3. link pointer 4. commit log 4MB Log entry fields - Page structure address - Operation type - Destination of - Commit bit Log entry size < 64bit Use mfence and clflush instructions for logging 1. Write log about movement or allocation 2. un-link pointer 55

57 Previous work and SAY-Go strength weakness Goal Memory-Storage division Swap Memorage pvm SAY-Go free up working space expansion persistent store efficient use of resources fixed fixed fixed dynamic Consistency not considered not considered yes yes Transparency yes yes no yes Runtime overhead yes yes no no Direct access to data no yes yes yes 56

58 Evaluation Test platform Machine specification Intel Xeon E5-2620v3 2.4GHz (24 cores) 256 DRAM Divide to DRAM and Pseudo-PM Linux v Workloads characteristics FFT Redis Scale Memory intensive application In- database Domain Signal processing Key-value store Benchmark suite Splash2x in Parsec 3.0 YCSB Input Native (largest) 1:1 (read:write) Max usage 12GB 20GB 57

59 Evaluation Comparison with swap Memory configuration _NORMAL capacity varied based on workload FFT: 16GB for DRAM Redis: 32GB for DRAM PMB to be used dynamically NORMAL MIGRATE STRG 16 or 32GB remainder 16GB Swap NORMAL swap area EXT4 with DAX 16 or 32GB remainder (set by /dev/pmem) DRAM Pseudo-PM DRAM Pseudo-PM 58

60 Evaluation Comparison with swap Runtime (sec) PMB Swap FFT Redis DRAM (GB) used by application (_NORMAL size: 16GB) DRAM (GB) used by application (_NORMAL size: 32GB) 59

61 Evaluation Measurement method: FFT example Runtime (sec) PMB Swap FFT DRAM (GB) used by application (_NORMAL size: 16GB) FFT maximum working set size: 12GB Limit amount of main used by FFT - Use stress tool to consume main x-axis refers to the maximum DRAM used by FFT stress 10GB NORMAL 16GB DRAM FFT 6GB 6GB MIGRATE 224GB Pseudo-PM STRG 16GB 60

62 Evaluation Measurement method: FFT example Runtime (sec) PMB Swap FFT DRAM (GB) used by application (_NORMAL size: 16GB) FFT maximum working set size: 12GB Limit amount of main used by FFT - Use stress tool to consume main x-axis refers to the maximum DRAM used by FFT stress 13GB NORMAL 16GB DRAM FFT 3 9GB MIGRATE 224GB Pseudo-PM STRG 16GB 61

63 Evaluation Comparison with swap Runtime (sec) PMB Swap FFT Redis DRAM (GB) used by application (_NORMAL size: 16GB) DRAM (GB) used by application (_NORMAL size: 32GB) 62

64 Evaluation Comparison with swap PMB Swap FFT Redis Swap performance degrades considerably Runtime (sec) PMB performance remains stable Two reasons behind results DRAM (GB) used by application (_NORMAL size: 16GB) DRAM (GB) used by application (_NORMAL size: 32GB) 63

65 Evaluation Comparison with swap Runtime (sec) PMB Swap FFT Redis Mechanism overhead - PMB migration and retrieval : ~ 400 ns - Swap-in / out : ~ 6 / 41 us DRAM (GB) used by application (_NORMAL size: 16GB) DRAM (GB) used by application (_NORMAL size: 32GB) 64

66 Evaluation Comparison with swap Runtime (sec) PMB Swap FFT Redis PMB unaffected by DRAM size - swap copies increase with reduced DRAM size DRAM (GB) used by application (_NORMAL size: 16GB) DRAM (GB) used by application (_NORMAL size: 32GB) 65

67 Evaluation Number of copied s Number of swapped s (unit: million) Swap-out Swap-in 8.3M FFT 21.8M 60.5M 63.8M Redis 7.8M 26.4M Reported by /proc/vmstat 0 0.0M DRAM (GB) used by application (_NORMAL size: 16GB) 0 0.0M 0.2M 0.4M DRAM (GB) used by application (_NORMAL size: 32GB) 66

68 Evaluation PM migration and retrieval behavior Measured per 3s interval # of s FFT Redis migration retrieval s in NORMAL moved from MIGRATE execution time (s) execution time (s) 67

69 Evaluation PM migration and retrieval behavior Measured per 3s interval # of s FFT Redis migration retrieval s in NORMAL moved from MIGRATE execution time (s) execution time (s) 68

70 Evaluation PM migration and retrieval behavior FFT and Redis executed simultaneously # of s migration retrieval s in NORMAL moved from MIGRATE Redis runs once FFT running time: 60 seconds - Restarted upon termination after a 20 second lapse execution time (s) 69

71 Evaluation PM migration and retrieval behavior FFT and Redis executed simultaneously s 20s 60s 20s 60s 20s 60s 20s Redis runs once # of s FFT running time: 60 seconds - Restarted upon termination after a 20 second lapse - Pages are being freed and retrieved in the vicinity of 60, 140, 220, 300 seconds execution time (s) 70

72 Summary Proposed a system called Storage-As-You-Go (SAY-Go) Transparently adjusts the use of PM PM can be used as as well as storage as need be Break the boundary between and storage Persistent Memory Buddy (PMB) Memory allocation service that can freely grow and shrink working Experimental results show PMB performs better than traditional swap 71

73 Future work DPFS Requires dynamic partition resizing support In the design and early implementation phase Virtual manager DPFS PMB More detailed policies for managing DRAM PMB PM Various optimizations 72

74 Q & A 73