Boosting Quasi-Asynchronous I/Os (QASIOs)

Transcription

1 Boosting Quasi-hronous s (QASIOs) Joint work with Daeho Jeong and Youngjae Lee Jin-Soo Kim (jinsookim@skku.edu) Computer Systems Laboratory Sungkyunkwan University

2 The Problem 2

3 Why? CPU? Bus? Power management? Metadata contention? s already dispatched to the storage? No room in memory or request queue? The foreground task is waiting for the completion of asynchronous s queued in the scheduler! Over 1 second on an MLC emmc Over 4 seconds on a TLC emmc (to complete a single file system call) 3

4 Priority Inversion A situation where a higher-priority job is unable to run because a lower-priority job is holding a resource it needs, such as a lock What really happened on Mars? lock_acquire() Bus management task priority inversion communications task meteorological data gathering task lock_acquire() lock_release() 4

5 A Closer Look at Synchronous read() write() followed by fsync() or sync() write() opened with O_SYNC Journal writes by jbd2 hronous write() Metadata writes (to their original locations) 5

6 CFQ Scheduler A Sync queue for each process An queue is shared among tasks with the same priority Sync queues have larger time slices than queues Sets a limit for requests that can be dispatched in a single time slice A new Sync request preempts other requests Time to complete an can be prolonged indefinitely s are optimized for throughput not for latency 6

7 Linux Kernel Path VFS Layer 1. write() Ext4 Filesystem 4. perform Ext4 delayed block allocation Page Cache File Data 2. make dirty > 2 secs or 5% Block Layer 3. flush out kworker Kthread 5. submit (file data, ASYNC) File Metadata 7. flush out kworker Kthread make dirty submit (metadata, ASYNC) Running Transaction JBD2 Journaling Committing Transaction 6. commit jbd2 Kthread submit (metadata, SYNC) CFQ Scheduler. Queues ioprio 3 ioprio 4 jbd2 Process A Sync. Queues Process B 8. dispatch request 7

8 Quasi-hronous (QASIO) An which is issued asynchronously, but has the synchronous property because one or more tasks are waiting for its completion An asynchronous is promoted to a QASIO at run time when a task is blocked on it Causes unexpected delay Task A Processing Order Wait on Write Back!! scheduler QASIO 8

9 Quasi-? 9

10 Dependencies on QASIO Process B Indirect dependency Process A Direct dependency scheduler QASIO 10

11 Types of Dependencies on QASIO Direct Dependencies D meta D data D sync D discard When modifying a metadata When modifying a data When guaranteeing data to be written back When completing discard commands Indirect Dependencies I jhandle I jcommit When unable to obtain a journal handle When unable to complete fsync() 11

12 Case Study: Dmeta rename(), write(), unlink(), chmod(), chown(), fsync() Apps 1. async. bulky write Page Cache 2. flush out kworker Scheduler 12

13 Case Study: Dmeta Apps 1. async. bulky write inode entries, directory entries, block bitmaps, inode bitmaps, group descriptors, super block Page Cache metadata buffer JBD2 3. COMMIT Journaling & make dirty 2. flush out kworker Scheduler 13

14 Case Study: Dmeta Apps 1. async. bulky write Page Cache 2. flush out kworker metadata buffer JBD2 3. COMMIT Journaling & make dirty 4. flush out & lock buffer Scheduler 14

15 Case Study: Dmeta rename(), write(), unlink(), chmod(), chown(), fsync() Apps 1. async. bulky write Page Cache 2. flush out kworker metadata buffer JBD2 3. COMMIT Journaling & make dirty 4. flush out & lock buffer Scheduler 15

16 Case Study: Dmeta rename(), write(), unlink(), chmod(), chown(), fsync() Page Cache Apps 1. async. bulky write 2. flush out kworker metadata buffer 5. invoke file operations (lock_buffer()) 4. flush out & lock buffer D meta JBD2 3. COMMIT Journaling & make dirty Scheduler QASIO 16

17 Case Study: Dsync Normal Case UI Task write() Page Cache Scheduler 17

18 Case Study: Dsync Normal Case UI Task Page Cache fsync() (flush & wait) Scheduler Sync. Req. 18

19 Case Study: Dsync Apps UI Task 1. async. bulky write Page Cache 2. flush out kworker Scheduler 19

20 Case Study: Dsync 1. async. bulky write Page Cache Apps 3. async. write UI Task write() 2. flush out kworker Scheduler 20

21 Case Study: Dsync 1. async. bulky write Page Cache Apps 3. async. write UI Task write() 2. flush out kworker 4. flush out Scheduler 21

22 Case Study: Dsync 1. async. bulky write Apps 3. async. write UI Task write() fsync() D sync 5. invoke fsync() (wait_on writeback()) Page Cache 2. flush out kworker 4. flush out Scheduler QASIO 22

23 Case Study: Ijhandle Apps Task A write() Page Cache 1. 1K partial write JBD2 Journaling Running Transaction handle kworker Scheduler 23

24 Case Study: Ijhandle Apps Task A write() Page Cache 1. 1K partial write JBD2 Journaling Running Transaction handle 2. flush out kworker Scheduler 24

25 Case Study: Ijhandle Page Cache Apps 1. 1K partial write Task A write() 3. 1K partial write JBD2 Journaling Running Transaction handle 2. flush out kworker Scheduler 25

26 Case Study: Ijhandle Page Cache Apps 1. 1K partial write Task A write() 3. 1K partial write JBD2 Journaling Running Transaction handle 2. flush out kworker 4. wait_on writeback() Scheduler QASIO D data 26

27 Case Study: Ijhandle Page Cache Apps 1. 1K partial write Task A 3. 1K partial write JBD2 Journaling Running Transaction handle 2. flush out kworker 4. wait_on writeback() 5. COMMIT & locked Scheduler QASIO D data 27

28 Case Study: Ijhandle Page Cache Apps 1. 1K partial write Task A 3. 1K partial write Other Tasks I jhandle JBD2 Journaling Running Transaction handle 2. flush out kworker 4. wait_on writeback() 5. COMMIT & locked Scheduler QASIO D data 28

29 Boosting QASIOs Just focus on resolving direct dependencies Step 1: Detect a QASIO at run time When a task is waiting for the completion of an async In the VFS, Page Cache, and Ext4 Filesystem layers Step 2: Notify the scheduler Requires a new interface to give the information on QASIO In the block layer Step 3: Prioritize the QASIO In the scheduler 29

30 Detecting QASIOs VFS Layer Page Cache EXT4 Filesystem lock_buffer(); wait_on writeback(); ext4_free_data_callback(); Detect D meta Detect D data & D sync Detect D discard Block Layer CFQ Scheduler 4 next req CFQ. Request Tree : red-black tree : request 30

31 Notifying Scheduler VFS Layer Page Cache EXT4 Filesystem lock_buffer(); wait_on writeback(); ext4_free_data_callback(); Detect D meta Detect D data & D sync Detect D discard Block Layer elv_boost(); CFQ Scheduler 4 next req CFQ. Request Tree : red-black tree : request 31

32 Prioritizing QASIOs VFS Layer Page Cache EXT4 Filesystem lock_buffer(); wait_on writeback(); ext4_free_data_callback(); Detect D meta Detect D data & D sync Detect D discard Block Layer elv_boost(); CFQ Scheduler 4 QASIO list next req. 2 6 Boosting CFQ. Request Tree : red-black tree : request 32

33 Evaluations Samsung Galaxy S5 Exynos 5422 (quad Cortex-A15 & quad Cortex-A7) 2GB DRAM 16GB emmc storage (SW: 54.5MB/s, RW: 10.4MB/s) Android platform version (KitKat) Linux kernel version Evaluation methodology Microbenchmarks (M1 ~ M5) Real-life scenarios (Contacts, Camera, Angry Birds) Android benchmarks 33

34 Microbenchmarks M1 Iterates the creation of a 4KB file, 500 times creat() write(4kb) fsync() close() Mimics the pattern of DBMS (e.g. SQLite) A 8GB file is being created in the background M1 jbd2 I jcommit D meta D discard scheduler QASIO (meta) QASIO (discard) 34

35 Microbenchmarks M1 results 35

36 Launching Contacts App The UI task updates states into databases or files rename(), write(), fsync(), unlink(), All modify metadata (Dmeta) fsync() has Ijcommit to jbd2 UI task jbd2 I jcommit D meta D discard scheduler QASIO (meta) QASIO (discard) 36

37 Burst Mode in Camera App The thumbnail maker creates thumbnails after burst shot Writes data in a small size (Ddata) The camera task has Ijhandle to the thumbnail maker Camera task Thumbnail maker task I jhandle D data scheduler QASIO (data) 37

38 Installing Angry Birds App The App installer saves the extracted data to the app repository Buffered write() + fsync() (Dsync) fsync() has Ijcommit to jbd2 App installer task jbd2 I jcommit D sync D discard scheduler QASIO (data) QASIO (discard) 38

39 Android Benchmarks Antutu & RLBench Antutu benchmark RLBench benchmark 39

40 Conference ;login: (June 2015) 40

41 Thank You! Jin-Soo Kim Computer Systems Laboratory Sungkyunkwan University