Dirty throttling How much dirty memory is too much?

Transcription

1 Dirty throttling How much dirty memory is too much? Wu Fengguang November 4, 2010

2 Outline writeback basics dirty limits dirty throttling algorithms Wu Fengguang (Intel OTC) dirty throttling China Linux Kernel Developers 2 / 27

3 page cache pages new page: no valid data PG_uptodate: have valid data PG_dirty: have valid data, to be synced to disk PG_writeback: have valid data, being synced to disk clean page: PG_uptodate &&!PG_dirty &&!PG_writeback dirty page: PG_uptodate && (PG_dirty PG_writeback) Wu Fengguang (Intel OTC) dirty throttling China Linux Kernel Developers 3 / 27

4 writeback: delaying the write IO. Benefits.. async IO: avoid blocking the apps... avoid IO (eg. temp files) batched IO: better throughput Question: When to writeback the dirty pages? Wu Fengguang (Intel OTC) dirty throttling China Linux Kernel Developers 4 / 27

5 option 1: sync syscalls - sync() - fsync() - fdatasync() - sync_file_range() - msync() - open(o_sync) - open(o_direct) Wu Fengguang (Intel OTC) dirty throttling China Linux Kernel Developers 5 / 27

6 option 2: the flusher thread(s) initiate writeback IO in the background one flusher thread per storage device. $ ps ax PID TTY STAT TIME COMMAND 2322? S 0:01 [flush-8:0] ?.. S 0:00 [flush-btrfs-1] Wu Fengguang (Intel OTC) dirty throttling China Linux Kernel Developers 6 / 27

7 when to writeback - dirty expire time: 30 seconds - background flush threshold: 10% memory dirtied - dirty throttling threshold: 20% memory dirtied Wu Fengguang (Intel OTC) dirty throttling China Linux Kernel Developers 7 / 27

8 who to initiate IO fsync() the call task SLOW sync() the flusher thread periodic writeback the flusher thread background writeback the flusher thread. problematic writeback paths.. balance_dirty_pages() the current dirtier slow page reclaim kswapd and/or VERY... page allocate task SLOW SLOW = IO inefficient + slow responsiveness Wu Fengguang (Intel OTC) dirty throttling China Linux Kernel Developers 8 / 27

9 dirty limits illustrated mem nr_dirtied (app) 20% nr_dirty 10% nr_written (disk) time Wu Fengguang (Intel OTC) dirty throttling China Linux Kernel Developers 9 / 27

10 per-device dirty limits solution for: inter device starvation Wu Fengguang (Intel OTC) dirty throttling China Linux Kernel Developers 10 / 27

11 per-task dirty limits solution for: inter process starvation Wu Fengguang (Intel OTC) dirty throttling China Linux Kernel Developers 11 / 27

12 balance_dirty_pages() sys_write() balance_dirty_pages() i f (task_dirty_exceeded ()) writeback_inodes( dirtied * 3/2); i f (over_bground_thresh ()) start_background_writeback (); Wu Fengguang (Intel OTC) dirty throttling China Linux Kernel Developers 12 / 27

13 balance_dirty_pages() parameters Ideal is: (1) dirtied pages < dirty_limit/100 (easy) (2) 1ms < pause time < 100ms (important) Wu Fengguang (Intel OTC) dirty throttling China Linux Kernel Developers 13 / 27

14 balance_dirty_pages() is not latency wise Problems: (1) pause time won t scale to storage speed (2) pause time fluctuates a lot in one task Wu Fengguang (Intel OTC) dirty throttling China Linux Kernel Developers 14 / 27

15 balance_dirty_pages() is not IO wise seeky IO parallel dirtiers => N dirtiers working on N inodes => interleaved IO to multiple disk regions small IO size pause time limit => small write size => small extent size => small read size Solution: IO-less balance_dirty_pages() Wu Fengguang (Intel OTC) dirty throttling China Linux Kernel Developers 15 / 27

16 try 1: wait for IO completion i f (task_dirty_exceeded ()) - writeback_inodes( dirtied * 3/2); + wait_for_writeback( dirtied * 3/2); - bumpy IO completion on NFS - accounting inaccuracy and overheads Wu Fengguang (Intel OTC) dirty throttling China Linux Kernel Developers 16 / 27

17 try 2: sleep for estimated time i f (task_dirty_exceeded ()) - wait_for_writeback( dirtied * 3/2); + sleep( dirtied * 3/2 / write_bandwidth); - estimation problem on multiple sleepers - estimation problem with advanced limits Wu Fengguang (Intel OTC) dirty throttling China Linux Kernel Developers 17 / 27

18 try 3: sleep for controlled time i f (task_dirty_exceeded ()) - sleep( dirtied * 3/2 / write_bandwidth); + sleep( dirtied / throttle_bandwidth); + directly control pause time + convenient for dynamic limit and IO controller Wu Fengguang (Intel OTC) dirty throttling China Linux Kernel Developers 18 / 27

19 throttle bandwidth device_limit = device_weight global_limit (1) task_limit = device_limit task_weight device_limit 16 (2) throttle_bandwidth = device_bandwidth task_limit nr_dirty task_limit/16 (3) device_weight/task_weight: the percent of pages written/dirtied recently by the device/task Wu Fengguang (Intel OTC) dirty throttling China Linux Kernel Developers 19 / 27

20 throttle bandwidth (state 1) heavy dirtier + light dirtier Wu Fengguang (Intel OTC) dirty throttling China Linux Kernel Developers 20 / 27

21 throttle bandwidth (state 2) light dirtier => heavy dirtier Wu Fengguang (Intel OTC) dirty throttling China Linux Kernel Developers 21 / 27

22 throttle bandwidth (state 3) stable state: two heavy dirtiers Wu Fengguang (Intel OTC) dirty throttling China Linux Kernel Developers 22 / 27

23 pause time pause_time = nr_dirtied / throttle_bandwidth if (pause_time < 1ms) max_nr_dirtied += 1; else if (pause_time > 100ms) max_nr_dirtied /= 2; Wu Fengguang (Intel OTC) dirty throttling China Linux Kernel Developers 23 / 27

24 dynamic dirty limit: rational dirty pages may hurt under memory pressure eats 20% memory triggers writeback on page reclaim 4k seeky IO high latency LRU list <======= * * *-----*-----**----*---*--*-*- reclaim [-] clean page [*] dirty page Wu Fengguang (Intel OTC) dirty throttling China Linux Kernel Developers 24 / 27

25 dynamic dirty limit via soft dirty limit stable state: two heavy dirtiers Wu Fengguang (Intel OTC) dirty throttling China Linux Kernel Developers 25 / 27

26 what s next per-cgroup dirty limits write IO controller Wu Fengguang (Intel OTC) dirty throttling China Linux Kernel Developers 26 / 27

27 Thank you! Wu Fengguang (Intel OTC) dirty throttling China Linux Kernel Developers 27 / 27