LET ME SLEEP ON IT. Improving concurrency in unexpected ways. Nir Soffer PyCon Israel 2017

Transcription

1 LET ME SLEEP ON IT Improving concurrency in unexpected ways Nir Soffer PyCon Israel 2017

2 $ whoami Taming the Vdsm beast since 2013 Tinkering with Python since 2003 Free software enthusiast Father of two

3 PYTHON THREADS SUCK

4 THE GIL The global interpretor lock will let only one thread run Python code at a time

5 WHAT'S GOING ON HERE? Py_BEGIN_ALLOW_THREADS n = read(fd, buf, count); Py_END_ALLOW_THREADS Python releases the GIL when possible

6 PYTHON THREADS ARE USEFUL You can do I/O or wait for other programs concurrently

7 SEQUENTIAL CODE IS EASY When using threads, you can write simple and clear code, as if you are the only one running

8 VDSM USES LOT OF THREADS

9 WHAT IS VDSM? Vdsm manages virtual machines on a hypervisor

10 VIRTUAL MACHINES NEED STORAGE Vdsm provides storage for virtual machines Typically shared storage

11 SHARED STORAGE IS EXTREMELY FAST Seen 750MiB/s writes using direct I/O to SSD disk array

12 SHARED STORAGE IS HORRIBLY SLOW $ ps -p o stat -o cmd STAT CMD D+ dd if=/dev/zero of=mnt/test bs=8m count=1280 oflag=direct

13 VDSM MONITORS STORAGE Every storage domain has a dedicated thread We can have 50 storage domains

14 MONITOR THREADS ARE ISOLATED If one thread gets stuck on unresponsive storage, other threads are not affected

15 VDSM USES LVM Vdsm manages block storage using LVM Creates logical volumes for VM disks and snapshots

16 ACCESSING LVM METADATA IS SLOW Vdsm caches LVM metadata

17 There are only two hard things in Computer Science: cache invalidation and naming things. -- Phil Karlton

18 MONITOR THREADS REFRESH LVM CACHE Monitor threads invalidate the cache and run LVM commands to reload the cache

19 OPERATION MUTEX

20 LVM CACHE class LVMCache(object):... def _invalidate_lvs(self, vg_name, lv_names): with self._opmutex.locked(lvm_op_invalidate): for lv_name in lv_names: self._lvs[(vg_name, lv_name)] = Stub(lv_name) def _reload_lvs(self, vg_name, lv_names): with self._opmutex.locked(lvm_op_reload): lvm_output = self._run_lvs(vg_name, lv_names) for lv in self._parse_lvs(lvm_output): self._lvs[(vg_name, lv_name)] = lv (Simpli ed)

21 LVM CACHE USES FANCY LOCKING Multiple threads can invalidate the cache at the same time Multiple threads can reload the cache at the same time Invalidate and reload cannot run at the same time

22 HOW OPERATION MUTEX WORKS 1. Thread-1 acquires the mutex for an invalidate operation 2. Thread-2 tries to acquire the mutex for a reload operation, waiting Thread-3 enters the mutex for an invalidate operation 4. Thread-1 exits the mutex 5. Thread-3 exits and release the mutex 6. Thread-2 wakes up and acquires the mutex for a reload operation

23 OPERATION MUTEX [1/3] class OperationMutex(object): def init (self): self._cond = threading.condition(threading.lock()) self._operation = None self._holders = def locked(self, operation): self._acquire(operation) try: yield self finally: self._release()

24 OPERATION MUTEX [2/3] def _acquire(self, operation): with self._cond: while self._operation not in (operation, None): self._cond.wait() if self._operation is None: self._operation = operation self._holders += 1 (Logging removed)

25 OPERATION MUTEX [3/3] def _release(self): with self._cond: self._holders -= 1 if self._holders == 0: self._operation = None self._cond.notify_all() (Logging removed)

26 OPERATION MUTEX IN PRACTICE (_invalidate_lvs) 'lvm reload' is holding the mutex, waiting... (_reload_vgs) 'lvm reload' is holding the mutex, waiting... (_invalidate_lvs) 'lvm reload' is holding the mutex, waiting... (_invalidate_lvs) 'lvm reload' is holding the mutex, waiting... (_reload_vgs) 'lvm reload' is holding the mutex, waiting... (_invalidate_lvs) 'lvm reload' is holding the mutex, waiting... (_reload_vgs) 'lvm reload' is holding the mutex, waiting... (_invalidate_lvs) 'lvm reload' is holding the mutex, waiting... (_invalidate_lvs) 'lvm reload' is holding the mutex, waiting... (_invalidate_lvs) 'lvm reload' is holding the mutex, waiting... (_invalidate_lvs) 'lvm reload' is holding the mutex, waiting... (_invalidate_lvs) 'lvm reload' is holding the mutex, waiting... (_invalidate_lvs) 'lvm reload' is holding the mutex, waiting... (_invalidate_lvs) 'lvm reload' is holding the mutex, waiting... (_invalidate_lvs) 'lvm reload' is holding the mutex, waiting... (_invalidate_lvs) 'lvm reload' is holding the mutex, waiting... (Simpli ed, colored)

27 WE HAVE A PROBLEM Too many threads waiting... Most of the time only one thread is inside the operation mutex

28 OPERATION MUTEX IS HARMFUL Lots of storage domains... Storage is overloaded... LVM commands become slow... Monitor threads waiting... Monitoring timeouts... Hypervisor goes down

29 IMPROVING CONCURRENCY

30 HOW ARE OPERATION MUTEX TESTS PASSING? There are no Operation Mutex tests

31 Code without tests is broken. -- Nir Soffer

32 ADDING TESTS Let's start with the easy case, allowing multiple threads to perform the same operation

33 TESTING SAME OPERATION def test_same_operation(): m = opmutex.operationmutex() def worker(n): with m.locked("operation"): time.sleep(1.0) elapsed = run_threads(worker, 50) assert elapsed < 2.0

34 WE NEED SOME HELP def run_threads(func, count): threads = [] start = time.time() try: for i in range(count): t = threading.thread(target=func, args=(i,), name="worker-%02d" % i) t.daemon = True t.start() threads.append(t) finally: for t in threads: t.join() return time.time() - start

35 TEST PASSES! opmutex_test.py::test_same_operation PASSED

36 REFRESH LVM CACHE FLOW 1. Invalidate cache 2. Reload cache

37 TIME MATTERS 1. Invalidate updates a dict - microseconds 2. Reload goes to storage - seconds

38 TESTING REFRESH FLOW def test_refresh_flow(run): count = 50 m = opmutex.operationmutex() cache = ["old"] * count def worker(n): with m.locked("invalidate"): cache[n] = "invalid" with m.locked("reload"): time.sleep(1.0) cache[n] = "new" elapsed = run_threads(worker, count) assert elapsed < 2.0 assert cache == ["new"] * count

39 TEST FAILS RANDOMLY Testing threading issues is tricky How can we make it fail consistently?

40 MAKING IT range(10)) def test_refresh_flow(run): count = 50 m = opmutex.operationmutex() cache = ["old"] * count def worker(n): with m.locked("invalidate"): cache[n] = "invalid" with m.locked("reload"): time.sleep(1.0) cache[n] = "new" elapsed = run_threads(worker, count) assert elapsed < 2.0

41 FAILING TESTS ARE GOOD opmutex_test.py::test_same_operation PASSED opmutex_test.py::test_refresh_flow[0] FAILED opmutex_test.py::test_refresh_flow[1] PASSED opmutex_test.py::test_refresh_flow[2] PASSED opmutex_test.py::test_refresh_flow[3] PASSED opmutex_test.py::test_refresh_flow[4] PASSED opmutex_test.py::test_refresh_flow[5] FAILED opmutex_test.py::test_refresh_flow[6] PASSED opmutex_test.py::test_refresh_flow[7] FAILED opmutex_test.py::test_refresh_flow[8] PASSED opmutex_test.py::test_refresh_flow[9] FAILED

42 TEST FAILURE LOG worker-00: Operation 'invalidate' acquired the mutex worker-00: Operation 'invalidate' released the mutex worker-00: Operation 'reload' acquired the mutex worker-01: Operation 'reload' is holding the mutex, waiting... worker-02: Operation 'reload' is holding the mutex, waiting... worker-03: Operation 'reload' is holding the mutex, waiting... worker-04: Operation 'reload' is holding the mutex, waiting... worker-05: Operation 'reload' is holding the mutex, waiting... worker-06: Operation 'reload' is holding the mutex, waiting... worker-07: Operation 'reload' is holding the mutex, waiting... worker-08: Operation 'reload' is holding the mutex, waiting... worker-09: Operation 'reload' is holding the mutex, waiting... worker-10: Operation 'reload' is holding the mutex, waiting...

43 WHY DOES IT FAIL? 1. worker-00 acquires the GIL 2. worker-00 acquires the operation mutex for an invalidate operation 3. Other workers could enter the operation mutex, but worker- 00 is holding the GIL worker-00 releases the operation mutex 5. worker-00 acquires the operation mutex again for a reload operation 6. worker-00 releases the GIL during sleep 7. Other workers cannot enter the operation mutex, waiting...

44 HOW SHOULD IT WORK 1. All workers enter the operation mutex for an invalidate operation 2. All workers exit the operation mutex 3. All workers enter the operation mutex for a reload operation 4. All workers exit the operation mutex

45 CAN WE FIX IT? Need to sleep on it...

46 THREADS ARE NOT POLITE When you enter a building, you hold the door so the next person can enter

47 HOW CAN WE MAKE THREADS POLITE? When entering the operation mutex, take a little nap, letting other threads in

48 TAKE A LITTLE def locked(self, operation): self._acquire(operation) try: # Give other threads chance to get in. time.sleep(0.01) yield self finally: self._release()

49 GREEN AGAIN! opmutex_test.py::test_same_operation PASSED opmutex_test.py::test_refresh_flow[0] PASSED opmutex_test.py::test_refresh_flow[1] PASSED opmutex_test.py::test_refresh_flow[2] PASSED opmutex_test.py::test_refresh_flow[3] PASSED opmutex_test.py::test_refresh_flow[4] PASSED opmutex_test.py::test_refresh_flow[5] PASSED opmutex_test.py::test_refresh_flow[6] PASSED opmutex_test.py::test_refresh_flow[7] PASSED opmutex_test.py::test_refresh_flow[8] PASSED opmutex_test.py::test_refresh_flow[9] PASSED

50 FIXED TEST LOG worker-00: Operation 'invalidate' acquired the mutex worker-01: Operation 'invalidate' entered the mutex worker-02: Operation 'invalidate' entered the mutex worker-03: Operation 'invalidate' entered the mutex worker-04: Operation 'invalidate' entered the mutex worker-05: Operation 'invalidate' entered the mutex worker-06: Operation 'invalidate' entered the mutex worker-00: Operation 'invalidate' exited the mutex worker-07: Operation 'invalidate' entered the mutex worker-01: Operation 'invalidate' exited the mutex worker-00: Operation 'invalidate' is holding the mutex, waiting... worker-02: Operation 'invalidate' exited the mutex...

51 FIX AVAILABLE IN OVIRT-3.6

52 ARE WE DONE?

53 WHY DO WE NEED THE OPERATION MUTEX? Need to sleep on it little bit more...

54 WE DON'T No need to separate invalidate and reload operations Multiple threads modifying the cache is not thread safe The real problem is monitor thread isolation Monitor threads should not refresh LVM cache

55 OPERATION MUTEX WAS REMOVED IN OVIRT-4.0 More on this in my PyCon Israel 2018 talk?

56 The Best Code is No Code At All. --

57 MORE INFO Fork this talk on github ovirt - open your virtual datacenter We are hiring

58 THANK YOU! Questions?