Deadlock Victim && && && && by Dr Heinz Kabutz The Java Specialists Newsletter

Transcription

1 1 Deadlock Victim by Dr Heinz Kabutz The Java Specialists Newsletter && && && && Olivier Croisier The Coder's Breakfast

2 2 You discover a race condition

3 3 Data is being corrupted

4 4 What ya gonna do?

5 5 Who ya gonna call?

6 6 Ghostbusters?

7 7 you do what we all do

8 8 you use synchronized

9 9 synchronized

10 9 synchronized

11 10 synchronized can be bad medicine

12 11 deadlock awaits you!

13 12

14 13 Deadlock Victim

15 14 Correctness fix can lead to liveness fault

16 15 lock ordering deadlock

17 16 at least two locks

18 17 and at least two threads

19 18 locked in different orders

20 19 can cause a deadly embrace

21 20 requires global analysis to fix

22 21 resource deadlocks

23 22 bounded queues

24 23 bounded thread pools

25 24 semaphores

26 25 class loaders

27 26 lost signals

28 27 The Drinking Philosophers

29 the Java Specialist Symposium in Crete

30 29 Symposium == sym + poto Literally - to drink together

31 30 our philosophers need two cups

32 31 they either want to drink or think

33 Table is ready, all philosophers are thinking

34 philosophers 5 wants to drink, takes right cup

35 Philosopher 5 is now drinking with both cups

36 philosophers 3 wants to drink, takes right cup

37 Philosopher 3 is now drinking with both cups

38 philosophers 2 wants to drink, takes right cup

39 Philosopher 3 finished drinking, returns right cup

40 Philosopher 2 is now drinking with both cups

41 Philosopher 3 returns Left cup

42 41 philosophers can get stuck

43 42 if they all pick up the right cup

44 A deadlock can easily happen with this design

45 Philosopher 5 wants to drink, takes right cup

46 Philosopher 1 wants to drink, takes right cup

47 Philosopher 2 wants to drink, takes right cup

48 Philosopher 3 wants to drink, takes right cup

49 Philosopher 4 wants to drink, takes right cup

50 Deadlock!

51 they will all stay thirsty

52 this is a deadly embrace

53 52 this can be detected automatically

54 53 search the graph of call stacks

55 54 and look for circular dependencies

56 55 ThreadMXBean does that for us

57 56 simple deadlocks are quickly found

58 57 but what do you DO with them?

59 58 what will the dog do with the car if he ever catches it?

60 59 databases choose deadlock victim

61 60 but what does Java do?

62 61 it should probably throw an Error

63 62 but the threads simply hang up

64 63 it is best to avoid causing them!

65 64 global order of locks in program

66 public class LeftRightDeadlock { private final Object left = new Object(); private final Object right = new Object(); public void leftright() { synchronized (left) { synchronized (right) { dosomething(); } } } public void rightleft() { synchronized (right) { synchronized (left) { dosomethingelse(); } } } } 65

67 Interleaving causes deadlock lock left trying to lock right A leftright() BLOCKED lock right trying to lock left B rightleft() BLOCKED 66

68 67 we define a fixed global order

69 68 e.g. left before right

70 69 deadlock is solved! public void rightleft() { synchronized (left) { synchronized (right) { dosomethingelse(); } } }

71 70 our drinking philosophers

72 71 each cup gets a unique number

73 72 lock the highest number cup first

74 73 then lock the lower number cup

75 74 drink

76 75 unlock lower order cup

77 76 unlock higher order cup

78 77 think

79 Global Lock ordering We start with all the philosophers thinking

80 Philosopher 5 takes cup 5 Cup 5 has higher number 1 Remember our rule!

81 Philosopher 1 takes cup 2 Must take the cup with the higher number first 1 2 In this case cup

82 Philosopher 2 takes cup

83 Philosopher 3 takes cup

84 Philosopher 1 takes cup 1 - Drinking

85 Philosopher 1 returns Cup 1 Cups are returned in the opposite order to what they are acquired

86 Philosopher 5 takes cup 1 - Drinking

87 Philosopher 5 returns cup

88 Philosopher 1 returns cup

89 Philosopher 2 takes cup 2 - Drinking

90 Philosopher 5 returns cup

91 Philosopher 4 takes cup

92 Philosopher 2 returns cup

93 Philosopher 2 returns cup

94 Philosopher 3 takes cup 3 - Drinking

95 Philosopher 3 Returns cup

96 Philosopher 3 Returns cup

97 Philosopher 4 takes cup 4 - Drinking

98 Philosopher 4 Returns cup

99 Philosopher 4 Returns cup

100 99 sorting locks by property is easy

101 100 e.g. account IBAN number

102 101 or employee number

103 102 or some other characteristic

104 103 maybe even System.identityHashCode()

105 104 use "open calls"

106 105 Vector synchronizes writeobject() private synchronized void writeobject(objectoutputstream s) throws IOException { s.defaultwriteobject(); }

107 106 synchronized writeobject() is invoking s.defaultwriteobject() private synchronized void writeobject(objectoutputstream s) throws IOException { s.defaultwriteobject(); }

108 107 this is not an "open call" private synchronized void writeobject(objectoutputstream s) throws IOException { s.defaultwriteobject(); }

109 108 but can it cause problems? private synchronized void writeobject(objectoutputstream s) throws IOException { s.defaultwriteobject(); }

110 109 yes it can! private synchronized void writeobject(objectoutputstream s) throws IOException { s.defaultwriteobject(); }

111 110 serialize both vectors at same time Vector v1 = new Vector(); Vector v2 = new Vector(); v1.add(v2); v2.add(v1);

112 111 this could happen with nested data structures

113 112 it happened to an IBM client

114 private void writeobject(objectoutputstream stream) throws IOException { Vector<E> cloned = null; // this specially fix is for a special dead-lock in customer // program: two vectors refer each other may meet dead-lock in // synchronized serialization. Refer CMVC synchronized (this) { try { cloned = (Vector<E>) super.clone(); cloned.elementdata = elementdata.clone(); } catch (CloneNotSupportedException e) { // no deep clone, ignore the exception } } cloned.writeobjectimpl(stream); } private void writeobjectimpl(objectoutputstream stream) throws IOException { stream.defaultwriteobject(); } 113

115 114 Java 7 uses an "open call" private void writeobject(objectoutputstream s) throws IOException { final ObjectOutputStream.PutField fields = s.putfields(); final Object[] data; synchronized (this) { fields.put("capacityincrement", capacityincrement); fields.put("elementcount", elementcount); data = elementdata.clone(); } fields.put("elementdata", data); s.writefields(); }

116 More synchronized writeobject()s File StringBuffer Throwable URL Hashtable and others 115

117 116 we have seen the deadly embrace

118 117 are there other deadlocks?

119 118 semaphore deadlocks

120 119 Bounded DB Connection Pools For example, say you have two DB connection pools Some tasks might require connections to both databases Thus thread A might hold semaphore for D1 and wait for D2, whereas thread B might hold semaphore for D2 and be waiting for D1

121 120 public class DatabasePool { private final Semaphore connections; public DatabasePool(int connections) { this.connections = new Semaphore(connections); } public void connect() { connections.acquireuninterruptibly(); System.out.println("DatabasePool.connect"); } } public void disconnect() { System.out.println("DatabasePool.disconnect"); connections.release(); }

122 121 ThreadMXBean does not show this as a deadlock!

123 122 JVM does not detect this deadlock

124 122 JVM does not detect this deadlock

125 123 Avoiding and diagnosing deadlocks Avoiding Liveness Hazards

126 Avoiding and diagnosing deadlocks You cannot get a lock-ordering deadlock with a single lock Easiest way to avoid deadlocks But not always practical 124

127 Avoiding and diagnosing deadlocks For multiple locks we need to define a lock order specify and document possible lock sequences Identify where multiple locks could be acquired Do a global analysis to ensure that lock ordering is consistent This can be extremely difficult in large programs 125

128 126 Use open calls whenever possible

129 127 Unit Testing for lock ordering deadlocks Code typically has to be called many times before a deadlock happens How many times do you need to call it to prove that there is no deadlock? Nondeterministic unit tests are bad - they should either always pass or always fail

130 Testing the Bank In the transfermoney() method, a deadlock occurs if after the first lock is granted, the first thread is swapped out and another thread requests the second lock public class Bank { public boolean transfermoney(account from, Account to, Amount amount) { synchronized (from) { // if thread is swapped out here, we can end up with a deadlock synchronized (to) { return doactualtransfer(from, to, amount); } } } } 128

131 129 We can force the context switch

132 130 simply add a sleep after first lock

133 Adding a sleep to cause deadlocks public class Bank { public boolean transfermoney(account from, Account to, Amount amount) { synchronized (from) { sleepawhilefortesting(); synchronized (to) { return doactualtransfer(from, to, amount); } } } protected void sleepawhilefortesting() {} } 131

134 In our unit test we use a SlowBank public class SlowBank extends Bank { private final long timeout; private final TimeUnit unit; public SlowBank(long timeout, TimeUnit unit) { this.timeout = timeout; this.unit = unit; } protected void sleepawhilefortesting() { try { unit.sleep(timeout); } catch (InterruptedException e) { Thread.currentThread().interrupt(); } } } 132

135 133 Won't this cause problems in production?

136 134 In production we only use Bank

137 135 HotSpot will optimize away empty sleepawhilefortesting()

138 136 it is thus a zero cost

139 137 as long as SlowBank is never used

140 138 How can you verify the lockordering deadlock?

141 139 CTRL+Break, jstack, kill -3, jconsole, etc.

142 140 but that won't be a good unit test

143 141 Better via Java's ThreadMXBean

144 142 ThreadMXBean Get instance with ManagementFactory.getThreadMXBean() findmonitordeadlockedthreads() Includes only "monitor" locks, i.e. synchronized Only way to find deadlocks in Java 5

145 143 ThreadMXBean Better is finddeadlockedthreads() Includes "monitor" and "owned" (Java 5) locks Preferred method to test for deadlocks But, does not find deadlocks between semaphores See

146 144 public class BankDeadlockTest { private final static ThreadMXBean tmb = ManagementFactory.getThreadMXBean(); private boolean isthreaddeadlocked(long tid) { long[] ids = tmb.finddeadlockedthreads(); if (ids == null) return false; for (long id : ids) { if (id == tid) return true; } return false; }

147 145 private void checkthatthreadterminates(thread thread) throws InterruptedException { for (int i = 0; i < 2000; i++) { thread.join(50); if (!thread.isalive()) return; if (isthreaddeadlocked(thread.getid())) { fail("deadlock detected!"); } } fail(thread + " did not terminate in time"); }

148 @Test public void testfortransferdeadlock() throws InterruptedException { final Account alpha = new Account(new Amount(1000)); final Account ubs = new Account(new Amount( )); final Bank bank = new SlowBank(100, TimeUnit.MILLISECONDS); Thread alphatoubs = new Thread("alphaToUbs") { public void run() { bank.transfermoney(alpha, ubs, new Amount(100)); } }; Thread ubstoalpha = new Thread("ubsToAlpha") { public void run() { bank.transfermoney(ubs, alpha, new Amount(100)); } }; alphatoubs.start(); ubstoalpha.start(); } } checkthatthreadterminates(alphatoubs); 146

149 Output with broken transfermoney() method We see the deadlock within about 100 milliseconds junit.framework.assertionfailederror: Deadlock detected! at BankDeadlockTest.checkThatThreadTerminates(BankDeadlockTest.java:20) at BankDeadlockTest.testForTransferDeadlock(BankDeadlockTest.java:55) 147

150 Output with broken transfermoney() method If we fix the transfermoney() method, it also completes within about 100 milliseconds This is the time that we are sleeping for testing purposes Remember that the empty sleepawhilefortesting() method will be optimized away by HotSpot 148

151 149 Timed lock attempts Another technique for solving deadlocks is to use the timed trylock() method of explicit Java locks

152 150 However, if trylock() fails it may be

153 151 a deadlock

154 152 or some thread taking a long time

155 153 or a thread in an infinite loop

156 154 But ThreadMXBean shows it as deadlocked

157 155 even if it is a temporary condition

158 156 The JVM has support for trylock with synchronized

159 157 but it is hidden away

160 158 So here it is

161 159

162 160 Told you it was hidden!

163 161 It is hidden for a good reason

164 162 if you need trylock functionality

165 163 rather use explicit locks (ReentrantLock)

166 164 trylock with synchronized Technically possible in Sun JVM using Unsafe.getUnsafe().tryMonitorEnter(monitor) Unsafe.getUnsafe().monitorExit(monitor)

167 165 Deadlock analysis with thread dumps

168 Deadlock analysis with thread dumps We can also cause a thread dump in many ways: Ctrl+Break on Windows or Ctrl-\ on Unix Invoking "kill -3" on the process id Calling jstack on the process id Only shows deadlocks since Java 6 Intrinsic locks show more information of where they were acquired 166

169 167 Can a deadlock be resolved?

170 168 NO with synchronized

171 169 YES with ReentrantLock

172 170 synchronized() goes into BLOCKED state

173 171 ReentrantLock.lock() goes into WAITING state

174 172 We can write a DealockArbitrator

175 173 This stops one of the threads with a DeadlockVictimError

176 Our special Error for deadlocks public class DeadlockVictimError extends Error { private final Thread victim; public DeadlockVictimError(Thread victim) { super("deadlock victim: " + victim); this.victim = victim; } } public Thread getvictim() { return victim; } 174

177 public class DeadlockArbitrator { private static final ThreadMXBean tmb = ManagementFactory.getThreadMXBean(); public boolean tryresolvedeadlock( int attempts, long timeout, TimeUnit unit) throws InterruptedException { for (int i = 0; i < attempts; i++) { long[] ids = tmb.finddeadlockedthreads(); if (ids == null) return true; Thread t = findthread(ids[i % ids.length]); if (t == null) throw new IllegalStateException("Could not find thread"); t.stop(new DeadlockVictimError(t)); unit.sleep(timeout); } return false; } 175

178 finding threads by id is tricky public boolean tryresolvedeadlock() throws InterruptedException { return tryresolvedeadlock(3, 1, TimeUnit.SECONDS); } } private Thread findthread(long id) { for (Thread thread : Thread.getAllStackTraces().keySet()) { if (thread.getid() == id) return thread; } return null; } 176

179 Applicability of DeadlockArbitrator Only use in extreme circumstances Code that is outside your control and that deadlocks Where you cannot prevent the deadlock Remember, it only works with ReentrantLock 177

180 178 Deadlock Victim Questions? by Dr Heinz Kabutz The Java Specialists Newsletter && && && && Olivier Croisier The Coder's Breakfast