Mining Specifications

Transcription

1 Mining Specifications Andreas Zeller Saarland University

2 WINNER OF JOLT PRODUCTIVITY AWARD ANDREAS ZELLER WHY PROGRAMS FAIL A GUIDE TO SYSTEMATIC DEBUGGING SECOND EDITION

3 FOSE 10

4 FSE 10 Keynote Avoiding the Classic Catastrophic Computer Science Failure Mode Ralph Johnson

5 Research Failures 1. Knowledge-Based Software Engineering no ontologies 2. Semantic Web no annotations 3. Software Verification no specifications

6 Research Failures no ontologies no annotations no specifications

7 Research Failures no specifications

8 Saarland Graduates Michael Backes Andrej Rybalchenko TR35 in 2009 TR35 in 2010

9 secure Michael protocols Backes Andrej loop termination Rybalchenko hard to verify

10 buffer overflow resource leaks information flow liveness secure protocols loop termination easy to specify hard to verify

11 i {0,..., x } : x [i] <x [i + 1] x = x i {0,..., x } : ιi {0,..., x } : x[i] =x [i ] i {0,..., x } : ιi {0,..., x } : x [i ]=x[i] easy to verify hard to specify

12 is-sorted(x ) is-permutation(x, x ) still hard to specify

13 microsoft word mobile phones travel booking operating systems airplane control banking systems easy to verify hard to specify

14 hard to specify new language duplicate effort can t abstract from details

15 no specifications

16 Specification Crisis

17 <init>() auth()! socket: null state: NOT_CON openport() socket: null state: PLAIN quit() socket: null state: AUTH auth()

18 Mining Specifications <init>() auth()! socket: null state: NOT_CON openport() socket: null state: PLAIN quit() socket: null state: AUTH auth()

19 <init>() auth()! socket: null state: NOT_CON openport() socket: null state: PLAIN quit() socket: null state: AUTH auth()

20 <init>() auth()! socket: null state: NOT_CON openport() socket: null state: PLAIN quit() socket: null state: AUTH auth()

21 what s the abstraction behind this?

22 state of the art

23 sqrt (x: REAL, eps: REAL): REAL is Square root of x with precision eps require x >= 0 eps > 0 precondition external csqrt (x: REAL, eps: REAL): REAL do Result := csqrt (x, eps) ensure postcondition abs (Result ^ 2 x) <= eps end sqrt

24 sqrt (x: REAL, eps: REAL): REAL is Square root of x with precision eps external csqrt (x: REAL, eps: REAL): REAL do Result := csqrt (x, eps) end sqrt How do we infer the postcondition?

25 sqrt (x: REAL, eps: REAL): REAL is Square root of x with precision eps end sqrt How do we infer the postcondition?

26 Static Analysis sqrt (x: REAL, eps: REAL): REAL is Square root of x with precision eps end sqrt Analysis applies to all possible runs Fails in presence of obscure code Hard to maintain precision while scaling

27 Dynamic Analysis sqrt (x: REAL, eps: REAL): REAL is Square root of x with precision eps end sqrt Analysis applies to observed runs Needs actual (many) executions Scales, but underapproximates

28 Daikon sqrt (x: REAL, eps: REAL): REAL is Square root of x with precision eps end sqrt require x 4 x 256 eps > 0 too specific ensure Result * Result x < eps Result 2 Result 16 abs() is missing is this important?

29 Mining Behavior We need to infer preconditions infer postconditions rank specifications Common behavior is correct behavior

30 Mining Behavior Learning behavior examples from real behavior generated behavior elicited behavior Common behavior is correct behavior

31 Mining Behavior Learning behavior examples from real behavior generated behavior elicited behavior Common behavior is correct behavior

32

33 <init>() auth()! socket: null state: NOT_CON openport() socket: null state: PLAIN quit() socket: null state: AUTH auth()

34 Specifications <init>() auth()! socket: null state: NOT_CON openport() socket: null state: PLAIN quit() socket: null state: AUTH auth() finite state models with pre- and postconditions (for now)

35 <init>() auth()! socket: null state: NOT_CON openport() socket: null state: PLAIN quit() socket: null state: AUTH auth()

36 <init>() auth()! socket: null state: NOT_CON openport() socket: null state: PLAIN quit() socket: null state: AUTH auth() Common behavior is correct behavior

37 <init>() auth()! socket: null state: NOT_CON openport() socket: null state: PLAIN quit() socket: null state: AUTH auth() Common behavior is correct behavior

38 AspectJ for (Iterator iter = itdfields.iterator(); iter.hasnext();) {... for (Iterator iter2 = worthretrying.iterator(); iter.hasnext();) {... should be iter2 } }

39 AspectJ public String getretentionpolicy () {... for (Iterator it =...; it.hasnext();) {... = it.next();... return retentionpolicy; }... should be fixed }

40 Conspire static int dcc_listen_init ( ) { dcc->sok = socket ( ); if ( ) { while ( ) { = bind (dcc->sok, ); } /* with a small port range, reuseaddr is needed */ setsockopt (dcc->sok,, SO_REUSEADDR, ); } listen (dcc->sok, ); } must occur before bind()

41 <init>() auth()! socket: null state: NOT_CON openport() socket: null state: PLAIN quit() socket: null state: AUTH auth() Common behavior is correct behavior

42 200,000, ,000, ,000, ,000, Lines of Code 0 C Projects

43

44

45 Mining Behavior Learning behavior examples from real behavior generated behavior elicited behavior Common behavior is correct behavior

46 Mining Behavior Learning behavior examples from real behavior generated behavior elicited behavior Common behavior is correct behavior

47 Enriching specifications void ProtocolTest() { Protocol p = new... p.conn(); p.send(x); p.quit(); } initial spec enriched spec Execute and extract initial spec Generate test mutants and enrich specs Dallmeier et al: Generating Test Cases for Specification Mining, ISSTA 2010

48 void ProtocolTest() { Protocol p = new... p.conn(); p.send(x); p.quit(); } void TestMutant1() { void Uncovered TestMutant2() Protocol p = new... Protocol new... 0: p.conn(); //p.conn(); send(x) p.send(x); p.send(x); p.conn(); quit() p.quit(); p.quit(); 1: conn() } SMTPProtocol <init>() 0 send(x)? send(x) start quit() conn() EX 1 send(x) conn()? conn() Dallmeier et al: Generating Test Cases for Specification Mining, ISSTA 2010

49 enriched initial Dallmeier et al: Generating Test Cases for Specification Mining, ISSTA 2010

50 Generate test cases to systematically explore specification Assess executions to learn about software behavior Dallmeier et al: Generating Test Cases for Specification Mining, ISSTA 2010

51 Evaluation

52 Do enriched specs contain more information? initial enriched SMTPProtocol Signature ZipOutputStream Enriched specs have more regular and exceptional transitions states exceptions transitions states exceptions transitions states exceptions transitions

53 How effective are enriched specifications? 100% 80% 60% 40% reported at correct location total reported SMTPProtocol Signature ZipOutputStream Enriched specs are can better almost suited as to effective finding as errors manually than initial crafted specs 20% 0% initial manual enriched initial manual enriched initial manual enriched

54 A new kind of Analysis Static analysis Dynamic analysis Experimental analysis 0 runs n given runs n generated runs

55 Mining Behavior Learning behavior examples from real behavior generated behavior elicited behavior Common behavior is correct behavior

56 Mining Behavior Learning behavior examples from real behavior generated behavior elicited behavior Common behavior is correct behavior

57 Generate test cases to systematically explore specification Assess executions to learn about software behavior Dallmeier et al: Generating Test Cases for Specification Mining, ISSTA 2010

58 Generate test cases to systematically explore specification Assess executions to learn about software behavior

59 Eliciting Behavior Generate test cases to systematically explore specification Assess executions to learn about software behavior

60 code initial test case

61 generated test case initial test case initial test case generated test case generated test case code

62 generated test case initial test case generated test case generated test case

63 generated test case generated test case initial test case generated test case generated test case

64 easy to understand include oracles act as specifications generated test case initial test case generated test case generated test case

65 sqrt (x: REAL, eps: REAL): REAL is Square root of x with precision eps end sqrt sqrt(4, 0) = 2?

66 sqrt (x: REAL, eps: REAL): REAL is Square root of x with precision eps end sqrt sqrt(9, 0) = 2?

67 sqrt (x: REAL, eps: REAL): REAL is Square root of x with precision eps end sqrt sqrt(9, 0) = 3?

68 sqrt (x: REAL, eps: REAL): REAL is Square root of x with precision eps end sqrt require x >= 0 eps > 0 sqrt( 1, 0) =? Oops!

69 sqrt (x: REAL, eps: REAL): REAL is Square root of x with precision eps end sqrt require x >= 0 eps > 0 Result ^ 2 = x?

70 sqrt (x: REAL, eps: REAL): REAL is Square root of x with precision eps end sqrt require x >= 0 eps > 0 ensure Result ^ 2 = x eps

71 Eliciting Behavior Generate test cases to systematically explore specification Assess test cases to refine software behavior

72 Eliciting Use Cases

73 Mining Behavior Learning behavior examples from real behavior generated behavior elicited behavior Common behavior is correct behavior

74 Mining Specifications