Lecture 7. Advanced Topics in Tes3ng

Transcription

1 Lecture 7 Advanced Topics in Tes3ng

2 Muta3on Tes3ng Muta3on tes3ng concerns evalua3ng test suites for their inherent quality, i.e. ability to reveal errors. Need an objec3ve method to determine quality Differs from structural coverage since it tries to define what is an error Basic idea is to inject defined errors into the SUT and evaluate whether a given test suite finds them. Killing a muta+on

3 Basic Idea We can sta3s3cally es3mate (say) fish in a lake by releasing a number of marked fish, and then coun3ng the number of marked fish in a subsequent small catch. Example: release 20 fish Catch 40 fish, 4 marked. Then 1 in 10 is marked, so es3mate 200 fish Pursue the same idea with marked SW bugs?

4 Muta3ons and Mutants The marked fish are injected errors, termed muta3ons The mutated code is termed a mutant Example: replace < by > in a one Boolean expression if ( x < 0 ) then becomes if ( x > 0 ) then If test suite finds muta3on we say this par3cular mutant is killed Make large set of mutants typically using a checklist of known muta3ons - muta3on tool

5 Muta3on score Idea is that if we can kill a mutant we can iden3fy a real bug too Mutants which are seman3cally equivalent to the original code are called equivalents Write Q P if Q and P are equivalents Clearly cannot kill equivalents Muta3on score % = Number of killed mutants total number of non- equivalent mutants *100

6 Why should it work? Two assump3ons are used in this field Competent programmer hypothesis i.e. The program is mostly correct and the Coupling Effect

7 Seman3c Neighbourhoods Let Φ be the set of all programs seman3cally close to P (defined in various ways) Φ is neighbourhood of P Let T be a test suite, f:d D be a func3onal spec of P Tradi3onally assume t T P.x = f(x) x D P.x = f(x) i.e. T is a reliable test suite Requires exhaus3ve tes3ng

8 Competent Programmer Hypothesis P is pathological iff P Φ Assume programmers have some competence Muta3on tes3ng assump3on Either P is pathological or else t T P.x = f(x) x D P.x = f(x) Can now focus on building a test suite T that would dis3nguish P from all other programs in Φ

9 Coupling Effect The competent programmer hypothesis limits the problem from infinite to finite. But remaining problem is s3ll too large Coupling effect says that there is a small subset μ Φ such that: We only need to dis3nguish P from all programs in μ by tests in T

10 Problems Can we be sure the coupling effect holds? Do simple syntac3c changes define a set? Can we detect and count equivalents? If we can t kill a mutant Q is Q P or is Q just hard to kill? How large is μ? May s3ll be too large to be prac3cal?

11 Equivalent Mutants Offut and Pan [1997] es3mated 9% of all mutants equivalent Bybro [2003] concurs with 8% Automa3c detec3on algorithms (basically sta3c analysers) detect about 50% of these Use theorem proving (verifica3on) techniques

12 Coupling Effect For Q an incorrect version of P Seman3c error size = Pr{Q.x P.x} If for every seman3cally large fault there is an overlap with at least one small syntac3c fault then the coupling effect holds. Selec3ve muta3on based on a small set of seman3cally small errors - Hard to kill

13 Early Research: 22 Standard (Fortran) muta3on operators AAR Array reference for array reference replacement ABS Absolute value inser3on ACR Array reference for constant replacement AOR Arithme3c operator replacement ASR Array reference for scalar replacement CAR Constant for array reference replacement CNR Comparable array name replacement CRP Constants replacement CSR Constant for Scalar variable replacement DER Do statement End replacement DSA Data statement altera3ons GLR Goto label replacement LCR Logical connector replacement ROR Rela3onal operator replacement RSR Return statement replacement SAN Statement analysis SAR Scalar for array replacement SCR Scalar for constant replacement SDL Statement dele3on SRC Source constant replacement SVR Scalar variable replacement UOI Unary operator inser3on

14 Recent Research: Java Muta3on Operators First leser is category A = access control E = common programming mistakes I = inheritance J = Java- specific features O = method overloading P = polymorphism

15 AMC Access modifier change EAM Accessor method change EMM Modifier method change EOA Reference assignment and content assignment replacement EOC Reference comparison and content comparison replacement IHD Hiding variable dele3on IHI Hiding variable inser3on IOD Overriding method dele3on IOP Overriding method calling posi3on change IOR Overridden method rename IPC Explicit call of parent s constructor dele3on ISK super keyword dele3on JDC Java supported default constructor create

16 JID Member variable ini3alisa3on dele3on JSC sta3c modifier change JTD this keyword dele3on OAO Argument order change OAN Argument number change OMD Overloaded method dele3on OMR Overloaded method contents change PMD Instance variable dele3on with parent class type PNC new method call with child class type PPD Parameter variable declara3on with child class type PRV Reference asssignment with other compa3ble type

17 Prac3cal Example Triangle program Myers complete test suite (13 test cases) Bybro [2003] Java muta3on tool and code 88% muta3on score, 96% statement coverage

18 Status of Muta3on Tes3ng Various strategies: weak muta3on, interface muta3on, specifica3on- based muta3on Our version is called strong muta3on Many muta3on tools available on the internet Cost of genera3ng mutants and detec3ng equivalents has come down Not yet widely used in industry S3ll considered academic, not understood?

19 Learning- based Tes3ng 1. Specifica3on- based Black- box Tes3ng 2. Learning- based Tes3ng paradigm (LBT) connec3ons between learning and tes3ng tes3ng as a search problem tes3ng as an iden3fica3on problem - tes3ng as a parameter inference problem 3. Example frameworks: 1. Procedural systems 2. Boolean reac+ve systems

20 Specifica3on- based Black- box Tes3ng 1. System requirement (Sys- Req) 2. System under Test (SUT ) 3. Test verdict pass/fail (Oracle step) Sys-Req pass/fail Test case Output TCG SUT Oracle Constraint solver Language runtime Constraint checker

21 Procedural System Example: Newton s Square root algorithm precondi3on x 0.0 Postcondi3on Ι y*y x Ι ε x=4.0 y=2.0 TCG Input SUT Output Oracle Constraint solver Newton Code Constraint checker x=4.0 sa3sfies x 0.0 Verdict x=4.0, y=2.0 sa3sfies Ι y*y x Ι ε

22 Reac3ve System Example: Coffee Machine Sys- Req: always( in=$1 implies ayer(10, out=coffee) ) pass/fail In 0 := $1 out 11 := coffee TCG Input SUT Output Oracle Constraint solver Coffee machine Constraint checker in 0 := $1, out 11 := coffee Sa3sfies always( in=1$ implies ayer(10, out=coffee))

23 Key Problem: Feedback Problem: How to modify this architecture to.. 1. Improve next test case using previous test outcomes 2. Execute a large number of good quality tests? 3. Obtain good coverage? 4. Find bugs quickly?

24 Learning-Based Testing Sys- Req pass/fail TCG Input SUT Output Oracle Sys-Model Learner Verdict Model based testing without a model

25 Basic Idea LBT is a search heuris3c that: 1. Incrementally learns an SUT model 2. Uses generalisa3on to predict bugs 3. Uses best predic3on as next test case 4. Refines model according to test outcome

26 Abstract LBT Algorithm 1. Use (i 1, o 1 ),, (i k, o k ) to learn model M k 2. Model check M k against Sys- Req 3. Choose best counterexample i k+1 from step 2 4. Execute i k+1 on SUT to produce o k+1 5. Check if (i k+1, o k+1 ) sa3sfies Sys- Req a) Yes: terminate with i k+1 as a bug b) No: goto step 1 Difficul3es lie in the technical details

27 General Problems Difficulty is to find combina3ons of models, requirements languages and checking algorithms (M, L, A) so that 1. models M are: - expressive, - compact, - par3al and/or local (an abstrac3on method) - easy to manipulate and learn 2. M and L are feasible to model check with A

28 Incremental Learning Real systems are too large to be completely learned Complete learning is not necessary to test many requirements (e.g. use cases) We use incremental (on- the- fly) learning Generate a sequence of refined models M 0 M 1 M i Convergence in the limit

29 Example: Boolean reac+ve systems 1. SUT: reac3ve systems 2. Model: determinis3c Kripke structure 3. Requirements: proposi3onal linear temporal logic (PLTL) 4. Learning: IKL incremental learning algorithm 5. Model Checker: NuSMV

30 LBT Architecture =&)<57+ 0)>#6+?()(/&65/ 4'1/0$!" B( $$$$$$$$$$$$$$$$$$$$$$$ ()*+,-+.$(/01/0$!!" # #$% C+* D#D,+E/4,+A+'0 " 8#9:;+ :5<(',-(*.(/ CD>56-(E$E 3#657&6& & ' + 1,+.4<0+.$ (/01/0$!& +B/&*'(!&$! ,'4'8 9*0(1 ' 13**9:$$$6$$$;!"!!"#$%&'()*(+ ' 012+!"#$%$& 3'45/$6-7 '!! 0,/+9*0(1

31 A Case Study: Elevator Model '(#4#'-## '-# '(# )*(,#*-# *(,#*-# '+# '+# )*+,!",#)./01,#)2(,#)2+,#)2-# *+,#!",)./01,#)2(,#)2+,#)2-# '(#4#'+# '-# *(,#)*-# *(,#*-# '(#4#'+#4#'-# '(#4#'-## *-,#!",)./01,#)2(,#)2+,#)2-# '(#4#'+#4#'-# '+# *(,#)*+# *(,#*+# '(# '+# '-# '(# '+#4#'-# '-# '(# '+# '(# '+#4#'-# )*(,#)*-# )*(,#*-# )*(,#)*+# )*(,#*+# '-# $%!&#!"# '+# 4#'-# '+# 4#'+# '-# $%!&#4#'-#4#'+# '-## $%!&#!"# '(## '-#4# '-#4# '-## '(#4#'-#4# $%!&#!"# )*-,#./01,#)2(,#)2+,#2-# '-# '-# '-# $%!&# '(# '(# )!"# '+# $%!&# '+# )!"# $%!&# '-# '-# )!"# )*+,)*-# )!",)*+,#*-# )!",*+,)*-# *+,#*-# '(# '(# '(# )*(,)*-# )!",)*(,#*-# )!",)*(,#*-# )!",*(,)*-# )!",*(,)*-# )!",*(,#*-# )!",*(,#*-# )*(,)*+# )!",)*(,#*+# )!",*(,)*+# )!",*(,#*+# '(# '+# '+# '+#'-# '+# '+# '+# '(# '+# '(# '+#4# '(# '+# '(#4# '(#4#'+#4# '(#4# '(#4# '(#4#'-#4# )*(,#./01,#2(,#)2+,#)2-# )*+,#./01,#)2(,#2+,#)2-# '-# *+,#*-# *+,#)*-# '(#4#'+#4#'-# '(#4#'+# '+# '+# '-# )*+,#*-# )*+,#)*-# '(# '(#4#'-# *(,!",#)./01,#)2(,#)2+,#)2-# '-# )*(,#*-# )*(,#)*-# '(# '+# '(# '+#4#'-### '-# *(,#*-# *(,#)*-# '(#4#'+#4#'-## '(#4#'+### *+,!",#)./01,#)2(,#)2+,#)2-# '+# '+# '-# *(,#*-# *(,#)*-# '(#4#'-## '(# )*+,#!",#)./01,#)2(,#)2+,#)2-#

32 Elevator Results Req t first (sec) t total (sec) MCQ first MCQ tot PQ first PQ tot RQ first RQ tot Req Req Req Req Req Req

33 Conclusions A promising approach Flexible general heuris3c, many models and requirement languages seem possible Many SUT types might be testable procedural, reac3ve, real- 3me etc. Open Ques3ons Benchmarking? Scalability? (abstrac3on, infinite state?) Efficiency? (model checking and learning?)