Introdutory Programming, IMM, DTU Systemati Software Test Peter Sestoft a Programs often ontain unintended errors how do you find them? Strutural test Funtional test Notes: Systemati Software Test, http://www.dina.kvl.dk/ sestoft/programmering/struktur.pdf Software test (afprøvning) Systemati ativity. Goal: to reveal errors in the program Input data set: arefully designed speially for this purpose. Systemati software test is quality assessment. a. Translated into English by Morten P. Lindegaard. Minor hanges and some additions made by nne Haxthausen. Sestoft, 18. november 200202100+02115+02199+02312 Introdutory Programming Page test-1 Sestoft, 18. november 200202100+02115+02199+02312 Introdutory Programming Page test-3 Motivation Non-trivial programs almost always ontain unintended (but avoidable) errors. Strutural test and funtional test Errors in programs may have severe onsequenes: Strutural test Funtional test In the Gulf war (1991), some Patriot missiles failed to hit inoming Iraqi Sud missiles (whih subsequently killed people on the ground). Errors in the software ontrolling the baggage handling system of Denver International irport delayed its opening by a year (1995), ausing losses of around 360 million dollars. The first launh of the European riane 5 roket failed (1996), ausing losses of hundreds of million dollars. (Programming error and insuffiient test). Errors in poorly designed ontrol software in the Thera-25 radio-therapy equipment (1987) exposed several aner patients to heavy doses of radiation, killing some. Our programs are simpler and ause smaller aidents. But the programs should still not ontain errors. You may prove that the programs are orret (by loop invariants et.). This is done for e.g. Metro ontrol-programs (Frane), misellaneous equipment in spaeraft (NS),... But it is often too expensive, too labourious, or too time-onsuming. Furthermore, it only prevents ertain kinds of errors. Badly designed user-interfaes are e.g. not prevented. Starting point the soure ode the problem Found logial errors unobserved ases kinds of errors wrong initialization of variables unobserved requirements Strutural test is also known as internal test or white-box test. Funtional test is also known as funtional test or blak-box test. In both ases, the first task is to design a test suite (afprøvning) ontaining: a table of input data properties a table of input data set and the orresponding, expeted output data To test the program, it is run with the input data sets, after whih the atual output data is ompared with the expeted output data. Sestoft, 18. november 200202100+02115+02199+02312 Introdutory Programming Page test-2 Sestoft, 18. november 200202100+02115+02199+02312 Introdutory Programming Page test-4
Strutural test: Conditional- and repetition statements Weak purpose: all parts of the program have been exeuted. Espeially, all branhes of onditional statements (if, swith, try-ath,... ) must have been exeuted. Strong purpose: every repetition statement has been exeuted zero times, exeuted exatly one time, and exeuted more than one time. This heks that variables have reasonable values before the first exeution of the body of the loop, as well as the following exeutions. Statement Cases to test if Condition and for Zero, one, and more than one exeutions while Zero, one, and more than one exeutions do-while One and more than one exeutions swith Every branh must be exeuted Strutural test, example 1: find the smallest and the greatest number (Minmax.java) publi lass Minmax { publi stati void main ( String[] args ) { int mi, ma; //urrent minimum and maximum System.out.println(""); mi = ma = Integer.parseInt(args[0]); for (int i = 1; i < args.length; i = i+1) { /* 2 */ int obs = Integer.parseInt(args[i]); if (obs > ma) ma = obs; /* 3 */ else if (mi < obs) mi = obs; /* 4 */ System.out.println("Minimum = " + mi + "; maximum = " + ma); Sestoft, 18. november 200202100+02115+02199+02312 Introdutory Programming Page test-5 Sestoft, 18. november 200202100+02115+02199+02312 Introdutory Programming Page test-7 Strutural test: Composite logial expressions Test all possible ombinations of truth values for terms. Conjuntion (logial and, &&): (x!= 0) && (1000/x > y) yields (x!= 0) && (1000/x > y) Disjuntion (logial or, ): (x == 0) (1000/x > y) yields (x == 0) (1000/x > y) Table of test ases Choie Input data set Input property 1 1 B t least one number 2 zero times B Exatly one number 2 one C Exatly two numbers 2 more than one E t least three numbers 3 C Current number urrent maximum 3 D Current number 4 E Current number urrent maximum urrent maximum and 4 E Current number maximum and urrent In the program: urrent maximum == ma, urrent minimum == mi, urrent number == obs urrent minimum urrent minimum Sestoft, 18. november 200202100+02115+02199+02312 Introdutory Programming Page test-6 Sestoft, 18. november 200202100+02115+02199+02312 Introdutory Programming Page test-8
Error! Table of input data sets and expeted output data Input data set Contents Expeted output B 17 Minimum = 17; maximum = 17 C 27 29 Minimum = 27; maximum = 29 D 39 37 Minimum = 37; maximum = 39 E 49 47 48 Minimum = 47; maximum = 49 Input data sets D and E yield wrong output: Minimum = 39; maximum = 39 Minimum = 49; maximum = 49 Reason: erroneous ondition at 4. Correted test ases Choie Input data set Input property 1 1 B t least one number 2 zero times B Exatly one number 2 one C Exatly two numbers 2 more than one E t least three numbers 3 C Current number urrent maximum 3 D Current number 4a E Current number urrent maximum urrent maximum and 4a E Current number maximum and urrent The old input data sets may be used again. urrent minimum urrent minimum Sestoft, 18. november 200202100+02115+02199+02312 Introdutory Programming Page test-9 Sestoft, 18. november 200202100+02115+02199+02312 Introdutory Programming Page test-11 Strutural test, example 2: find the two smallest numbers (possibly equal) (Mintwo.java) Correted program publi lass Minmax { publi stati void main ( String[] args ) { int mi, ma; //urrent minimum and maximum System.out.println(""); mi = ma = Integer.parseInt(args[0]); for (int i = 1; i < args.length; i = i+1) { /* 2 */ int obs = Integer.parseInt(args[i]); if (obs > ma) ma = obs; /* 3 */ else if (obs < mi) mi = obs; /* 4 */ System.out.println("Minimum = " + mi + "; maximum = " + ma); publi stati void main (String[] args) { int mi1 = 0, mi2 = 0; System.out.println(""); mi1 = Integer.parseInt(args[0]); if (args.length == 1) /* 2 */ System.out.println("Smallest = " + mi1); int obs = Integer.parseInt(args[1]); if (obs < mi1) /* 3 */ { mi2 = mi1; mi1 = obs; for (int i = 2; i < args.length; i = i+1) { /* 4 */ obs = Integer.parseInt(args[i]); if (obs < mi1) /* 5 */ { mi2 = mi1; mi1 = obs; else if (obs < mi2) /* 6 */ mi2 = obs; System.out.println("The two smallest are " + mi1 + " and " + mi2); Sestoft, 18. november 200202100+02115+02199+02312 Introdutory Programming Page test-10 Sestoft, 18. november 200202100+02115+02199+02312 Introdutory Programming Page test-12
Table of test ases Choie Input data set Input property 1 1 B t least one number 2 B Exatly one number 2 C t least two numbers 3 C Seond number first number 3 D Seond number first number 4 zero times D Exatly two numbers 4 one E Exatly three numbers 4 more than one H t least four numbers 5 E Third number urrent minimum 5 F Third number 6 F Third number urrent minimum urrent minimum and 6 G Third number minimum and urrent seond least seond least Correted program publi stati void main (String[] args) { int mi1 = 0, mi2 = 0; System.out.println(""); mi1 = Integer.parseInt(args[0]); if (args.length == 1) /* 2 */ System.out.println("Smallest = " + mi1); int obs = Integer.parseInt(args[1]); mi2 = obs; if (obs < mi1) /* 3 */ { mi2 = mi1; mi1 = obs; for (int i = 2; i < args.length; i = i+1) { /* 4 */ obs = Integer.parseInt(args[i]); if (obs < mi1) /* 5 */ { mi2 = mi1; mi1 = obs; else if (obs < mi2) /* 6 */ mi2 = obs; System.out.println("The two smallest are " + mi1 + " and " + mi2); Sestoft, 18. november 200202100+02115+02199+02312 Introdutory Programming Page test-13 Sestoft, 18. november 200202100+02115+02199+02312 Introdutory Programming Page test-15 Table of input data sets Input data set Contents Expeted output B 17 Smallest = 17 C 27 29 The two smallest are 27 and 29 D 39 37 The two smallest are 37 and 39 E 49 48 47 The two smallest are 47 and 48 F 59 57 58 The two smallest are 57 and 58 G 67 68 69 The two smallest are 67 and 68 H 77 78 79 76 The two smallest are 76 and 77 Error! Input data set C produes wrong results: The two smallest are 27 and 0 The variable mi2 is not assigned a value before it is printed (in ase C). Funtional test: Does the program solve the problem? Goal: to see whether the program solves the given problem. Method: try to show that the program does not solve the problem. Prerequisites for funtional test 1. fairly preise desription of the problem. 2. Ideas of diffiult ases and wrong ways to solve the problem. 3. The expeted output data an by alulated or approximated without using the program. Designing a funtional test may reveal ambiguities in the desription of the problem. Designing a funtional test may be a good way to begin developing the program. It retains its initial value, 0. n appropriate assignment of mi2 is neessary. Sestoft, 18. november 200202100+02115+02199+02312 Introdutory Programming Page test-14 Sestoft, 18. november 200202100+02115+02199+02312 Introdutory Programming Page test-16
Table of input data sets and expeted output data Input data set Contents Expeted output Funtional test, example 1: find the smallest and the greatest number Given a (possibly empty) sequene of numbers, find the greatest and the smallest of these numbers. mbiguity: What should we do with an empty list of numbers? Clarifiation: We assume that an error message should be given. (What other sensible possibility is there?) B 17 Minimum = 17; maximum = 17 C1 27 27 Minimum = 27; maximum = 27 C2 35 36 Minimum = 35; maximum = 36 C3 46 45 Minimum = 45; maximum = 46 D1 53 55 57 Minimum = 53; maximum = 57 D2 67 65 63 Minimum = 63; maximum = 67 D3 73 77 75 Minimum = 73; maximum = 77 D4 89 83 85 Minimum = 83; maximum = 89 Sestoft, 18. november 200202100+02115+02199+02312 Introdutory Programming Page test-17 Sestoft, 18. november 200202100+02115+02199+02312 Introdutory Programming Page test-19 Table of input data properties Input data set B C1 C2 C3 D1 D2 D3 D4 Input property One number Two numbers, equal Two numbers, inreasing Two numbers, dereasing Three numbers, inreasing Three numbers, dereasing Three numbers, greatest in the middle Three numbers, smallest in the middle Funtional test, example 2: Find the greatest differene Given a (possibly empty) sequene of numbers, find the greatest differene between two onseutive numbers. mbiguity: What should we do with a sequene ontaining zero or one number? Clarifiation: We assume that error messages should be given and Only one number, respetively mbiguity: Greatest signed differene or unsigned differene? Clarifiation: We assume that is should be the numeri differene (i.e. unsigned differene). Sestoft, 18. november 200202100+02115+02199+02312 Introdutory Programming Page test-18 Sestoft, 18. november 200202100+02115+02199+02312 Introdutory Programming Page test-20
Table of input data properties Strutural versus funtional test Strutural and funtional test often use the same input data set. Table of input data sets and expeted output data dvantages of strutural test Input data set Input property Input data set Contents Expeted output Mehani, demands a systemati approah but not a deep understanding of the problem. Finds logial errors in the program. B One number B 17 Only one number Gives the person doing the testing (or the programmer) an opportunity to study the program in detail. C1 Two numbers, equal C1 27 27 0 May lead to modifying the program and as a result a better program. C2 Two numbers, inreasing C2 36 37 1 Covers all the details of the solution to the problem. C3 Two numbers, dereasing C3 48 46 2 D1 Three numbers, inreasing differene D1 57 56 59 3 dvantages of funtional test D2 Three numbers, dereasing differene D2 69 65 67 4 Independent of the program. Need not be hanged when the program is hanged (but when the problem is hanged). Gives the person doing the testing an opportunity to study the (desription of the) problem in detail. May lead to modifying the desription of the problem, making it more preise. Sestoft, 18. november 200202100+02115+02199+02312 Introdutory Programming Page test-21 Sestoft, 18. november 200202100+02115+02199+02312 Introdutory Programming Page test-23 Given a day of the month Funtional test, example 3: Legal dates and a month non-leap year. The day and the month are given as integers. Examples: 31/12 and 31/8 are legal dates, whereas 29/2 and 1/13 are not., deide whether they determine a legal data in a Input data set Contents Expeted output 0 1 Illegal 1 0 Illegal 1 1 Legal 31 1 Legal 32 1 Illegal 28 2 Legal 29 2 Illegal 31 3 Legal 32 3 Illegal 30 4 Legal 31 4 Illegal 31 5 Legal 32 5 Illegal 30 6 Legal 31 6 Illegal 31 7 Legal 32 7 Illegal 31 8 Legal 32 8 Illegal 30 9 Legal 31 9 Illegal 31 10 Legal 32 10 Illegal 30 11 Legal 31 11 Illegal 31 12 Legal 32 12 Illegal 1 13 Illegal Pratial hints about rerunning tests Rerun a test after eah modifiation or improvement of the program. Save the input data sets so they easily an be rerun, e.g. in a file testminmax.bat under Windows, e.g.: java Minmax >> testminmax.res java Minmax 17 >> testminmax.res java Minmax 27 29 >> testminmax.res java Minmax 39 37 >> testminmax.res or in a shell sript testminmax under Linux/Unix, e.g.: #! /bin/sh java Minmax >> testminmax.res java Minmax 17 >> testminmax.res java Minmax 27 29 >> testminmax.res java Minmax 39 37 >> testminmax.res The file must be on your path, and for Linux/Unix you should give it exeution status with the ommand: hmod a+x testminmax Running suh a sript will ause output data to be saved in a textfile testminmax.res. Before running the test, write expeted results in a file testminmax.exp. The results an be ompared automatially with expeted results in testminmax.exp (using diff under Unix/Linux or f under MS DOS). Sestoft, 18. november 200202100+02115+02199+02312 Introdutory Programming Page test-22 Sestoft, 18. november 200202100+02115+02199+02312 Introdutory Programming Page test-24
Testing (non main) methods of a lass So far we have onsidered how to test a main method taking arguments on the ommand line. For a non main method, the priniples for making a test suite (presented in two tables) are the same as for main methods. However, to exeute the test, you must write a speial test lass that has a main method that invokes the method with eah input data set of the test suite. The main method an either provide you with the results of the invokations (so that you an ompare them with expeted output) or even better make the omparison for you. Testing graphial user interfaes Testing graphial user interfaes (with windows and a mouse) is umbersome: One must desribe arefully step by step what ations the test person must perform, and what the program s expeted reations are. Exeuting the test must be done manually; it annot be rerun automatially. Sestoft, 18. november 200202100+02115+02199+02312 Introdutory Programming Page test-25 Sestoft, 18. november 200202100+02115+02199+02312 Introdutory Programming Page test-27 Sample struture of a test lass lass X { stati ResType m(rgtype x) {... //method to be tested publi lass XTest { // test lass publi stati void main (String[] args) { //test the data sets of the test suite test(input1, expetedoutput1);... test(inputn, expetedoutputn); private stati void test(rgtype a, ResType expeted) { ResType res = X.m(a); Test in perspetive Testing an improve our onfidene in a program, but it an never prove that a program has no errors. The saying of the statistiian is relevant: bsene of evidene isn t evidene of absene! The tester thinks that the test is suessful if it does find errors. The programmer thinks that the test is suessful if it does not find errors. When tester and programmer is the same person, there is a psyhologial onflit. It takes time to design a test. This motivates for avoiding superfluous hoie- and repetition statements (simplifies the strutural test); avoiding superfluous speial ases in the desription of the problem (simplifies the funtional test). Programs must be tested if ( res differs from expeted ) System.out.println("m(" + a + ") returns " + res + " -- differs from expeted " + expeted); if errors an damage humans or animals; if errors an lead to onsiderable eonomi losses; if they are used to draw sientifi onlusions. It is out of sope of this ourse to desribe all aspets of testing. Sestoft, 18. november 200202100+02115+02199+02312 Introdutory Programming Page test-26 Sestoft, 18. november 200202100+02115+02199+02312 Introdutory Programming Page test-28