Appendix: Instruments

Transcription

1 Appendix A Appendix: Instruments This appendix presents the instruments used in the experiment. These instruments can be used verbatim to repeat the experiment; a electronic copy is available upon request. These instruments include the hard-copy directions, pictures o the process-centered system that the subjects used, explanations o the test techniques, o-line and on-line question sheets, proram speciications, and source code. All instruments were developed or native German speakers. This appendix oers Enlish versions o all documents. All translations are as true to the oriinal as possible. All documents with the exception o the prorams and their outputs have been translated. The source code is presented without chanes, which may make it diicult or people who do not know German to isolate typoraphical (cosmetic) aults. 133

2 A.1 Script used to introduce subjects to MVP-S 1. General (a) Two test exercises, or a total o about 3 hours. (b) The oal is to ind as many ailures as possible in as little time as possible. (c) Writin on all sheets is naturally OK. (d) Time entries should relect the total time needed. (e) Your ID is ::: () Please read the instructions and perorm the steps. More about the on-line tools 2. Menu bar: (a) One button per tool (b) Quit when inished 3. MVP-S: (a) Explain main window i. Column 1: Instance name o a process ii. Column 2: Model name o a process iii. Column 3: Status o a process iv. Menu item Action/Send Event to inorm the machine o a status chane. v. send start or the preparation step. vi. Menu item Ino/Context to et the instructions or each process (b) Explain context window i. Comments constitute the directions or each process ii. Entry and exit criteria are also displayed in the window. 4. Question sheet: (a) Help or each question appears at the bottom. (b) Bu: data are not read in aain. (c) Data are retrieved later. 134

3 A.2 Instructions or unctional testin as used o-line In this exercise you will test a proram named cmdline. Your oal is to ind as many ailures as possible in as little time as possible. You will enerate and execute test cases with the help o the speciication. Then you will search or ailures. Because isolatin aults is not part o this exercise, you will not receive the source code or the proram. Guideline: In the past, similar exercises required on averae circa 60 minutes. For this exercise you will need the ollowin materials in addition to these intructions: the question sheet Functional testin, the instruction sheet Buildin equivalence classes, and the speciication o the proram cmdline. At the end you will receive a sample solution. Step 1: Preparation Read over the question sheet Functional testin. Enter your identiier and answer the irst two questions. I you are unamiliar with equivalence classes, read throuh the instructions or developin and usin equivalence classes. Lo on to a computer. Create a new directory or your work, and chane into the new directory with the cd command. Fetch the necessary iles by enterin the ollowin command: tar x lott/expt/t-cmdline.tar Thereater the ollowin iles must be present: Makeile cmdline run-suite test-dir Step 2: Generate test cases You received the speciication or the proram with your materials. Read throuh the speciication careully. Use it to derive the equivalence classes. Guideline: In the past, on averae 16 validand4invalidequivalenceclassesweredevelopedorsimilarcomponents. Generate test cases by choosin boundary values rom the equivalence classes. A test case consists o an input and an expected output. Guideline: In the past, on averae 20 test cases were enerated or similar components. When you are inished, enter in the question sheet the number o valid and invalid equivalence classes, the number o test cases, and the amount o time you needed. I you also typed in your test cases while developin them, please try to separate your entries or total time between the eneration and the typin-in steps. Step 3: Type in the test cases In this exercise, a test driver is used to apply the test cases to the proram. The test driver reads parameter iles and invokes the proram with the parameters speciied in those iles. Example: i the proram cmdline is supposed to be invoked with cmdline 1 2 3, than the expression should be placed in the parameter ile. 135

4 You must create the parameter iles. A test case = a parameter ile. The expressions in a parameter ile speciy one invocation o the proram or a one run o the test driver. Parameter iles are named with.test as the ile suix. The results are written into a ile. A complete test o the component is accomplished in this way. Type in your test cases in iles under the directory test-dir. Follow the speciication o the test driver while doin so. When you are inished, enter the amount o time you needed in the question sheet. Step 4: Run test cases Apply the test cases to the component by typin in the command run-suite. I you made some typin mistakes, please correct the mistakes now, type make clean, and repeat the tests by enterin the command run-suite aain. The results are in individual iles under the directory test-dir and are also summarized in a ile test-results.summary. When you are inished with this step, enter the amount o time you needed in the question sheet. Print out the test results rom the ile test-results.summary. Step 5: Search or ailures Look over the results careully. Find possible ailures by comparin the expected results accordin to the speciication with the output o your test cases. Mark the detected ailures in both copies o the outputwith circles, etc. Guideline: In the past, on averae 8 ailures were detected in similar components. When you believe that you have detected all ailures, please enter the time you required in the question sheet. Complete the rest o the question sheet and hand it in with your test results. 136

5 A.3 Instructions or unctional testin as used on-line For this exercise you will need the ollowin materials in addition to these intructions: the instruction sheet Buildin equivalence classes, and the speciication o the proram cmdline. At the end you will receive a sample solution. 1. Lo on to a computer. Start the menu bar or the inormation system by enterin the ollowin command: lott/expt/menu-t 2. Use the menu bar to start one instance o the inormation system MVP-S. Select the project Funktionales Testen, wherein you take on the role Test Inenieur. 3. Use the menu bar aain to start one instance o the question sheet tool. 4. Please use the inormation system or the rest o the inormation about the exercise. 137

6 Fiures A.1, A.2 and A.3 show the initial menu bar, the MVP-S work context window and the process context windows, respectively. These interaces support the subjects who use on-line directions to perorm unctional testin. Fiure A.1: Menu bar to start MVP-S and a question sheet tool Fiure A.2: MVP-S work-context window that supports unctional testers 138

7 Fiure A.3: MVP-S process-context window or unctional testin step schritt 1 139

8 A.4 Question sheet unctional testin as used o-line Your ID: Date: Note: For all time-related entries, please deduct time or pauses, etc. 1. How motivated are you or the experiment? [Please mark the scale.] Estimate Value Comparison not at little so-so pretty quite very all much abit much 2. When did you start the experiment? : [Hour:Minute] 3. How many equivalence classes did you derive? Valid: [Count], Invalid: [Count] 4. How many test cases did you enerate? [Count] 5. How much time did you need to construct the test cases (step 2)? [Minutes] 6. How much time did you need to type in the test cases (step 3)? [Minutes] 7. How lon did the execution o test cases (step 4) require, i more than zero? [Minutes] 8. How lon did you search or ailures (step 5)? [Minutes] 9. How many dierent ailures did you ind? [Count] 10. When did you end the experiment? : [Hour:Minute] 11. Would you be inclined to perorm another iteration in order to ind more ailures? [yes, no] 12. How well do you eel you ve mastered unctional testin? [Please mark the scale] Estimate Value Comparison not at little so-so pretty quite very all much abit much 140

9 A.5 Question sheet unctional testin as used on-line Fiure A.4 shows the on-line question sheet used by subjects who use the process-centered system to perorm unctional testin. Fiure A.4: On-line question sheet used by unctional testers 141

10 A.6 Buildin equivalence classes Functional testin based on equivalence classes requires the input to be divided into equivalence classes rom which similar behavior is expected. The ollowin overview ives heuristics or such a division [Reerence: G. Myers, The Art o Sotware Testin, 1979.] 1. I an input condition speciies one reion, identiy one valid and two invalid equivalence classes. Example: For input condition the number A to be entered varies rom 1 to 999, deine the valid equivalence class 1 A 999, and the invalid equivalence classes A < 1, 999 < A. 2. I an input condition speciies a number o values, identiy one valid and two invalid equivalence classes. Example: For input condition the number o search expressions is minimum 1 and maximum 6, deine the valid equivalence class 1 number 6 and the invalid equivalence classes number = 0, number > I an input condition speciies a set, and there are rounds or belie that the set elements are treated dierently, determine one valid equivalence class or each element in the set, plus one additional invalid class or an element that is not contained in the set. Example: For input condition the input is o type ender, we assume dierent treatment o the ender values. Choose valid equivalence classes male, emale, neuter and the invalid one martian. 4. I an input condition speciies a must situation, identiy one valid and one invalid equivalence class. Example: For input condition the irst character in the word must be a letter, choose the valid equivalence class w j w is a word and the irst character is a letter and the invalid equivalence class w j w is a word and the irst character is not a letter. 5. I there are rounds or belie that elements in an equivalence class are treated dierently, divide the equivalence class appropriately into new ones. Example: even values, odd values. Once the equivalence classes have been determined, test data must be selected. A procedure named boundary value analysis speciies which data are chosen rom the equivalence classes. Instead o choosin just any element rom an equivalence class, values are used that are near or on the boundaries o the equivalence classes. Instead o usin only the input conditions, test cases are also determined by observin the result space. One can work accordin to the ollowin schema: 1. Assin each equivalence class a unique number. 2. Until all valid equivalence classes are covered, write one test that covers as many as possible o the valid, not-yet-covered equivalence classes. 3. For each invalid equivalence class, write one test that covers this and only this one. 142

11 A.7 Speciication o proram cmdline Name cmdline Syntactic and semantic analysis o a command line Usae cmdline hile cmdline mass <measure> [Search-option] File [ File.. ] Description cmdline analyzes a command line o a measurement tool or syntactic and partially or semantic correctness. The units that are eventually supposed to be measured are stored in iles. However, the measurement unctionality is not part o this tool. The current version o cmdline only evaluates the command-line parameters. The user must ive the proram at a minimum one measure and one ile as arument. In addition to the aruments (measure, ile, etc.), exactly one so-called search option may be iven. The order o the measurement options and search options is not important. When the command line is evaluated, all aruments beinnin with the irst non-option are treated as ile names. The existence o such iles is not checked. I successul (i.e., i the syntax and meanin o the aruments is ound to be leal), a summary o the aruments is printed out. Otherwise, an explanatory error messae is printed. The summary consists o the measure to be computed, the search options i any were iven (includin a value or -unter and -ueber), and a list o the iles to be read. Options Options and aruments or options can be abbreviated to their unique preixes. In the ollowin text, the unique preix o each option appears beore the square brackets. A maximum o one search option may be used. h[ile] Help option. A helpul text is printed and nothin else is done.? Help option. See above. mas[s] <measure> Valid measures are GKO[M], LKO[M], GKH[M], LKH[M], GI[HE], and LI[HE]. a[lle] Search option. The proram should evaluate the measure or all units. max Search option. The proram should determine the maximum value o the measure or all units. 143

12 mi[n] Search option. The proram should determine the minimum value o the measure or all units. du[rchschnitt] Search option. The proram should determine the averae value o the measure or all units. un[ter] <value > Search option. The proram should identiy all units or which the measure lies under the boundary value. The <value> may be any real number. ue[ber] <value > Search option. The proram should identiy all units or which the measure lies over the boundary value. The <value> may be any real number. Example % cmdline -mass GKHM -alle datei1 Die Auabe ist: Mass: GKHM Suche: -alle Anzahl der zu lesenden Dateien: 1 Die Dateien sind: datei1 Authors Baumärtner, Claßen, Gieseke, Lott. 144

13 A.8 Source code o proram cmdline cmdline.h: #inde CMDLINEDOTH #deine CMDLINEDOTH #deine HILFE 0 #deine MASS 1 #deine GKOM 2 #deine LKOM 3 #deine LKHM 4 #deine GKHM 5 #deine LIHE 6 #deine GIHE 7 #deine ALLE 8 #deine MAX 9 #deine DURCH 10 #deine MIN 11 #deine UNTER 12 #deine UEBER 13 struct keyword entry char keyword; int min len; int id; ; struct auabe double zahl; char mass; char such; ; #endi cmdline.c: #include <stdio.h> #include <strin.h> #include <ctype.h> extern double ato(); #include "cmdline.h" static struct keyword entry keyword table[] = "-?", 2, HILFE, "-hile", 2, HILFE, "-mass", 4, MASS, "GKOM", 3, GKOM, "LKOM", 3, LKOM, "LKHM", 3, LKHM, "GKHM", 3, GKHM, "LIHE", 2, LIHE, "GIHE", 2, GIHE, "-alle", 2, ALLE, "-max", 4, MAX, 145

14 "-durchschnitt", 3, DURCH, "-min", 3, MIN, "-unter", 3, UNTER, "-ueber", 3, UEBER, ; static int keyword table size = sizeo(keyword table) = sizeo(struct keyword entry), 1; void usae(pron) char pron; print(stderr, "Verwendun: %s -mass <MASS> [ Suchption ] Datei [ Datei... ]\n", pron); exit(1); static int is str keyword(ar, valid ar, min verl len) char ar, valid ar; int min verl len; int arlen, rc; rc=0; arlen = strlen(ar); i (strncmp(ar, valid ar, arlen) == 0) rc=1; return rc; static int code strin(str) char str; int i, rc; rc =,1; or (i = 0; i < keyword table size; ++i) i (is str keyword(str, keyword table[i].keyword, keyword table[i].min len)) rc=keywordtable[i].id; return rc; void print auabe(au) struct auabe au; i ( (au,>mass!= NULL && strcmp(au,>mass, "-hile") == 0) jj (au,>mass == NULL && au,>such == NULL)) usae("cmdline"); return; 146

15 print("die Auabe ist:\n"); i (au,>mass) print("mass: %s\n", au,>mass); i (au,>such) print("suche: %s\n", au,>such); i (strcmp(au,>such, "-unter") ==0 jj strcmp(au,>such, "-ueber") ==0) print("zahl: %.1\n", au,>zahl); void print dateien(ari, arc, arv) int ari, arc; char arv; int i; i (arc == ari) print("es ehlt eine Datei\n"); else print("anzahl der zu lesenden Dateien: %d\n", arc, ari); print("die Dateien sind:\n"); or(i=ari;i < arc; ++i) print(" %s\n", arv[i]); int process switches(arc, arv, au) int arc; char arv; struct auabe au; int i, rc = 0, position = 0; au,>zahl = 0.0; au,>mass = NULL; au,>such = NULL; or (i = 1; i < arc; ++i) switch (code strin(arv[i])) case HILFE: au,>mass = "-hile"; case MASS: i (au,>mass!= NULL) print(stderr, "Es sind zuviele Masse aneeben\n"); rc =,1; else i (i++ >= arc) print(stderr, "Es ist kein Mass aneeben\n"); rc =,1; 147

16 else switch (code strin(arv[i])) case GKOM: au,>mass = "GKOM"; case LKOM: au,>mass = "LKOM"; case LKHM: au,>mass = "LKOM"; case GKHM: au,>mass = "GKHM"; case LIHE: au,>mass = "LIHE"; case GIHE: au,>mass = "GIHE"; deault: print(stderr, "Es ist kein uelties Mass aneeben\n"); rc =,1; case ALLE: i (au,>such!= NULL) print(stderr, "Es sind zuviele Suchoptionen aneeben\n"); rc =,1; else au,>such = "-alle"; case MAX: i (au,>such!= NULL) print(stderr, "Es sind zuviele Suchoptionen aneeben\n"); rc =,1; else au,>such = "-max"; case DURCH: i (au,>such!= NULL) print(stderr, "Es sind zuviele Suchoptionen aneeben\n"); rc =,1; else au,>such = "-durchschnitt"; 148

17 case MIN: i (au,>such!= NULL) print(stderr, "Es sind zuviele Suchoptionen aneeben\n"); rc =,1; else au,>such = "-min"; case UNTER: i (au,>such!= NULL) print(stderr, "Es sind zuviele Suchoptionen aneeben\n"); rc =,1; else au,>such = "-unter"; ++i; i (i >= arc jj!isdiit(arv[i])) print(stderr, "Es ist keine Zahl aneeben\n"); rc =,1; else au,>zahl = atoi(arv[i]); case UEBER: i (au,>such!= NULL) print(stderr, "Es sind zuviele Sucharumente aneeben\n"); rc =,1; else au,>such = "-ueber"; ++i; i (i >= arc jj!isdiit(arv[i])) print(stderr, "Es ist keine Zahl aneeben\n"); rc =,1; else au,>zahl = ato(arv[i]); deault: position = i; i (rc < 0 jj position == i) i (rc < 0) 149

18 print(stderr, "Die Optionen bzw. Arumente sind ehlerhat\n"); else print auabe(au); print dateien(position, arc, arv); return rc; = Hier beinnt die Testumebun. Bitte die Testumebun nicht testen, keine Abstraktionen bilden etc. = int main(arc, arv) int arc; char arv; struct auabe a; int rc; i (arc == 1) usae(arv); rc =,1; else rc=processswitches(arc, arv, &a); return rc; 150

19 A.9 Sample solution or unctional testin The component cmdline exhibits the ollowin ailures. 1. The option ueber is not reconized. 2. A misspelled word ( Suchption, should be Suchoption ) appears in the output. 3. Abbreviated options are enerally interpreted as unique and associated with the irst matchin option. For example, m is reconized as mass althouh the unique preix ma was not seen. 4. Search options without an accompanyin measure are not treated as an error. 5. I the option -mass is used without a correspondin leal measure arument, no error messae is printed provided that a ile name (really any text) ollows. However, i no arument ollows the option, the proram crashes. 6. The option LKHM is not processed correctly; instead, LKOM is mistakenly reported. 7. I the -min option is iven, an error messae complainin about too many search options appears because the code or unter is also mistakenly executed. 8. Only the inteer part o a real number is used. 9. Neative numbers are not reconized as valid numbers. 10. I all options were reconized correctly, missin data iles are not detected. The command name and supplied options are mistakenly listed as iles to be read. The next pae lists the equivalence classes and test cases necessary to reveal the ailures described above. 151

20 Equivalence classes: Valid classes G 1 := exactly 1 measure G 2 := max. 1 Option G 3 := min. 1 ile G 4 := uniquely abbreviated option G 5 := over/under value real G 6 := Order measure, option G 7 := Order option, measure G 81 := Option -max G 82 := Option -min G 83 := Option -durch G 84 := Option -unter G 85 := Option -ueber G 86 := Option -alle G 87 := Option -hile -? G 91 := Maß -LKOM G 92 := Maß -GKOM G 93 := Maß -GKHM G 94 := Maß -LKHM G 95 := Maß -GIHE G 96 := Maß -LIHE Invalid classes U 1 := no measure, U 2 := more than 1 measure U 3 := more than 1 option U 4 := no ile U 5 := ambiuously abbreviated option U 6 := missin over/under value U 7 :=unknown measure Test cases: Nr. Covered classes Invocation o the proram Fail. nr. 1 U 1 cmdline -max Datei1 4 2 U 2 cmdline -mass LKOM -mass GKOM -max Datei1 3 U 3 cmdline -mass LKOM -alle -min Datei 7 4 U 4 cmdline -mass LKOM -durch 10 5 U 5 cmdline -mass LKOM -m Datei1 3 6 U 6 cmdline -mass GIHE -unter Datei1 7 U 7 cmdline -mass T Datei1 8 G 1 ;G 2 ;G 3 ;G 92 ;G 82 ;G 6 cmdline -mass GKOM -min Datei1 7 9 G 4 ;G 6 ;G 7 G 91 ;G 83 cmdline -du -mass LKO Datei1 10 G 93 ;G 85 ;G 5 cmdline -mass GKHM -ueber 4.2 Datei G 94 ;G 81 ;G 2 cmdline -mass LKHM -max Datei1 Datei G 95 ;G 84 ;G 6 cmdline -mass GIHE -unter 3.14 Datei G 95 ;G 84 ;G 6 cmdline -mass GIHE -unter -2 Datei G 96 ;G 86 cmdline -mass LIHE -alle Datei1 15 G 87 cmdline -hile 2 16 U 1 ;U 4 cmdline -mass 5 Note 1: Valid equivalence classes are only named the irst time that they are covered. Note 2: One obtains test case number 16 when one covers two invalid equivalence classes in a sinle test case. This contradicts the uidelines or usin equivalence classes, but is the only possibility or revealin ailure 5. This example shows that there is no perect method. 152

21 A.10 Instructions or structural testin as used o-line In this exercise you will test a proram named tokens. Your oal is to ind as many ailures as possible in as little time as possible. You will enerate and execute test cases with the help o the speciication and the source code. These test cases should lead to 100 percent coverae o branches and other code constructs. You will execute the test cases and then search or ailures. Isolatin aults is not part o this exercise. Guideline: In the past, similar exercises required on averae circa 70 minutes. For this exercise you will need the ollowin materials in addition to these intructions: the question sheet Structural testin, the instruction sheet Coverae criteria, and the speciication and source code or the proram tokens. At the end you will receive a sample solution. Step 1: Preparation Read over the question sheet Structural testin. Enter your identiier and answer the irst two questions. I you are unamiliar with code coverae criteria, read throuh the instructions or attainin branch coverae etc. Lo on to a computer. Create a new directory or your work, and chane into the new directory with the cd command. Fetch the necessary iles by enterin the ollowin command: tar x lott/expt/st-tokens.tar Thereater the ollowin iles must be present: Makeile ct-map run-suite test-dir tokens Step 2: Generate test cases You received the speciication and code or the proram with your materials. Read throuh both careully. (I you should notice possible aults in this or any o ollowin steps, mark them. However, please do not invest any time in a precise analysis.) Bein to enerate test data that will lead to 100% coverae o the ollowin criteria: branch coverae, loop coverae, multiple condition coverae, and relational operator coverae. A test case consists o an input and an expected output. Think careully in this step in order to save yoursel time later. Guideline: In the past, on averae 6 test cases were enerated or similar components. When you are inished with this step, enter the amount o time you required in the question sheet. Step 3: Type in the test cases In this exercise, a test driver is used to apply the test cases to the proram. The test driver reads parameter iles and invokes the proram with the parameters speciied in those iles (options, iles, etc.). I the proram requires input iles, these must be made available and mentioned in the parameter ile. Example: i the proram tokens is supposed to be 153

22 invoked with tokens < input, than the expression < input should be placed in the parameter ile. You must create not only the parameter iles but also the input iles. A test case = a parameter ile. The expressions in a parameter ile speciy one invocation o the proram or one run o the test driver. Parameter iles are named with.test as the ile suix. Input iles should be named sensibly, or example empty ; they may not, however, use the ile suix.test. The results are written into a ile. A complete test o the proram is accomplished in this way. Type in your test cases in iles under the directory test-dir. Follow the speciication o the test driver while doin so. When you are inished, enter the amount o time you needed in the question sheet. Step 4: Run test cases and enerate more Apply the test cases to the component by typin in the command run-suite. The results are in individual iles under the directory test-dir and are also summarized in a ile test-results.summary. Look at the summary o coverae with the help o the commands summary und report. For example: % summary test-dir/gctlog % report test-dir/gctlog Try to brin all coverae values up to 100% by eneratin urther tests, or convince yoursel that 100% coverae is not possible or some very ood reasons. Guideline: In the past, on averae 85% total coverae was attained or similar components. You can add new tests in directory test-dir. Ater you have chaned the test data, run the tests aain by issuin the ollowin sequence o commands: % make clean % run-suite % summary test-dir/gctlog % report test-dir/gctlog When you are inished with this step, enter the number o test cases, the coverae values attained, and the amount o time you needed in the question sheet. Print out the test results rom the ile test-results.summary. Step 5: Search or ailures Look over the results careully. Find possible ailures by comparin the expected results accordin to the speciication with the output o your test cases. Mark the detected ailures in both copies o the outputwith circles, etc. Guideline: In the past, on averae 6 ailures were detected in similar components. When you believe that you have detected all ailures, please enter the time you required in the question sheet. Complete the rest o the question sheet and hand it in with your test results. 154

23 A.11 Instructions or structural testin as used on-line For this exercise you will need the ollowin materials in addition to these intructions: the instruction sheet Coverae criteria, and the speciication and source code or the proram tokens. At the end you will receive a sample solution. 1. Lo on to a computer. Start the menu bar or the inormation system by enterin the ollowin command: lott/expt/menu-st 2. Use the menu bar to start one instance o the inormation system MVP-S. Select the project Strukturelles Testen, wherein you take on the role Test Inenieur. 3. Use the menu bar aain to start one instance o the question sheet tool. 4. Please use the inormation system or the rest o the inormation about the exercise. 155

24 Fiure A.5: MVP-S process-context window or structural testin step schritt 2 156

25 A.12 Question sheet structural testin as used o-line Your ID: Date: Note: For all time-related entries, please deduct time or pauses, etc. 1. How motivated are you or the experiment? [Please mark the scale.] Estimate Value Comparison not at little so-so pretty quite very all much abit much 2. When did you start the experiment? : [Hour:Minute] 3. How much time did you need to construct the test cases (step 2)? [Minutes] 4. How much time did you need to type in the test cases (step 3)? [Minutes] 5. How lon did the execution/eneration o test cases (step 4) require? [Minutes] 6. How many test cases did you inally enerate? [Count] 7. List the coverae values that you inally attained with your test cases: Binary branch [%], Switch [%], Loop [%], Operator Instrumentation [%], Summary o all Condition Types [%] 8. How lon did you search or ailures (step 5)? [Minutes] 9. How many dierent ailures did you ind? [Count] 10. When did you end the experiment? : [Hour:Minute] 11. Would you be inclined to perorm another iteration in order to ind more ailures? [yes, no] 12. Have you mastered unctional testin? [Please mark the scale] Estimate Value Comparison not at little so-so pretty quite very all much abit much 157

26 A.13 Question sheet structural testin as used on-line Fiure A.6 shows the on-line question sheet used by subjects who use the process-centered system to perorm structural testin. Fiure A.6: On-line question sheet used by structural testers 158

27 A.14 Explanation o coverae criteria Binary branch & switch instrumentation Fulillin the branch coverae criteria makes certain that each branch in a component was executed at least once. Branches in C prorams are created by i,?, or, while, anddo-while statements. Each o these statements creates two conditions (branches): the evaluation o the test expression must yield true once and alse once. In GCT terminoloy, this type o branch coverae is called binary branch instrumentation. Branches are also created by the switch-case construct. To test all branches, all case labels must be branched to at least once. This includes the deault label, even i it was not explicitly written in the source code. Each case label produces a sinle condition. In GCT terminoloy, this type o branch coverae is called switch instrumentation. Loop instrumentation A do-while loop creates two conditions or this criteria: one or executin the loop exactly one time, and one or executin the loop more than one time. The or and while loop add a third condition: the idea o executin the loop zero times; i.e., the test expression is alse when it is evaluated or the irst time. Multiple condition instrumentation Multiple conditions are expressions constructed usin the operators loical and (&&) and/or loical or ( ). To ulill these criteria or a two-part expression (i.e., an expression with exactly one loical operator), each side o the expression must be true once and alse once. In other words, a two-part multiple-condition expression creates our conditions: alse/alse, true/alse, alse/true, and true/true. Nested expressions create correspondinly more conditions. When checkin the coverae o multiple conditions, all expressions that evaluate to a boolean value are checked, not only the test expressions rom i statements. Relational operator instrumentation Relational operators are the operators that compare values with each other (e.., =,!=, <, <=, etc.). The relational operators (<, <=, >, >=) are oten the cause o aults, whether because o swapped operators (e.., < instead o >), incorrect use (e.., < instead o <=), or incorrect boundary values (e.., a < 99 instead o a < 100 ). A relational operator creates two (sometimes three) conditions: 1. For all operators: The let side must equal the riht side. 2. For all operators: The let side must not equal the riht side. 3. For comparisons o relative size, the ollowin boundary values should be tested: (a) For the operators <= und >: the let side must be 1 smaller than the riht side (b) For the operators >= und <: the let side must be 1 larer than the riht side Note: when you write your test cases, try to cover as much unctionality o the proram as possible with each case. This will help you avoid an unnecessarily lare number o test cases. 159

28 A.15 Speciication o proram tokens Name tokens sort and count alphanumeric tokens Usae tokens [ ai ] [ c chars ] [ m count ] Description tokens reads words rom the standard input ( stdin ). A word is a series o sequential characters rom a iven character set. This character set includes by deault the alphabetic characters A Z and a z as well as the diits 0 9. Dependin on the options (see below), the diits can be eliminated rom the character set, or other characters can be added to it. Characters that are not in the character set (e.., spaces) divide words rom each other, but are otherwise inored. At the end o the input, each word is printed exactly once with its requency o appearance in the input. Thereby the words are sorted in increasin lexicoraphic order. Options a : Allow only alphabetic characters in tokens (no diits 0 9). c chars : Allow chars to be part o tokens. i : Inore the dierence between upper and lower case by mappin all input to lower case. m count : The value count indicates the minimum requency needed or a word to be printed. Example To sort and count all alphabetical words in ile xyz : tokens -a < xyz Limits The maximum count o unique words is limited by the memory o the computer. The maximum requency count o a sinle word is limited by the lenth o an inteer. Author Gary Perlman 160

29 A.16 Source code o proram tokens The proram tokens is copyriht 1992 by Gary Perlman. Adapted with permission. =COPYRIGHT (c) Copyriht 1992 Gary Perlman = #include <stdio.h> #include <strin.h> #include <assert.h> int Inore = 0; int Mincount = 0; int Alpha = 0; char MapAllowed[256]; typede struct tnode char contents; int count; struct tnode let; struct tnode riht; TNODE; void treeprint(tree) TNODE tree; i (tree!= NULL) treeprint(tree,>let); i (tree,>count > Mincount) print("%7d\t%s\n", tree,>count, tree,>contents); treeprint(tree,>riht); TNODE install(strin, tree) char strin; TNODE tree; int cond; assert(strin!= NULL); i (tree == NULL) else i (tree = (TNODE ) calloc(1, sizeo(tnode))) tree,>contents = strdup(strin); tree,>count = 1; cond = strcmp(strin, tree,>contents); i (cond < 0) tree,>let = install(strin, tree,>let); else i (cond == 0) tree,>count++; else 161

30 return(tree); tree,>riht = install(strin, tree,>riht); char etword(ioptr) FILE ioptr; static char strin[1024]; char ptr = strin; reister int c; assert(ioptr!= NULL); or (;;) c = etc(ioptr); i (c == EOF) i (ptr == strin) return(null); else i (!MapAllowed[c]) i (ptr == strin) continue; else ptr++ = MapAllowed[c]; ptr = \0 ; return(strin); void tokens(ioptr) FILE ioptr; TNODE root = NULL; char s; assert(ioptr!= NULL); while (s = etword(ioptr)) root = install(s, root); treeprint(root); int main(arc, arv) int arc; char arv; int c, errcnt = 0; extern char optar; while ((c = etopt(arc, arv, "ac:im:"))!= EOF) switch(c) case a : Alpha=0; case c : while (optar) MapAllowed[optar] = optar; optar++; 162

31 case i : Inore = 1; case m : Mincount = atoi(optar); deault: errcnt++; i (errcnt) print(stderr, "Usae: %s [-i] [-c chars] [-m count]\n", arv); return(1); or (c = a ; c <= z ; c++) MapAllowed[c] = c; or (c = A ; c <= Z ; c++) MapAllowed[c] = Inore? c, A + a : c; i (!Alpha) or (c = 0 ; c <= 9 ; c++) MapAllowed[c] = c; tokens(stdin); return(0); 163

32 A.17 Sample solution or structural testin The component tokens has the ollowin aults and exhibits the ollowin ailures. 1. Fault in unction treeprint, line 25: The symbol > should be >=. Causes ailure: I a boundary value n is iven with the -m arument, the value n + 1is used instead o just n. 2. Fault in unction etword, line 64/77: The lenth o the input is not checked. Causes ailure: The proram dumps core i a very lon word is read. 3. Fault in unction main, line 97 (circa): The array MapAllowed is never initialized. Causes ailure: Compiler-dependent; non-alphanumeric symbols could be mistakenly accepted. 4. Fault in unction main, line 101: The variable Alpha is initialized with 0 instead o 1. Causes ailure: The arument -a has no eect. 5. Fault in unction main, line 110: The proram does not check whether a valid number was supplied. Causes ailure: Invalid numbers are accepted without an error messae. 6. Fault in unction main, line 115: The arument -a is not part o the usae messae. Causes ailure: The help messae says nothin about the -a arument. The ollowin test cases attain the maximum possible coverae values or the proram. 1. Test case: Unknown option Invocation with: -q Input: none Expected output: Error messae, usae messae or the proram. 2. Test case: No options, empty ile. Invocation with: Input: none. Expected output: < /dev/null none. 3. Test case: One option, empty ile. Invocation with: -c < /dev/null Input: none. Expected output: : none. 4. Test case: Few options, very short ile. Invocation with: -c / < shortile 164

33 Input: A ile named shortile with the sinle word nonl and no urther characters beore the end o the ile; i.e., no so-called newline. Expected output: The ollowin sinle line: 1 nonl 5. Test case: Many options, short ile. Invocation with: -a -i -c /. -m 1 < inile Input: A ile named inile with the ollowin entries: word1 word1 word1 word2 word2 b a Expected output: The ollowin lines: 1a 1b 3 word1 2 word2 End o the necessary test cases. 165

34 The ollowin coverae values result rom executin the test cases listed above. % summary test-dir/gctlog BINARY BRANCH INSTRUMENTATION (38 conditions total) 2 ( 5.26%) not satisied. 36 (94.74%) ully satisied. SWITCH INSTRUMENTATION (5 conditions total) 0 ( 0.00%) not satisied. 5 (100.00%) ully satisied. LOOP INSTRUMENTATION (21 conditions total) 7 (33.33%) not satisied. 14 (66.67%) ully satisied. OPERATOR INSTRUMENTATION (15 conditions total) 0 ( 0.00%) not satisied. 15 (100.00%) ully satisied. SUMMARY OF ALL CONDITION TYPES (79 total) 9 (11.39%) not satisied. 70 (88.61%) ully satisied. The values that are less than 100% can be explained urther usin the command report. That command yields the ollowin output: % report test-dir/gctlog "tokens.c", line 38: i was taken TRUE 5, FALSE 0 times. "tokens.c", line 64: loop zero times: 0, one time: 3, many times: 8. "tokens.c", line 118: loop zero times: 0, one time: 0, many times: 3. "tokens.c", line 120: loop zero times: 0, one time: 0, many times: 3. "tokens.c", line 122: i was taken TRUE 3, FALSE 0 times. "tokens.c", line 123: loop zero times: 0, one time: 0, many times: 3. Better coverae value cannot be obtained, as discussed below: [Line 38:] This i -condition tests whether calloc actually allocated memory. It is diicult to cause calloc to ail, especially or users who are not amiliar with the shell s limit command. [Line 64:] This is a loop with empty start and termination criteria. Thereore this loop is always executed at least once. [Lines 118, 120, and 123:] These or loops have hard-coded, ixed termination criteria, and are always executed more than once. [Line 122:] This i condition is always true due to a ault in the proram (the Alpha variable is not assined a value). 166

35 A.18 Observer s data sheet Subject s ID: Date: 1. How lon did the subject wait to report results, either on-line by manipulatin the eedback system or o-line by recordin inormation? 2. How well did the subject comprehend the process that s/he was perormin when asked? I.e., how well did the subject reply when asked about the inputs, outputs, and exit criteria o the process? 3. How lon did the subject wait to report data about a completed process? 4. How well did the subject conorm to the prescribed process? Was the process ollowed aithully? 5. Other comments. 167

36 A.19 Debriein questionnaire Your ID: Date: You have taken part in an experiment that compared the use o process inormation on paper with process inormation on the screen. Please enter your opinions about this experiment below. Naturally you can use the reverse side or additional sheets. 1. Was suicient inormation provided to perorm the exercises? 2. Did you eel that the instructions were too restrictive or the work? Please explain. 3. How helpul were the quantitative uidelines? How did you use them? 4. What did you like better, workin with instructions on paper or on the screen? Please justiy your opinion! 5. Would you preer enterin data on a conventional data-collection orm or on the screen? Why? 168

37 6. How understandable were the instructions? Were the instructions made more or less understandable by movin them onto the computer screen? 7. Would a raphical or other non-text version o the instructions have been better, or was the textual description best suited or the oal o understandin the exercise? 8. Other comments? 169

38 170