Chapter 2.6: Testing and running a solution 2.6 (a) Types of Programming Errors When programs are being written it is not surprising that mistakes are made, after all they are very complicated. There are three types of error which can be made when program code is being produced. 1. Syntax errors. These are errors in the use of the language. The rules have been broken. A command word might be PRINT. If it was typed in as PLINT the translator program would not recognise it and so would not be able to translate it. This is not a difficult error to spot because when the programmer tells the translator to translate it, a message will be returned saying that a mistake has been made and telling the programmer which word it cannot recognise. There are other ways of breaking the rules, for instance a language might be set up to only accept a mathematical formula if it is in the form of a variable followed by an equals sign and then a collection of variables. So X = 3*(2+A) would be accepted by the translator, but 3*(2+A) = X would be rejected as an error even though, mathematically, it is identical. The following code contains many different syntax errors: DIM X no data-type has been stated X=1 IF X > 1 no THEN statement Y=(2*X+3 no closing bracket PLINT Y misspelling of a keyword NEXT NEXT statement without a FOR (Should have been END IF) 2. Logic errors (Run Time). A logic error is a mistake in the way the program solution has been designed. For example, an instruction in a program may tell the computer to jump to another part of the program. This is fine as long as the instruction tells the computer to go to the right part of the program. If it tells the computer to go to the wrong place, it is highly unlikely that the program will produce the results that were expected. Unfortunately, such a mistake does not usually make the program stop working; it just makes it produce wrong results. This makes it difficult to spot that a mistake has been made, and even when the programmer realises that something has gone wrong, finding where the error is can be very difficult. Another example of a typical logic error is when an incorrect formula is used e.g. Total_number = Previous_total New_number. There is nothing obviously wrong with the formula so it will work, but it won t produce the right answer because it should have been +. Notice that this is not an arithmetic error, the computer will do the arithmetic perfectly, and it is the logic that is wrong. Page 1 of 8
3. Arithmetic error. This is sometimes thought of as a type of logic error but is rather different because the computer is asked to do something that is impossible. A good example is when a computer is asked to divide by 0, this is impossible and would cause the program to fail. The following code will produce an arithmetic error on the fourth line because X was initialised to zero and the first iteration will attempt to divide by X when it is this initial value: X=0 X is initialised to zero Y=1 FOR i=1 TO 10 Y=Y/X attempted division by zero on first iteration X=X+i NEXT Page 2 of 8
2.6 (b) Testing Methods When a system is designed it is important that some consideration is given to making sure that no mistakes have been made. A schedule should be drawn up which contains a test for every type of input that could be made and methods of testing that the program actually does what it was meant to do. This schedule is known as the test plan. Note that it is produced before the system is produced. There are a number of ways of testing a program. 1. Black box: Different values can be input for variables to determine whether the program can cope with them. These values should include typical values, borderline values and values which are not acceptable. For example, if a program is written which uses marks out of 100 from a maths examination as input, the test data would include typical data like 27, 73.., borderline data which would be 0 and 100, and unacceptable data like 34, 123, 16.345 This type of testing is called black box testing. 2. White box testing is testing the program to determine whether all the possible paths through the program produce the desired results. As a large program can have a very large number of routes, when you take into account the different condition statements and loops, white box testing is rarely carried out exhaustively. Think of black box as a test where you cannot see into the box (program) all you see is what comes out at the end. White box testing means that you are able to see what is happening as the data goes through the box because it is transparent. 3. Alpha and Beta testing. When you have written a program and you sit down to test it, you have a certain advantage because you know what to expect. After all, you wrote the program. This can be extended to the whole software company, as the employees are all computer-minded people. Testing carried out by people like this is known as alpha testing. Eventually, the company will want ordinary users to test the program because they are likely to find errors that the software specialists did not find. Testing carried out by the users of the program is called beta testing. If you continue the course next year, in order to turn your AS grade into an A level, you will have to write a project. This involves using a computer to solve a problem for someone. When you have finished your project you will be expected to test whether or not it works, this is alpha testing. You will also get marks if you persuade the person whose problem you are solving to test it for you, this is the beta testing. 4. User Acceptance Testing is often the final step before rolling out the application. Usually the end users who will be using the applications test the application before accepting the application. This type of testing gives the end users the confidence that the application being delivered to them meets their requirements. This testing also helps nail bugs related to usability of the application. Page 3 of 8
2.6 (c) Selection of test data If a solution is to be tested, someone has to choose what data is going to be used to do the testing. The test data is usually chosen by the programmer because they know what they want to test. It is important to test as many different things as possible, the important word there being different. The problem for the programmer is that they know what inputs were expected so they find it very difficult to think of the sort of inputs that the user of the program might try to put in. When you are asked to think up different inputs to test a program, it must be different types of input, not just changing numbers. Imagine a question that states that a program has been written that will work out the mean of three numbers. You have to come up with different test data and the reasons for doing those tests. That last bit is in bold because that is what you get the marks for and the reasons for the tests are the things that have to be different. In this example you would me thinking of 1, 2, 3 to test whether integers will give an integer answer 1, 2, 4 to test whether the software can cope with a recurring decimal answer (Note that 1, 2, 4 to test a different set of integers would not get a mark because the reason for the test is not different) 1, 2.5, 3 to test whether the program can use decimal inputs 1, 2½, 3 to test whether fractions are allowed -1, -2, -3 to test whether negative numbers can be handled 1, 2 to test what happens when only two values are input There are many more that would be acceptable. The important thing to notice is that the numbers themselves are almost identical but that the reasons for choosing them are very different. Also pay attention to NORMAL, ABNORMAL and BORDERLINE data. Example: Software calculates the cost of parking at a council car park as follows: Free from 5am to 8am 3 per hour or part of an hour between 8am and 5pm Free from 5pm to 12 midnight No parking from 12 midnight to 5am Solution: Suitable test data will include: data within each of the valid parking periods; data that crosses more than one of the valid parking periods data at the boundaries of the valid parking periods; data not within the valid parking periods. The test data should be presented in a table similar to the following: Page 4 of 8
Example: A procedure has been written which will accept any two numbers and output the larger. Give different sets of data, which could be used to test the procedure. Solution: Page 5 of 8
2.6 (d) Trace tables and dry run Students should do class practices with related past paper questions to learn dry running/desk checking an algorithm with the help of their teachers. Page 6 of 8
2.6 (e) Debugging Errors in computer solutions are called bugs. They create two problems. One is that the error needs to be corrected; this is normally fairly straightforward because most errors are caused by silly mistakes. The second problem, however, is much more complicated, the errors have to be found before they can be corrected. Finding where the error is and identifying it, can be very difficult and there are a number of techniques available for solving such problems. 1. Translator diagnostics. Each of the commands that are in the original program is looked at separately by the computer translator to execute it. Each command will have a special word which says what sort of command it is. The translator looks at the special word in the command and then goes to its dictionary to look it up. The dictionary tells the translator program what the rules are for that particular special word. If the word has been typed in wrongly, the translator will not be able to find it in the dictionary and will know that something is wrong. If the word is there, but the rules governing how it should be used have not been followed properly, the translator will know that there is something wrong. Either way, the translator program knows that a mistake has been made, it knows where the mistake is and, often, it also knows what mistake has been made. A message detailing all this can be sent to the programmer to give hints as to what to do. These messages are called translator diagnostics. 2. Debugging tools. Sometimes the program looks alright to the translator, but it still doesn t work properly. Debugging tools are part of the software which help the user to identify where the errors are. The techniques available include: a) Cross-referencing. This software checks the program that has been written and finds places where particular variables have been used. This lets the programmer check to make sure that the same variable has not been used twice for different things. b) Traces. A trace is where the program is run and the values of all the relevant variables are printed out, as are the individual instructions, as each instruction is executed. In this way, the values can be checked to see where they suddenly change or take on an unexpected value. c) Variable dumps (check/watch). At specified parts of the program, the values of all the variables are displayed to enable the user to compare them with the expected results. 3. Desk checking is sometimes known as a dry run. The user works through the program instructions manually, keeping track of the values of the variables. Most computer programs require a very large number of instructions to be carried out, so it is usual to only dry run small segments of code that the programmer suspects of harbouring an error. 4. In section 2.1.c we discussed the splitting up of a problem into smaller and smaller parts, until each part was a manageable size. When the parts were combined they would produce a solution to the original problem. Because we are starting with a big problem and splitting it into smaller problems, this was called top-down design. When the program is written, each small program is written separately, allowing the small programs to be tested thoroughly before being combined. This is called bottom-up programming. The technique is particularly useful for testing the Page 7 of 8
finished program because it is far easier to test a lot of small programs than it is to test one large one. One problem can arise, because the small programs have to be joined together, these joints have to be tested too to make sure that no silly mistakes have been made like using the same variable name for two different things in two parts of the program (tested by cross referencing). 5. Test strategies are important to establish before the start of testing to ensure that all the elements of a solution are tested, and that unnecessary duplication of tests is avoided. 6. (This section is with respect to VB6) If you reach a point in your code that calls another procedure (a function, subroutine, or the script associated with an object or applet), you can enter (step into) the procedure or run (step over) it and stop at the next line. At any point, you can jump to the end (step out) of the current procedure and carry on with the rest of the application. You may want to step through your code and trace code execution because it may not always be obvious which statement is executed first. Use these techniques to trace the execution of code: Break points can be set within program code so that the program stops temporarily to check that it is operating correctly to that point. Step Into: Traces through each line of code and steps into procedures. This allows you to view the effect of each statement on variables. Step Over: Executes each procedure as if it were a single statement. Use this instead of Step Into to step across procedure calls rather than into the called procedure. Step Out: Executes all remaining code in a procedure as if it were a single statement, and exits to the next statement in the procedure that caused the procedure to be called initially. Run To Cursor: Allows you to select a statement in your code where you want execution to stop. This allows you to "step over" sections of code (for example, large loops). To trace execution from the current statement From the Debug menu, choose Step Into, Step Over, Step Out, or Run To Cursor. To trace execution from the beginning of the program From the Debug menu, choose Step Into, Step Over, Step Out, or Run To Cursor. Page 8 of 8