CHAPTER 7 SYSTEM TESTING. Testing is conducted to ensure that a system is production-ready (Loveland, 2005).

Transcription

1 CHAPTER 7 SYSTEM TESTING 7.1 OVERVIEW Testing is conducted to ensure that a system is production-ready (Loveland, 2005). Before testing was conducted on the PL-Analyser system, a test plan was prepared based on the Institute of Electrical and Electronics Engineers (IEEE) Standard 829 for Software Test Documentation (IEEE Standard for Software Test Documentation, 1998). Functional tests were carried out with the help of 30 users who were randonly picked. Broken link, performance and usability tests were carried out by three 2-member volunteer teams, using tools such as Microsoft Application Centre Test and Web Validator. This chapter describes the test cases, test approaches and results of testing. 7.2 TEST REFERENCE The PL-Analyser system was tested to meet the requirements documented in Chapter 5. The broken link, performance and usability tests were performed based on the architecture defined in Chapter FEATURES TESTED The PL-Analyser system was tested to ensure that the functions in the use cases are met by the system. The tests were conducted using a top-down approach, and covered the following functions: 106

2 i. Surfing information - this is to ensure that the PL-Analyser system allows users to access external sites on the Web using the provided links and to seek information. ii. Register - this is to ensure that the PL-Analyser system accurately registers user particulars and returns a user ID and password. iii. Take survey - this is to ensure that the PL-Analyser system allows users to participate in the surveys and that the system can capture input from the respondents, correctly. iv. View results - this is to ensure that the PL-Analyser system retrieves, calculates and displays the survey results correctly. v. Provide feedback - this is to ensure that the PL-Analyser system captures feedback provided by the users. The PL-Analyser system was also tested to ensure that the system has no broken links, all the spelling is correct and the texts are legible. Usability tests were carried out to determine if the content layout is in proper order and the text is easy to read, and the navigation from screen to screen is logical. 7.4 FEATURES NOT TESTED The PL-Analyser system was not tested on external elements such as the network infrastructure that the PL-Analyser server is located in; the security and general settings of the firewall and Dynamic Host Configuration Protocol (DHCP) servers; and the architecture of the server that the PL-Analyser system is hosted. The PL-Analyser system was also not tested using different Internet browsers such as Opera, Firefox, Mac and Netscape, and under different screen resolutions such as 600x800 and 1024x

3 7.5 TEST APPROACH The functional tests of the PL-Analyser system were performed with the help of thirty volunteers. The volunteers were explained the functionalities of the system, and provided with test cases as guidance. The functional test cases are highlighted in Section 7.7. The broken link, performance and usability tests were performed to ensure that the PL-Analyser system has no broken links and that the database, presentation and user interface tiers perform at the defined requirements, described in Table 5.8. The broken link, performance and usability tests were conducted using automated testing tools such as Microsoft Application Centre Test, Web Validator, Link Checker Pro, Xenu link s Sleuth, Alert Link Runner, and Automation Anywhere. The results of the functional, broken link, performance and usability tests are tabulated and analysed to determine areas for improvements. 7.6 TEST CASE IDENTIFIERS In the system testing phase, each test case is assigned an identifier. For the functional tests, the identifier of the test cases will have the abbreviation FTC for Functional Test Case, followed by a number. An example would be FTC-1, which refers to functional test case number 1. Similar to the functional test cases, the broken link test cases will have the abbreviation BLTC for Broken Link Test Case followed by a number. An example would be BLTC-1, which refers to broken link test case number 1. Similarly, performance test cases will have the abbreviation PTC for Performance Test Case, followed by a number, and usability test cases will have the abbreviation UTC for Usability Test Case, followed by a number. 108

4 7.7 FUNCTIONAL TEST CASES The functional Functional the PL-Analyser system are described in Table 7.1 to Table 7.7. Test Case ID: Test Condition: Table 7.1: Functional Test Case 1 FTC-1 Opening a browser and connecting to the PL-Analyser Website. Procedures: 1. Open Internet Explorer via Start -> All Programs -> Internet Explorer. Input data: Expected Results/Objectives: Actual Results: 2. Type pl-analyser.fsktm.um.edu.my at the Address box and click Enter. 1. URL: 1. Displays the main page of the PL-Analyser site. Resulting data: - Test Status: Pass: Fail: Comments: Table 7.2: Functional Test Case 2 Test Case ID: Test Condition: Procedures: FTC-2 Browse literature within the PL-Analyser site. 1. Using the same browser in test case 1, the main page contains the day s feature article under the PL-Analyser NEWS: Today's Article section. Click on the Full Story link to browse article. 2. Click on the About PL-Analyser link under the About section on the left to access information about the PL-Analyser system. 3. Click on the About Us link under the About section on the left to access information about the university and the development team. 4. Click on the Links & Site Services link under the About section on the left to access information about the general layout of the PL-Analyser system, user documentation, and links to similarly sites available in 109

5 Input data: - Expected Results/Objectives: Actual Results: the World Wide Web. 5. Click on the Personality Traits or Leadership Styles or Career Codes link under the Main Title section on the top to access information about these theories. 1. Displays the feature article under the PL-Analyser NEWS: Today's Article section. 2. Displays the content of the About PL-Analyser section. 3. Displays the content of the About Us section. 4. Displays the content of the Links & Site Services section. 5. Displays the content of the Personality Traits or Leadership Styles or Career Codes. Resulting data: - Test Status: Pass: Fail: Comments: Test Case ID: Test Condition: Procedures: Input data: - Expected Results/Objectives: Actual Results: Table 7.3: Functional Test Case 3 FTC-3 Link and branch out to other sites from the PL-Analyser site. 1. Click on the Links & Site Services link under the About Us section on the left. 2. Click on the Website URL The Big Five Quickstart. 1. The browser should open The Big Five Quickstart Website. Resulting data: - Test Status: Pass: Fail: Comments: 110

6 Table 7.4: Functional Test Case 4 Test Case ID: Test Condition: Procedures: Input data: Expected Results/Objectives: Actual Results: FTC-4 Register as a user and change password. 1. Click on the Register link to start the registration process. Select Individual as membership type and click on the Next button to continue. 2. At the Register - Page 2 of 5 page, fill in the necessary personal details in this page and click on the Next button to continue. 3. At the Register - Page 3 of 5 page, fill in the necessary education details in this page and click on the Next button to continue. 4. At the Register - Page 4 of 5 page check the details in this page and click on the Next button to confirm registration. 1. Membership type: Individual. 2. Gender: Male. 3. Ethnic group: Indian. 4. Nationality: Malaysian. 5. Age group: rmrama@yahoo.com. 7. Hobby 1: Reading. 8. Hobby 2: Listening to music. 9. Profession type: Working Group. 10. Education level: Bachelors Degree. 11. Working experience: 5 and below. 12. Department: ATTD-EA. 13. Position: Operations. 14. Industry: Manufacturing. 1. The confirmation page with a user ID and password appears. Resulting data: 1. User ID: rmrama@yahoo.com. 2. Password: system generated 4-digit number. Test Status: Pass: Fail: Comments: 111

7 Table 7.5: Functional Test Case 5 Test Case ID: Test Condition: Procedures: Input data: Expected Results/Objectives: Actual Results: FTC-5 Take a survey. 1. From the main page, click on the Take Survey link to open the Survey Introduction - Page 1 of 2 page. Type in user ID and password and click on the Next button to continue. 2. At the Survey Introduction - Page 2 of 2 page check on the Leadership Scales option and click on the Next button to continue. 3. At the Leadership Styles Survey Page - Page 1 of 3 click on the Next button to continue. 4. At the Leadership Style Survey Page - Page 2 of 3 page select your survey response and click on the Next button to continue. 1. User ID: rmrama@yahoo.com. 2. Password: Click on button A on all questions 1 to Confirmation page that survey responses have been successfully captured appears. Resulting data: - Test Status: Pass: Fail: Comments: Table 7.6: Functional Test Case 6 Test Case ID: Test Condition: Procedures: Input data: FTC-6 View results. 1. From the main page, click on the View Survey Results link to open the Results - Page 1 of 4 page. Type in user ID and password and click on the Next button to continue. 2. At the Results - Page 2 of 4 page select the desired report and fill in the required parameters (if any) and click on View to continue. 1. User ID: rmrama@yahoo.com. 2. Password:

8 Expected Results/Objectives: Actual Results: 3. Select report: 1 - Sample statistics under Demographics Summary heading. 1. Selected report should display results. Resulting data: - Test Status: Pass: Fail: Comments: Test Case ID: Test Condition: Procedures: Table 7.7: Functional Test Case 7 FTC-7 Provide feedback and forward recommendations. 1. At the report display screen, click on the results feedback link. 2. At the Results Feedback page, select the appropriate response and click on the Next button to continue. Input data: 1. Click on button 100% accurate for questions 1. Expected Results/Objectives: Actual Results: 2. Click on button 100% accurate for questions Click on button Very easy for questions Message confirming feedback has been saved appears. Resulting data: Test Status: Pass: Fail: Comments: 113

9 7.8 SUCCESS CRITERIA The system testing of the PL-Analyser system is considered successful if the functional, broken link, performance and usability tests results show that every test has a PASS status. From the perspective of functional testing, each test case can be repeated until the users agree that the test criteria of re-run test cases have been met. As for broken link, performance and usability testing, tests using automated test tools can be repeatedly carried out until all technical issues have been fixed and the system is confirmed to be performing according to the requirements in Chapter ENVIRONMENTAL REQUIREMENTS In order to carry out the tests on the PL-Analyser system, the servers that are hosting the PL-Analyser system must be online. Other components such as network connectivity and power supply must also be functioning. The hardware, software and the automated test tools listed in Table 7.8 to Table 7.10 were used during the testing process. Table 7.8: Hardware required for testing Type Lab Computer Descriptions Used to install the PL-Analyser system. Computer keyboard Mouse Allow user to key in the data into the system. Used to scroll the page up and down. Click to view specific information. Table 7.9: Software required for testing Type Windows XP Microsoft Office Word Microsoft Visual Studio Microsoft SQL Server 2005 Internet Explore version 5.0 and above Descriptions Operating system. For test documentation. System development tool to run the PL-Analyser in local host and to review the codes for testing and debugging. Database to store and retrieve the data in the system. To access to PL-Analyser. 114

10 7.10 STAFFING REQUIREMENTS The testing on the PL-Analyser system was led by a Test Lead. The Test Lead was responsible for guiding the thirty volunteers to complete the functional test cases, and to collect and analyse the test results. The Test Lead was also assisted by six testers to perform the broken link, performance and usability tests on the PL-Analyser system. The six testers were divided into three groups with two testers in one group to perform the tests using the automated tools such as Microsoft Application Centre Test and Web Validator. The test team structure is illustrated in Figure 7.1. Test Lead M. Ramanathan Broken link, performance and usability test teams Functional Test 30 volunteers Team 1 Sheyda Shahidi Tan Fung Ling Tools: Alert Link Runner Link Checker Pro Automation Anywhere Web Validator Team 2 Asma Raihan Tengku Hasliza Tools: Alert Link Runner Link Checker Pro Automation Anywhere Xenu link s Sleuth Team 3 S.Menaka Maryam Zakery Tools: Kyosoft Link Checker Alert Link Runner Web Validator Ms App. Centre Test Figure 7.1: PL-Analyser test team structure 7.11 RISKS AND ISSUES The risk items identified by the test teams during the system testing phase are as follows: 115

11 i. The test teams may not be able to test every possible aspect of the system, every possible combination of events, every dependency, or everything that could go wrong. ii. Owing to time constraints, the quality of the tests may be compromised. iii. Owing to new technologies, the learning curve of the team might be high. This will results in a longer installation and learning process, which may reduce the time available for testing. iv. The testing team installed the PL-Analyser system in one of the computers in the faculty s computer laboratory. As a result, there will be a delay for testers to complete their tests, whenever the laboratory is used for class. v. Some of the testing tools are online. If the server that hosts the tools is down, then this will impact the testing activities TEST RESULTS The results of the functional tests, broken link tests, performance tests and usability tests are documented in the following sections Functional Test Functional test was performed by thirty volunteers and iterated two times, with each iteration designed to fix the errors detected in the previous test. The test results of the two iterations are listed in Table 7.10 and Table Table 7.10: Summary of functional test results (iteration 1) Test case ID Test Result (PASS/FAIL) Severity of error(s) Description of error(s) FTC-1 PASS - - FTC-2 FAIL Low Link error. FTC-3 FAIL Low Link error. FTC-4 FAIL Medium Error validating format. FTC-5 FAIL Medium Error saving 116

12 survey response. FTC-6 PASS - - FTC-7 PASS - - The first iteration of the functional test revealed errors pertaining to wrongly coded links and validation of data entered. FTC-2 revealed error on the link that leads to the Leadership Styles article. FTC-3 revealed an error on the link that leads to the The Big Five Quickstart external Website. All links were fixed by coding the right URLs in the PL-Analyser Web pages. FTC-4 revealed error in validating the ID of the users. The PL-Analyser system was found to save IDs that are not in the correct format. This error was fixed by coding in the right regular expression validation into the PL-Analyser Web pages. FTC-5 revealed errors in the way PL-Analyser handles survey responses. PL- Analyser was found to display a misleading error message when checking to make sure that users have answered all the survey questions. The PL-Analyser system was also not able to correctly save the survey responses of the user after the user navigates to other survey pages. Both errors were fixed by correcting the validation codes in the PL- Analyser Web pages. After the errors that were reported were fixed, the functional test cases were then rerun. The test results of the second iteration are described in Table 7.11, and the functional test results are shown in Appendix D. Table 7.11: Summary of functional test results (iteration 2) Test case ID Test Result (PASS/FAIL) Severity of error(s) Description of error(s) FTC-2 PASS - - FTC-3 PASS - - FTC-4 PASS - - FTC-5 PASS

13 Automated Tools for Broken Link, Performance and Usability Tests Broken link, performance and usability tests were performed using the automated test tools listed in Table Table 7.12: Automated test tools used for testing PL-Analyser Test tools Link Checker Pro (Version ) Xenu link s Sleuth (Version 1.2i) Alert Link Runner (Version 6.01) Automation Anywhere (Version 4.0) Web Validator (Version 4.7) Kyosoft Link Checker (Version 1.1) Microsoft Application Centre Test Description Used for Website analysis and detection of broken and other problem links. Able to create graphical Website maps from any Website data saved in Link Checker Pro format. Checks Website for broken links. Link verification is done on "normal" links, images, frames, plug-ins, backgrounds, local image maps, style sheets, scripts and Java applets. Checks the validity of hyperlinks on any Web page, Website, or across an entire enterprise network. Able to repair bad links for individual site or Web enterprise. Able to automate software testing, Website testing or distributed load testing without requiring any programming. Checks links for accuracy and availability, find broken links and links containing syntactic errors. Also able to find spelling errors with the built-in spell checker. Checks internal and external links within their Internet site. Tests Web servers and analyses performance and scalability problems of Web applications. Simulates a large group of users by opening multiple connections to the server and rapidly sending HTTP requests Broken Link Test The broken link tests were performed by three testing teams using the automated test tools listed in Table Analysis of the broken link errors (BLTC-5) revealed a mailto: directive in the codes that held an address that could not be verified. This error was fixed by making the address a configurable placeholder in the Web.config file. Another link that led to a non-existent page (BLTC-9) was removed. 118

14 The spelling mistakes (BLTC-6) were corrected by changing the text on the relevant web pages. The link error reported by BLTC-1, BLTC-3, BLTC-7, BLTC-8 and BLTC- 9 was caused by a link that could not be accessed by the automated test tools. This error was corrected by removing the link. Table 7.13: Test results of broken link testing performed using automated test tools Test case ID BLTC-1 BLTC-2 BLTC-3 BLTC-4 BLTC-5 BLTC-6 BLTC-7 Automated test tool Alert LinkRunner version 6.0 Link Checker Pro version Link Checker Pro version Link Checker Pro version Web Link Validator version 4.7 Web Link Validator version 4.7 Automation Anywhere version 3.5 Test case Verify the hyperlinks in the system. Check the validity of the hyperlinks in the system. Detect the bad connection status hyperlinks in the system. Detect the missing hyperlinks in the system. Detect the broken hyperlinks in the system. Detect any spelling errors in the system. Find the Website broken hyperlinks. Test Result (PASS/FAIL) FAIL PASS - FAIL FAIL FAIL FAIL FAIL Severity and description of error(s) Severity: Low. Error with link Circuit.com. Severity: Low. Error with link Circuit.com. Severity: Low. Error with link Circuit.com and mailto: Line. Severity: Low. Error in the mailto: Line. Severity: Low. Detected in wfmain.aspx in words ersonality, eadership, Malaya and easurement. Severity: Low. Error with link Circuit.com. 119

15 BLTC-8 BLTC-9 Xenu link s Sleuth version 1.2i Kyosoft Link Checker version 1.1 Verify the hyperlinks in the system. Verify the hyperlinks in the system. FAIL FAIL Severity: Low. Error with link Circuit.com. Severity: Low. Error with link Circuit.com Performance Test The performance test was conducted using the Microsoft Application Centre Test tool. This test was performed five times. All the tests were performed during the day time. The performance test was initially configured to simulate one user using the PL- Analyser system. The number of concurrent users was then increased to five, ten, twenty and fifty. Each test was executed for a duration of three minutes, and the results are documented in the denomination of seconds in Table Time (sec) Table 7.14: Outcomes of performance testing performed using Microsoft 1 user 5 concurrent users HTTP HTTP HTTP HTTP request errors request errors / sec / sec / sec / sec Application Centre Test 10 concurrent users HTTP HTTP request errors / sec / sec 20 concurrent users HTTP request/ sec sec HTTP errors/ 50 concurrent users HTTP request/ sec HTTP errors/ sec % error at 180 sec 31.8% 31.0% 31.3% 95.0% 95.5% 120

16 Table 7.14 shows the number of HTTP requests per second made by the automated test tool to the PL-Analyser system, and the number of HTTP errors per second returned by PL-Analyser to the automated test tool. The HTTP errors per second returned can be made up of a combination of errors from pages that cannot be returned due to bad HTTP requests (e.g. : pages that require user ID and password to be entered) and errors from the server that cannot process the errors due to heavy load. As a rule of thumb, the PL- Analyser system must report low HTTP errors per second as compared to HTTP requests per second made by the automated test tool. In the test to simulate one, five and ten concurrent users, the average percentage of HTTP errors per second constitutes 31.4% of the HTTP requests per second made by the automated test tool. This shows that 68.6% of the HTTP requests per second made by the automated test tool are replied by PL-Analyser. This shows that PL-Analyser is capable of handling up to 10 concurrent users. In the test to simulate twenty and fifty concurrent users, the percentage of HTTP errors per second constitutes 95.3% of the HTTP requests per second made by the automated test tool. This shows that only 6.6% of the HTTP requests per second made by the automated test tool receive a reply from the PL-Analyser system. This means that the PL-Analyser system is not capable of handling more than 20 concurrent users. The results of the performance tests are graphically illustrated in Figure 7.2 and Figure 7.3. The two graphs show the number of HTTP requests per second made by the automated test tool within the time span of three minutes, and the number of HTTP errors per second returned by the server that hosts the PL-Analyser system within the same time span. 121

17 Figure 7.2: Performance of PL-Analyser serving 1 user Figure 7.2 shows that the PL-Analyser system returns a very low number of HTTP errors per second from 0 to 160 seconds. The numbers of HTTP requests per second on 20th, 40th, 60th, 80th, 100th, 120th, 140th and 160th seconds are 3, 1, 5, 6, 2, 5, 2 and 5, respectively. The numbers of HTTP errors per second on 20th, 40th, 60th, 80th, 100th, 120th, 140th and 160th seconds are 0, 0, 1, 2, 0, 2, 0 and 1, respectively. The number of HTTP requests per second made by the automated test tools spike on 180 seconds to 22 and this caused the HTTP errors per second also to increase to 7. However, it can be seen that the number of HTTP errors per second recorded are low as compared to the number of HTTP requests per second made. Figure 7.3: Performance of PL-Analyser serving 20 concurrent users 122

18 Figure 7.3 shows that the PL-Analyser system returns a high number of HTTP errors throughout the 180 seconds. The number of HTTP requests per second on 20th, 40th, 60th, 80th, 100th, 120th, 140th, 160th and 180th seconds are 950, 1050, 1000, 1000, 1050, 1000, 1100, 1100 and 1000, respectively. The number of HTTP errors per second on 20th, 40th, 60th, 80th, 100th, 120th, 140th, 160th and 180th seconds are 900, 1000, 900, 850, 950, 950, 1000, 1050 and 950, respectively. It can be seen that the number of HTTP errors per second recorded are almost identical to the number of HTTP requests made per second. This shows that the PL-Analyser system is under strain during the tests that simulates twenty concurrent users. The performance test results simulating five, ten and fifty users are included in Appendix C Usability Test The usability testing using Automation Anywhere version 3.5 returned no errors, and the test results are described in Table In this table, it can be seen that the PL- Analyser system is able to execute the functionalities described in Chapter 5. However, it was discovered from the tests that the information displayed on the Web pages of the PL-Analyser system are lengthy and this makes it difficult for the users to understand the content due to congested text. To solve this issue, a lengthy Web page was divided into multiple pages, with each page fitting on the computer screen. Links were provided in each page so that users can navigate from one page to another. Table 7.15: Test results of usability testing performed using Automation Anywhere Test case ID Test case version 3.5 Test Result (PASS/ FAIL) UTC-1 Test the usability of the PL- PASS - Analyser main menu page. UTC-2 Verify the user name and PASS - Severity and description of error(s) 123

19 UTC-3 UTC-4 UTC-5 UTC-6 UTC-7 password used in the system. Test the usability of the personal traits survey. Test the usability of the leadership styles survey. Test the usability of the individual summary result. Test the usability of the demographics summary result. Verify the legibility of the text on the Web page. PASS - PASS - PASS - PASS - PASS The information is lengthy. User find it difficult to understand the content due to overcrowded text SUMMARY This chapter describes the testing approaches, test tools used to test the PL-Analyser system, and documents the test results. The functional test results show that the PL- Analyser system meets all the functional requirements defined in the requirements analysis phase. The broken link test results show that all the links inside the PL- Analyser system refers to valid links. The performance and usability test results show that the PL-Analyser system is capable of handling requests made by 1-10 concurrent users at the performance level of 68.7%. 124