Software Testing 2. OOD and Testability. White box vs Black box Testing. Software Testing 2 Semester 1, 2006

Transcription

1 Software Testing 2 Jens Dietrich OOD and Testability Component based design and component based unit testing. Design that isolates component that are difficult to test (automatically) (such as user interfaces). Replace objects by dummies in test environments. Document software. Stable component APIs. Component interface that reveals the state of the component (this can contradict encapsulation and other design goals). Be careful to use state-changing method to read the state of a component (like lazy initialisation). Postgraduate Certificate in Professional Development Slide 2 White box vs Black box Testing White box testing control structure is used to derive test cases. Black box testing test cases based on functional requirements, use public interfaces of the software. Black box tests tend to be less "complete" than white box tests. Black box tests are very often the only tests possible if 3rd party components have to be tested. This becomes more and more important as it is usually cheaper to buy software and not to make it. However, testing is very often necessary for components bought off the shelf. (e.g. when assessing or integrating them). Postgraduate Certificate in Professional Development Slide 3 Postgraduate Certificate in Professional Development 1

2 Basis Path Testing White box testing technique. Goal: each statement is executed at least once. Consider all paths of program execution. For this purpose it is useful to represent a program as graph. Nodes are blocks of code, directed edges are invocations of other code blocks and represent flow of control. Nodes with two exiting rows contain conditional statements. The graph contains initial and terminal nodes. A path is a sequence of nodes connecting initial and terminal nodes, an independent path is a path introducing at least one new node. Coverage tools (such as Clover and JCoverage) can be used to make sure that test cases cover all execution paths. Postgraduate Certificate in Professional Development Slide 4 Basis Path Testing (ctd) The number of independent paths can be used to measure the complexity of a program. It is called cyclomatic complexity and can be computed as follows: V(G) = Edges Nodes + 2. Cyclomatic complexity is an upper bound for a base set of path, I.e. a set of path that includes each node. If each path is translated into a test case, each statement will be tested at least once. Postgraduate Certificate in Professional Development Slide 5 Boundary Value Testing A relatively high number of errors occurs at the boundaries of the input domain (e.g. when using min and max value permitted). This is often caused by the fact that boundaries represent extreme situations, and that software is designed and implemented by people who ignore that these situations are realistic. Postgraduate Certificate in Professional Development Slide 6 Postgraduate Certificate in Professional Development 2

3 Error Handling Testing Modern software can usually handle errors and can avoid total failure (system crashes) if errors occur. This is a design goal called fault tolerance. There are different error handling strategies: crash dumps user messages, logging of error events, retry, etc. Error handling has to be tested as well. Postgraduate Certificate in Professional Development Slide 7 Organisational Aspects Software projects should have a test strategy which encompasses the following aspects: Plan testing. Design test cases. Run tests. Reporting test results. Postgraduate Certificate in Professional Development Slide 8 Organisational Aspects (ctd) Part of the test strategy is answering the following questions: Who does the testing (programmers, independent test teams)? How often is testing done? E.g., when the software is build, before it is deployed, at each milestone. Is testing done bottom-up or top-down? (starting with the components (parts) or starting with the entire system?). What is the procedure to follow if defects are detected? Postgraduate Certificate in Professional Development Slide 9 Postgraduate Certificate in Professional Development 3

4 Verification vs Validation Testing is done to verify and to validate software. Verification the software implements a certain function correctly. Validation software implements the requirements. Postgraduate Certificate in Professional Development Slide 10 Test Directions bottom-up testing top-down testing Tested: Component aggregation buy component make component buy component make component integration tests unit tests whitebox blackbox requirements design implementation bottom-up testing Postgraduate Certificate in Professional Development Slide 11 Regression Testing Tests must be repeated continuously. Any change (modifying components, adding new components, removing components) brings the software into an untested ( dirty ) status, and tests must be repeated. Recommended: automated testing. Tools: unit testing frameworks or capture / playback tools (in particular useful for user interface testing. Postgraduate Certificate in Professional Development Slide 12 Postgraduate Certificate in Professional Development 4

5 Documenting Test Results Test results must be documented. E.g., these documents can be used for project planning and controlling. Test reports should contain information about the test cases which succeeded and failed, the reasons for failure, and performance data. Some test tools such as JUnit generate test reports automatically. Postgraduate Certificate in Professional Development Slide 13 Validation Testing Validation / acceptance testing test whether software implements the requirements. Usually performed by end user. High level integration test (end user often not aware of the component structure). Very often non-automated (although record/playback tools could be used), test protocols should be written. Postgraduate Certificate in Professional Development Slide 14 System Testing Testing non-functional requirements, including: Security testing (e.g., test cases to make sure that only users can login, or that only certain ports or protocols are available). Recovery testing (whether software can recover into a consistent state). Stress testing (testing software with extreme data volumes and transaction rates in particularly important for server applications). Performance testing (testing the runtime performance of the software). Compatibility testing (e.g., test with different versions / products of interfacing software components, e.g., web browsers or databases). Postgraduate Certificate in Professional Development Slide 15 Postgraduate Certificate in Professional Development 5

6 Terminating Testing There is never enough testing, but time and money is limited. Formal metrics: test coverage. If test coverage is sufficient, and all test cases succeed, testing ends. Example: clover test coverage tool (commercial, free licenses for academic and open source applications), JCoverage (free GPL version). Postgraduate Certificate in Professional Development Slide 16 Example Test Coverage Metrics Postgraduate Certificate in Professional Development Slide 17 Measuring Software Quality Metrics can be used to measure (quantify) characteristics of SDLC processes and artifacts (models, software, documents). Example: design model complexity (e.g., function points), code size (e.g., KLOCs). This includes characteristics related to quality. Several metrics are related to several steps in the SDLC (analysis, design, implementation). Postgraduate Certificate in Professional Development Slide 18 Postgraduate Certificate in Professional Development 6

7 Measuring Software Quality (ctd) Some metrics can be calculated based on empirical data. E.g. MTBF (mean time between failures). Other metrics (coupling) can be calculated before deployment by analyzing design models and source code. Test coverage metrics can measure the quality of tests. Metrics are an active field of research, an example on coupling is presented here. Postgraduate Certificate in Professional Development Slide 19 Measuring Software Design Complexity Idea: measure the dependencies between modules. What is a module? Integrated set of software components addressing a set of requirements. Sometimes, module and component are used as synonyms, sometimes a module refers to a more complex structure consisting of many components or sub-modules. Not clear what a module is for an OO language such as Java. E.g., a module could be a package or a class. Sometimes, a package represents several modules (e.g., if we call the date related classes such as Data, Calendar and Gregorian Calendar a module, and the java collection library another module, the java.util package contains at least 2 modules. If we consider the AWT as a module, then this module contains at least 2 packages (java.awt and java.awt.event). Postgraduate Certificate in Professional Development Slide 20 Dependencies between Classes and Packages Dependency between classes: class A depends on class B if B is referenced in A. This may include: B is the type of a variable of A, B is return type, parameter type of exception type of a method (in A), B is referenced within the code (e.g., method, static initializer, variable initialization block or constructor) of B, A is superclass or interface subclasses. These dependencies can be verified by the compiler (at least if the language supports strong typing [like Java, C++, C#, unlike Smalltalk]). Or "soft references" using reflection (like Java's Class.forName, invoke, Smalltalk perform ) cannot be verified by compiler. Package A depends on package B if at least one class in A depends on one class in B. Postgraduate Certificate in Professional Development Slide 21 Postgraduate Certificate in Professional Development 7

8 Analyzing Code To gather metrics w.r.t. dependencies, code must be analyzed. Options are: Analyzing byte code. Analyzing source code. Reflection. Observing code executing (e.g., using a debugger interface). Reflection is very easy to implement, but does only reveal some dependencies (e.g. class occurrences in declarations). The other options are more difficult to implement, but yield better results. Postgraduate Certificate in Professional Development Slide 22 Measuring Coupling between Packages Afferent Couplings How many other packages depend upon a package. An indicator of the package's responsibility. Efferent Couplings On how many other packages depends a package. An indicator of the package's independence. Coupling between packages should be part of the design model, metric can be used to verify the design (as part of the SQA). Circular dependencies / couplings usually indicate problems. Postgraduate Certificate in Professional Development Slide 23 Measuring Coupling between Packages(ctd) This translates often into runtime problems like Increased program size. Platform / version dependencies. Configuration problems. Related: abstractness (number of abstract classes / interfaces). Abstract packages have more afferent couplings, Packages with many efferent couplings should have a low abstractness level. Abstractness A = interfaces+abstract classes/total classes JDepend ( is a good free dependency checker for java that can be integrated into an ANT script. Postgraduate Certificate in Professional Development Slide 24 Postgraduate Certificate in Professional Development 8

9 Measuring Coupling between Packages (ctd) A couple of derived metrics can be computed from afferent (CA) and efferent couplings (CE). Instability I = CE / (CE + CA). This metric is an indicator of the package's resilience to change [ Factor is high if there are relatively many dependencies on other packages this package is more likely to change. Abstract packages should be more stable, implementation packages are generally more instable. Good packages are close to the main line A+I=1 (good balance between stability and abstractness). Open questions: Context for metrics (e.g., should test classes, packages count). Should exception classes be considered as abstract? Measuring abstractness on a sub-class level (for abstract classes). Postgraduate Certificate in Professional Development Slide 25 Couplings and Architectural Design Example: typical couplings between modules Other Module Shared Module Tests Application layers Abstract Implementation CA = 0 CE = 2 I = 1 A = 0 presentation (GUI) CA = 4 CE = 0 I = 0 A = 0.8 CA = 2 CE = 1 I = 0.33 A = 0 logic persistency Postgraduate Certificate in Professional Development Slide 26 Applied Testing Some Case Studies on Testing in Java. JUnit primer. Writing JUnit test cases. Measuring test coverage with Clover. Integrating JUnit into ANT. Testing complex applications: Server side application design. Mock objects. In container testing: Cactus. Postgraduate Certificate in Professional Development Slide 27 Postgraduate Certificate in Professional Development 9