Software Testing. 14. Application Systems. International Funds Transfer System Development Team - Objects work, objects are right The System Works!

Transcription

1 Software Testing 14. Application Systems Daniel Riegelhaupt - Thomas De Vylder - Loots Gertjan - Saquet Tim - Philip De Smedt 14. Application Systems Testing Application Systems Test Design Patterns Implementation-specific Capabilities Post-development Testing Notes on Testing Performance Objectives 1 International Funds Transfer System Development Team - Objects work, objects are right The System Works! Consequence - Incorrect transaction processing - Bizarre results - Mad customers - Robbery - Lawsuits - Bankruptcy - 2 Testing Object-oriented Application Systems Individual verification of components does not guarantee a correctly functioning system System Scope Testing - Is based on requirements and capabilities (mostly implementation independent) - Should be done after Integration Testing (ch. 13) 3 1

2 Application System Test Application System = a collection of subsystems that cooperate to support a feature set intended for a user or customer General System Test Strategy 1. Develop extended uses cases (slide 31, ch. 12 Subsystems) for entire system + testable implementation-specific requirements 2. Do Unit + Integration tests 3. Develop test suite using extended uses cases Goals: Reveal system scope bugs Show SUT implements all required capabilities Provide information to answer Can we ship it yet? 4. (optional) allocate tests by profile (~frequency, ~testing productivity, ~budget, -> resource allocation) 5. Run test suite, debug, regression testing 6. Develop and run implementation- specific tests 7. Perform execute-debug-retest cycle until objectives are met (or time/ budget gone) 8. Release system Application Systems Testing Application Systems Test Design Patterns Implementation-specific Capabilities Post-development Testing Notes on Testing Performance Objectives Test Design Patterns Develop test suite for essential requirements - Test relationships implied by use case Covered in CRUD - Test operations not reached by use cases - Develop tests in proportion 6 7 2

3 Intent - Model essential capabilities as extended use cases to develop an application system test suite Context - Use when most essential requirements for SUT can be expressed as use cases Example: ATM 8 9 Context - Use cases are not always good models (complex algorithms used for computing) Example: complex simulation to compute and render animated high resolution graphics Context - Possible to develop test cases by imagining specific instances for use cases TABLE 14.1 Some Use Cases and Scenarios for an ATM Use Case Actor Possible Input/Output Combinations Establish Session Bank Customer (1) Wrong PIN entered once; corrected PIN entered. Display Menu. (2) PIN OK; customer bank not online. Display Try Later. (3) PIN OK; customer s accounts are closed. Display Call your bank. (4) Stolen card inserted; valid PIN entered. Retain card. Cash Withdrawal Bank Customer (1) Requests 50; account open; balance 1.234,56; 50 dispensed. (2) Requests 100; account closed. (3) Requests 155,39; account open; 150 dispensed. Cash Replenishment ATM operator with armed guard (1) ATM opened; cash dispenser is empty; is added. (2) ATM opened; cash dispenser is full

4 Context - Test points best selected based on use case s implicit constraints/relationships to avoid missing relationships to test - Enumeration leads to large test suites (inefficient!) - Efficient and effective testing by Modelling constraints and relationships Generating enough test cases - Can be used where use cases are used (system/ class/cluster/subsystem- scope) Fault Model - Use case: inputs -> responses Can be modeled as decision table (slide 9 ch. 2 Models) - Domain faults (boundaries, invariant boundaries slide 35 ch. 3 Classes) - Logic faults (AND/OR faults) - Incorrect handling of don t care conditions - Incorrect/missing dependency on states by prior use cases Extra (generic system scope) faults: - Unwanted feature interactions - Wrong output - Abnormal termination - Unacceptable response times - Omitted/Extra capabilities Strategy: Test Model - Extended use case contains All operational variables All domain constraints of each operational variable Operational relation for each use case Relative frequency of each use case (supports Allocate Test By Profile) Strategy: Procedure 1 Identify operational variables (per scenario) Explicit inputs and outputs Environmental conditions leading to different actor behaviour Abstraction of the state of SUT Example: ATM state (out of cash, ready, ), PIN on card, PIN entered, response of bank,

5 Strategy: Procedure 2 Define the domains of the operational variables Set of valid and invalid values for each variable Example: PIN domain: Strategy: Procedure 3 Develop the operational relation Relationships among the operational variables are modeled (Decision Table) Example: Table 14.2 Operational Relation for Established Session Use Case Variant Operational Variables Expected Result Card PIN Entered PIN Customer Bank Response Customer Account Status Message Card Action 1 Invalid DC DC DC Insert an ATM card Eject 2 Valid Matched Card PIN 3 Valid Matched Card PIN Bank ACKs Closed Contact your bank Bank ACKs Open Select a transaction Eject None Strategy: Procedure 4 Develop test cases At least 2 test cases for each variant (true and false test case) False case can be true case for other variant Strategy: Oracle - Develop expected results manually - Look at existing systems that provides similar capabilities (ch. 18 Oracles) Example: Table 14.3 Minimal Test Suite for Established Session Use Case Variant Operational Variables Expected Result Card PIN Entered PIN Customer Bank Response Customer Account Status Message Card Action 1 Invalid DC DC DC Insert an ATM card 1T %*@# Insert an ATM card Eject Eject 1F Any true test for variants

6 Entry criteria - Extended use cases are developed and validated - SUT passes an integration test suite Shows components supporting the use case are (minimally) operable Exit criteria - All requirements must be exercised (Test Coverage metric indicates completeness) Variant Coverage (at least one true and one false test pair for each variant) Every use case has a test that passes (use case/test case tracebility) Disadvantages - Unclear level of use case abstraction Too much/little detail -> extra work for tester - Use cases don t specify performance, fault tolerance, -> should be tested - Composite use cases (includes/extends) must be flattened Advantages - Use cases are widely used - Use cases are simple - Use cases are often only documentation available - Use cases imply good object oriented analysis and design - Use cases reflect user point of view - Known Uses - Telecommunications systems - High-volume financial applications

7 Covered in CRUD Intent - Verify that all basic operations (Create, Read, Update, Delete) are exercised for each problem domain object in the SUT Context - Simple technique to identify omissions of problem domain classes - Used when many use cases share responsibility for maintaining problem domain classes Fault Model Covered in CRUD Problem domain class = a presentation of an external entity that is necessary for an implementation-independent model of the SUT - Example in order-processing system: Customer, Order, and Product are problem domain classes. LinkedList is not Covered in CRUD Fault Model - CRUD action not in any use case If not explicitly specified: - System may not perform CRUD action on that class (Surprises) Else: - Suite not adequate enough to find faults Strategy: Test Model - Create CRUD coverage Matrix Identify all system use cases Identify problem domain classes Prepare use case/class matrix and analyse it Covered in CRUD

8 Covered in CRUD Strategy: Test Model - Create CRUD coverage Matrix If test suite does not cover all actions, develop test cases (with, Round-trip Scenario Test, and Class Test) Use Case Test Coverage of Basic Action Required Behaviour for a Basic Action Required Not Required Prohibited Strategy: Test Procedure Covered in CRUD 1. Develop matrices (previous slides) 2. Develop & execute Suite 3. Map Suite to matrix 4. Add test cases to achieve CRUD coverage Exercised, passes Validated Surprise Failure Exercised, throws expected exception Failure Validated Validated Not exercised Incomplete test suite: Incomplete test suite: Incomplete test suite: add exception test add exception test add exception test Covered in CRUD Entry Criteria - Must have use cases and problem domain classes - Value is limited to accuracy/completeness/ consistency of these inputs Exit Criteria - Test suite achieves coverage when all CRUD operations for all problem domain classes are exercised and all tests pass Intent - Allocate testing budget to each use case in proportion to its relative frequency Context - While planning for suite, How many test for each variant? How should these tests be interleaved? - Develop test cases until budget/time runs out

9 - Example: Test Case Allocation for ATM Fault Model - Frequency of failures seen by customers ~ usage frequency - Well suited for Object-Oriented systems - Maximize chance of reaching/triggering faults that customers would find Strategy: Test Model - Operational profile: If system has 2 use cases: update + search On average 10,000 updates/day and 90,000 searches/day Operational profile = [{Search, 0.9}, {Update, 0.1}] Strategy: Test Procedure 1. Prepare the operational profile for the SUT 2. Estimate teams testing productivity 3. Establish system testing effort budget 4. Allocate testing time for each use case 5. Generate test cases for each use case 6. Choose strategy to interleave test cases (possibly random)

10 Strategy: Test Procedure - Keep in mind (Good engineering judgement) Rarely used but critical use cases -> separate operational profile Often used use cases but trivial implementation -> merge use case with other use case when testing Many rarely used use cases -> merge with other use cases when testing Entry Criteria - Adequate unit and integration testing is done on components and subsystems - Sufficient resources are committed to estimating the profile for all use cases Exit Criteria - Testing is done under limited time/budget - See (at least all-variants coverage) Advantages - Makes best use of available testing resources - Support stop testing decision - Cutting/reducing low-frequency use cases will have the least effect on overall reliability Disadvantages - Difficult/Impossible to obtain accurate operational profile (use heuristic, effort in most cases low) Known Uses - Weapon systems - Microsoft desktop application - Windows NT operating system - Embedded systems - NASA Space Shuttle - Tandem s Guardians operating system - Cleanroom Development approach

11 14. Application Systems Testing Application Systems Test Design Patterns Implementation-specific Capabilities Post-development Testing Notes on Testing Performance Objectives Implementation-Specific Capabilities Use cases don't require specific implementation Some critical system capabilities cannot be modeled by use cases System testing should exercise all SUT capabilities Many effective strategies exist (for system scope testing of object-oriented and component-based implementations) Generally, used after use case based testing Configuration and Compatibility (1) Applications often need to be compatible with many different environments (both platform and interoperation) Compatibility is a broad term, can be expressed in terms of configuration combinations that must be supported Examples of required SUT compatibility: - other applications - computer systems (any approved combination of hardware components) - operating systems, versions of an OS - GUI systems, versions of a GUI Configuration and Compatibility (2) Expected: same behavior for all configurations example: Java (though not total yet) Compatibility Strategy 1. Compatibility/configuration requirements analysis 2. For each configuration: identification of variables, allowed deviations, DC conditions, unspecified variables 3. Choice of subset of combinations 4. Test suite development

12 Performance (1) Requirement: the SUT must accomplish something within a specified time interval Performance-critical examples: desktop, high-volume transaction processing Not meeting this requirement is as much a bug as incorrect output or system crash Performance (2) Quantitative formulations in terms of response time, throughput, or utilization Some ways to develop these performance objectives measures are discussed at the end of this presentation Must be translated into constraints on execution time or resource use, under a stated load Often set for worst-case performance (e.g. for mission-critical systems), or for average performance (e.g. data-processing systems) Performance (3) Assumptions for factors that affect performance (incl. platform capabilities and state)! unclear assumptions => unreliable results Variations of performance testing: Load testing: often uses a load simulator in client/server systems (usually for transaction-processing systems) Volume testing: transmit high quantity of input at normal system load (usually for batch-processing systems) Concurrency Testing Context: SUT relies on (some form of) concurrent execution with shared resources Common issues: resource contention, scheduling, deadlock avoidance, priority inversion, race conditions Strategy: 1. Identify typical configurations of concurrent execution and loading levels 2. Test suite starts processes/threads, simulates typical input patterns 3. Goal: cause abnormal termination

13 Stress Testing Context: SUT has fail-safe response when quantity/rate of input beyond design limits (equivalent to vehicle crash testing) Goal: push SUT beyond design limits, causing failures (e.g. 10 times maximum) Types of faults revealed: lack of fail-safe behaviour load-sensitive bugs Restart/Recovery Testing Context: SUT supports automatic/manual recovery from failures Strategy 1. Trigger failure modes, leading to restart/ recovery 2. Verify the correct restart/recovery process is started 3. Use regression testing to check repaired state Human-Computer Interaction Illustration of importance: deaths caused by Therac-25 system HCI-related bug Information provided to the user (manuals, error messages,...) must be clear Complex requirements, many ways to test Basic Strategy: 1. List all SUT capabilities 2. Define objective pass/no-pass criterion per capability under test HCI Testing (1) Usability testing: keystroke recording, measure physiological and psychological responses Security testing: basic security features tested like others; advanced: attempt to breach Localization testing: multiple languages or other locale parameters; can be difficult to test Installation testing: (de)installation, upgrades, patches, different platform and installation configurations

14 HCI Testing (2) User documentation testing: correctness and completeness of user-targeted information Operator Procedure testing: correctness and completeness of operator-targeted info (all parts of system operation); may overlap with restart/ recovery testing Serviceability testing: field upgrades, repairs, reconfiguration Afterwards: Murphy's law! (human interaction) 14. Application Systems Testing Application Systems Test Design Patterns Implementation-specific Capabilities Post-development Testing Notes on Testing Performance Objectives Post-Development Testing (1) Alpha test By independent group, but often within the same company (internal acceptance test) Focus: simulate real-world usage Beta test By representative groups of users/customers To set timing of the general release Used for mass market desktop systems (e.g. browsers), computer games,... Post-Development Testing (2) Acceptance test By customer who commissioned the system Passing often required to receive payment Producer and consumer negotiate the criteria and test plan Compliance Test Test if the system meets the specified regulations/standards (government, IEEE,...)

15 14. Application Systems Testing Application Systems Test Design Patterns Implementation-specific Capabilities Post-development Testing Notes on Testing Performance Objectives Testing Performance Objectives Performance Objectives: Throughput (# accomplished tasks per time unit, e.g. transactions/sec) => productivity Response time (time interval between input and output, e.g. button) => responsiveness Utilization (busy/available ratio for a single component over a fixed interval, e.g. server utilization) => ability of a component to respond to a high load Batch Systems (1) POs: batch window, transaction throughput Batch window: commonly used, time interval for the completion of a batch job Start and finish times as for PERT (if parallel activity possible, critical path can be found) Two other factors can shrink the window: O: operator action time p(f): probability of failure (=> job restart) Batch Systems (2) BP (Batch Performance) objective = interval in which batch job must complete all processing Optimistic: BP < W - O Conservative: BP < (W - O) / 2 Compromise: BP < (W - O) / (1 + p(f)) Performance testing: set up batch runs with average/worst-case input quantities; simulate any other load

16 Interactive Systems POs: average/worst-case response time Continuous dialogue with user(s) => response time scaled to human tolerance Examples: reservation systems, business transaction-processing systems, CLIs IP (Interactive Performance) objective = interval in which software must produce a response Average utilization level of all resources (e.g. client, network, server) should be specified POs: critical interval Real-time Systems (1) Must keep pace with external physical process, and respond while that process occurs, so the response can control the process Signal types: if repeating, the rate defines the response window; if critical, a response must be given within a fixed interval Two factors determine the objective: S: signaling rate I: critical interval Real-time Systems (2) RP (Real-time Performance) objective = interval in which SUT must respond to an input signal Repeating: RP < S Critical: RP < I Repeating critical: BP < minimum [ S I ] Performance testing: simulate external signal at average/worst-case frequency; simulate any other load; monitor end-to-end response time Real-time Systems (3) Ideally, monitoring software is not run on the computer running the SUT (affects results) Monitoring computer logs all input and output passing between it and the target machine Average performance times are calculated using the log

17 The End 64 17