Model Based Software Testing GOTCHA-TCBeans AGEDIS

Transcription

1 Model Based Software Testing GOTCHA-TCBeans AGEDIS Alan Hartman Kenneth Nagin 06/ Copyright International Business Machines Corporation ALL RIGHTS RESERVED.

2 Outline Background and motivation Success stories Methodology GOTCHA-TCBeans AGEDIS FSP testing with GOTCHA-TCBeans Model Based Testing

3 HRL hardware/software validation history HRL is COC for hardware testing ( ) CFSM hardware test generator ( ) Genevieve project- GOTCHA born ( ) TestMaster/GOTCHA-TCBeans bake-off (6/99) GOTCHA-TCBeans adopted by IBM STCL (7/99) GOTCHA-TCBeans parallel development & deployment (98-02) AGEDIS project starts (11/00) Patents- 6 Published papers- 6

4 Defect cost over time (Apar $15-40,000) $14,000 85% %Defects Introduced in this phase Percentage of Bugs $25 Coding $130 Unit Test $250 Funct Test $1000 Field Test Post Release % Defects found in this in phase $ Cost to repair defect in this phase Source: Applied Software Measurement, Capers Jones,1996

5 Deployments ESW (Austin) - in-progress (new) IGS (Vancover) - in-progress (new) MQSeries (Hursley) - in-progress ( ) Shark (Haifa) - in-progress ( ) Amit (Haifa) - in-progress ( ) Speech Technologies (China) - completed (2002) Zos-System (Poughkeepsie) - completed (2001) Zos-Extentions (Poughkeepsie) - 3 line items completed ( ) CallPath (RTP) - completed successfully ( ) Host On Demand (RTP) - completed successfully (1999) Many pilots

6 Sysplex File System (OS-390) Retest of Sysplex functions Modelling and translation by testers Conclusions: Comparison Original test: 18 bugs,12 PM Pilot test: 15 original bugs + 2 escapes, 10 PM (INCLUDING learning curve) Efficient way to free the tester for creative testing Replaces a large part of the manual test case writing Status: completed Contact: Ann Totten DEFECTS BY SEVERITY # % DEFECTS BY # % ODC TRIGGERS Coverage Variation Sequencing Interaction Load 1 5.8

7 Websphere Contact Center Director (CallPath) Live test of Java based Call Center Modelling and Test Execution Function Test Regression Test System Test: Reusing function test distributing test on many hosts multiplying test clients Status: completed Contact: Gary Bosko DEFECTS BY SEVERITY # % DEFECTS BY # % TRIGGERS Coverage Variation Sequencing Interaction Reliability Recovery 2 5.4

8 ESW's Flexible Service Processor (FSP) test ESW's Flexible Service Processor (FSP) Components Under Test State Manager Event Notifier Challenges Asynchronous Simulation environment Hardware environment Status: in-progress 4 Defects discovered (4 code, 1 simulator) Contact: Gerald Kreissig, Rahul Yaksh

9 GOTCHA-TCBeans Process (simplified) Design Design Interface System Specs. Legend 1.Modelling Text Editor State Machine Model 2.Generation GOTCHA Abstract Test Suite 3.Translation TCBeans 4.Execution Code Test Scripts Executor Data 5.Execution Tool Bug Interface TCBeans Code System Under Test Test Execution Log

10 Benefits of model-based testing technology Starting from specification Involves testers early in the ESG development process Teams testers with developers Forces testability into product design Building behavioural model and test interface Finds design and specification bugs - before code exists The model is the test plan - and is easily maintained Automated test suite generation Coverage is guaranteed - increases testing thoroughness Zero test suite maintenance costs Automated test suite execution Finds code and interface bugs Includes a framework for the testing of distributed applications Reduces test execution costs

11 Example - File system Users Open_for_Read Open_for_Write Close Accept_Command Reject_Command File Status file file 3 file 2 1 Files File Control Blocks

12 File system - FSM file file 3 file 2 1 CCC CCC CCC R f1 u1 C f1 u1 file file 3 file 2 1 RCC CCC CCC Actions: R, W, C Files : f1, f2, f3 Users: u1, u2, u3 State: CCC file file 3 file 2 1 CCC W f2 u2 RCC W f2 u3 file file 3 file 2 1 CCC CCC CWC R f2 u3 file file 3 file 2 1 CCC CCC CWR Transition: R f2 u3

13 Behavioural model Ruleset f : fileid_t; u: userid_t Do Rule "Open_for_Read(int f,int u)" Begin switch system_status[f][u] case Reading, Writing: response := Unchanged; case Closed: if File_Available_For_Read(f,u) then response := Accept_Command; system_status[f][u] := Reading; else response := Reject_Command; endif; endswitch; End; End; -- of ruleset

14 Abstract Test Suite (ATS) Test Suite Browser

15 Translation & Execution GOTCHA Generation GOTCHA TCBeans A2C XML or Java User Engine Translation Execution TCBeans A2E

16 Test Execution Engine Challenges (problem) Cost of developing and supporting private engines in each lab Gap between developer and tester environment Gap between unit, function and system test System Under Test complexity Model complexity --> keep it abstract Abstract to concrete mapping Many modeling languages How to reuse models and test components to save time and $

17 Test Execution Engine (solution) General Purpose execution engine: Input: ATS (abstract), TED (concrete + extensions) Output: SET (traceable back to ATS) Multiple platform (e.g. Linux, Windows) support Multiple external interface (Java,C++,C) support Modular Design Simulator support Enhanced test distribution + centralized logging Interactive test case execution GUIs to view execution progress, inputs, and outputs Concurrent Events Testing: Object Multiplication and Stepwise synchronization Non-determinism support Asynchronous event support (next release) ConTest (next release) Data input variation test support (next release)

18 Succeeded Failed Undecided Not Executed Suite Execution Trace (SET) Test Case Status

19 Suite Execution Trace State

20 AGEDIS - Automated Generation & Execution of tests for DIstributed component-based Systems Consortium Partners & Roles IBM - HRL imbus AG Intrasoft SA Oxford University France Telecom IBM Hursley Universite Joseph Fournier Funding 4.3M Euro for 3 years 50% EU with matching

21 AGEDIS Mission, Objectives, Status Mission Automate software testing of distributed component-based systems Industrial strength, leading edge technology Products freely available throughout IBM Objectives Create methodology, technology, and tools Tools with open public interfaces Methodology adopted within major industrial players Status Project started November case studies completed UML Modelling language defined (on public Web Site) ATS and SET defined (on public Web Site) Prototype delivered May 2002

22 AGEDIS Architecture Behavioral Model Compiler Intermediate Format Generator Test Generation Directives User Interface Analyzer Abstract Test Suite Test Execution Directives Visualizer Suite Execution Trace Executor user input software tool data exchange standard

23 Testing is changing Testers become modellers Write models not only test scripts Model becomes spec. extension & test plan Makes test plan review productive Testing starts at HLD Early bugs are cheaper to fix Reduces grunt work Attracts smart testers Increases productivity Promotes reusability