Software Architecture-based Testing and Model-checking - ECI 2005, University of Buenos Aires - Course Web-site: [

Transcription

1 Software Architecture-based Testing and Model-checking - ECI 2005, University of Buenos Aires - Course Web-site: [ Lecture 5: Model-Checking driven Testing Lecturer: Henry Muccini Assistant Professor, Computer Science Department University of L'Aquila -Italy muccini@di.univaq.it [ [ Copyright Notice» The material in these slidesmaybe freelyreproduced and distributed, partiallyor totally, asfar asan explicit reference or acknowledge to the material author ispreserved. Henry Muccini 2 Acknowledgment» This work is joined with Patrizio Pelliccione (University of L Aquila), Pierluigi Pierini (Siemens CNX), and Antonio Bucchiarone (ISTI CNR)» Published in ITM 2004, Lecture Notes in Computer Science, LNCS, vol. 3236, pp (2004). 3 1

2 Agenda» Introduction and Motivations: - ModTestand TeStor» ModTestin Siemens C.N.X.» Challengesand Future Work 4 Considerations» Different Analysis require different notations - testing, model-checking, performance and reliability analysis require specificformalisms and annotations on UML-basedmodels [UML&SA04] - There is a hugerelationship between whatwespecify and whatkind of analysis wemayperform > Modeling for documentingvs. Modelingfor analysis But 5 Considerations» different Analysis techniques are usually related - supposing an industry is interested in deadlockanalysis and performance analysis, a complete result is obtained only using two different ADLs.» Modelingis really expensive - we want to reuse the same model for manyanalysis techniques 6 2

3 Integrating Analysis Techniques validate the SA model conformance with respect to selected functional properties Charmy Project [ provide confidence on the implementation fulfillment to its architectural spec SA-based Testing 7 ModTest: Model-Checking driven Testing [ITM04, CBSE05,TR_March05] 8 Software Model Checking and Software Testing» Model Checking: - It checks whether a certain property is valid for a certain model of a system [Ruys_PhDThesis] > Model checking is a model-based, automatic technique that, given a finite-state model M of a system and a property P, checks the validity of P in M» Testing: - Software testing consists of the dynamicverificationof the behavior of a program on a finite set of test cases, suitably selectedfrom the usuallyinfinite executionsdomain, against the specified expected behavior [Bertolino_SWEBOK] 9 3

4 Advantages Model Checking exhaustive approachto completelycheckthe system -completely automated skills on formalmethods state explosion problem onlymodel-based Testing clever selection of limited and relevant test cases -usuallyleft to the tester experience skillson formalmethods generally not required test case identification problem code-based, model-based, specification-based 10 Goals and Motivations > 1/2» General Goal: - integration of model-checking and testing to provideanuseful tool totest moderncomplex software systems» In related approaches: - By using model-checking features, counter-examples are produced, successively used to derive test cases - Main Limitations: > P1 : due to models complexity, the model checker techniques become inapplicable, thus not allowing to identify test cases; > P2 : even on little examples, the number of generated test cases causes the intractability 11 Goals and Motivations > 2/2» OurGoal: To apply Model-checking and Testing in a Software Architecture-based (SA) process, where: > Model-checking techniquesare used to validatethe SA model conformance withrespecttoselected functional properties + avoiding state explosion problem 12 > while testing techniquesare used to provide confidence on the implementation fulfillmenttoits architectural specification +Test case selection driven by model-checking 4

5 Our Proposal validate the SA model conformance with respect to selected functional properties Charmy 13 provide confidence on the implementation fulfillment to its architectural spec SA-based Testing SA-based model-checking Charmy Project [ FMS Model of the SA» Model-checking: validate the SA model conformance to selected functional properties Property expressedthrough scenarios 14 > It checks whether a certain property is valid for a certain FSM model of a system [Ruys_PhDThesis] Charmy: Informally Requirements Software Architecture Translate them into Buchi Automata Property M.C. 15 Translate them into Promela SPIN True or False 5

6 SA-based Testing SA-based Testing [ICSE01, FASE04,TSE04]» Goal: - Provide confidence on the implementation fulfillment to its architectural specification 16 SA-based Testing in ModTest 17 Question» Howcan we integrate both techniques? - Rephrasing: howcan weusemodel-checkingresultsforgeneratingtest specifications?» Usually: - Byfocussingon model-checkingdriventesting, counter-examples are produced and successively used to derive test cases» Our idea: - To usethe properties to derive test specifications, by recovering missing information from state machines 18 > In this way, the SA is validated withrespect to requirements and the implementationconformance to the SA is tested - SA-based analysis process 6

7 The TeStor Algorithm» TeStor: - Inputs: > sequence diagram (insd) representing properties - test generation directives > components behavioral models (in terms of components' state machines), - Outputs: - the model of the software under test > a set of test specifications, still in the form of sequence diagrams (outsd) 19 TeStor objective Test directives Model of the system Test specification m9 m3 20 m8 m1 TeStor objective Test directives Model of the system Test specification m9 m3 21 m8 m1 7

8 How TeStor can achieve this goal» HowTeStor could work: - Parallelcomposition of the state-machines models, and their traversal (likemodel-checkers), or - Challenge: to avoid parallelcomposition > To limit state explosionproblems 22 The Algorithm» The TeStor algorithm can be split into two macro-steps: - State machines(sm) Linearization > Decomposes SM in a set of linear traces - Test SequenceGeneration 23 > Looks at eachlinearized trace in order to identify the sup-trace of the insd. > This macro-step is composed by a Validation part, whichchecks when and how sup-traces need to be combined to produce the outsd. > The Merge algorithm glues together the validated traces Linearization» Startingfrom the initialstate in the components state diagrams; - It createsa trace at anytime a state witha branchisreachedor an alreadyvisitedstate isreached» The algorithm is iterated, startingfrom the previously reached state, untilunvisited statesstillexist. m8 m1 m7 S3 S4 S5 S0 m8 S0 S0 S5 S1 S4 S3 S2 24 m9 S3 S3 m9 m3 m10 S0 S1 S2 S3 8

9 From State Machines to Traces 25 Test Sequence Generation» Startingfrom the tracesgenerated from each component state machine -Foreach message mi in the insd, TeStor identifies those traces which contain it Property > Validate algorithm C2 s i.s. = S0 C3 s i.s. = S0 C3 C2 C1 C4 m9 m3 m10 m8 m1 m7 SEA e) Property Group to be verified S0 m9> S1 S0 m9> S0 26 S0 --m9-> S1 --m3->s2 --m10->s3 --m9->s3 Test Sequence Generation» Startingfrom the tracesgenerated from each component state machine -Foreach message mi in the insd, TeStor identifies those traces which contain it Property > Validate algorithm > Merge algoritm C3 C2 C1 C4 m9 m3 m10 m8 m1 m7 SEA e) Property Group to be verified 27 9

10 Supporting Tools The TeStor algorithm hasbeen implemented hasa plug-in component forcharmy, ourvalidation frameworkfor architecturalanalysis. 28 ModTest: Model-Checking driven Testing Model of the system Test directives validate the SA model conformance with respect to selected functional properties Charmy [ TeStor 29 provide confidence on the implementation fulfillment to its architectural spec SA-based Testing ModTest in Siemens C.N.X

11 Siemens CNX : main research areas» SiemensCNX S.p.a.is a Siemens R&Dlab; its mission is the design and developmentof SDH (1) TLC equipments» relevantresearchareas: - Formal design methodologies - System and software performance analysis - Test design methodologies - Intelligentagent application - Network Processors - Ethernetfirst mile - Opticsand cristal properties - Electromagneticcompatibility 31 1) SDH Synchronous DigitalHierarchy Test Design Methodology > objective»improvethe tetsdesign process System Requirements System Requirements System Architecture Design feedbacks System Tests Design System Architecture Design Review NOK NOK Review Model Check NOK OK OK OK Implementation Modification Test Implelemtation Equipment Develop Requests Design Test Generation Engine Implementation/ Equipment Develop NOK OK Test Exec Test Exec OK NOK 32 System Release a System Release b 33 Case Study > some definitions» A SDH Network Element (NE, i.e. equipment) is modeled using the functional model standardized by ETSI and ITU-T.» The functional model is built around two specific concepts: - network layering", with a client/server relationship between adjacent layers; - atomic functions (connection, termination and adaptation), to specify the behavior of each layer.» applicative functions should reside on top of a layer providing specific processing on transmitted information» A virtual network connection can be established between mate network layers (or atomic/applicative functions) belonging to different NEsby means of transport services offered by the underlying layers 11

12 ?call Case Study > some definitions 34 Case Study > EOW architecture» The EOW supports a telephone link between NEsusing dedicated voice channels defined on the SDH frame (i.e. the EOW SubNetwork [eowsn]); HS HS EOW Node CM eowsn» An EOW node consists of: CM - A handset (HS) that manage the physical phone device; HS CM 35 - a conference manager (CM) that control the handset connection to the EOW subnetwork; Case Study > EOW components localnumsign1 CM1 callrequest HS1 call1 eowkeydigit localnumsign2 call CM2 Request e HS2 call2 o eowkeydigit w S N localnumsign3 call CM3 Request HS3 call3 eowkeydigit S4 Handset (HS) timeout Congestion Busy of fh o o k config onhook offhook Init Config Idle Check [digit==0]/!call timeout onhook Ringing o nh oo k onhook timeou t Dialing [(digit==x)&&(number!=1)]/!callrequest(digit,number)?callrequest(digit,number) S1? lo c a ln u m S ig n /c b u s y = f a ls e!callrequest(digit,number) [cbusy==true] offhook [(d ig it= = x ) & & (n u m b e r! = 1 ) ] /!c a ll/c b u s y = tru e onhook [cbusy==false]!loca ln um S ig n?call?call eowkeydigit(digit,num ber)!loc alnumsign InConference S2?eowKeyDigit(digit,number) S1 S2!callRequest(digit,number) S3 eowsubnetwork Conference Manager (CM) 36 12

13 Case Study > Functional Requirements» EOW FunctionalRequirements/Properties: A) when an operator makes a call dialling a selective number, the target operator must receive the call. B) it must be possible to entera busy conference (with the special number-sign key) when a call isalreadyin progress. 37 C) It must be alwayspossible to exit to the conference (cleanlyterminate a call). Case Study > Functional Requirements 38 Case Study Results Interactivesimulation& Test generation» Simulation withoutconstraintwillresultin an intractablenumberof traces;» Simulation conditionedby the given properties;» Up to 36 test traces was extracted; - Most of themare eligible tobecometest cases;» Test selection focuson some optimization criteria like: - Maximization of system coverage, - Minimization of globalnumber of tests 39 - Minimization of test lenght(i.e. number of steps) 13

14 Some Considerations Advantages:» Model complexity and the state explosion reduction obtainedby: SAlevelmodel chekcing, iterative approach and abstraction ;» Charmy easy touse, practical, approach tomodel-checking, hiding the modeling complexity;» interactivesimulation wemay identify traces of interest fortesting the wholesystem or just a relevantsubsystem.» test specifications are identified from the architectural model (not from requirements) - Easiest alignment between SA and Test specifications; 40 - Easiest control of the design steps and evolution Some Considerations Limitations:» The Test Generator Engine can be automated; its implementation is in progress.» The executable tests implementation from the generated test specifications isnot automated yet. Weapproach this point with the aim to automate also thisstep.» Modelsdimension and complexitystillremain an issue, even if the iterative approach reducesthe state explosion problem. 41 Challenges 42 14

15 Challenge: SA-centric Analysis Process 43 Charmy and TeStor [TR_March05] 44 QuARS, ModTest, CowTest and UIT [QoSA05] 45 15

16 Future Work 46 Lessons Learned» From this experience we learned integration is possible: - Analysisintegration: ModTest > Future work will integrate other analysis techniques» But we also need: - Notation extension: It is possible to extendthe same UMLbased notation - Tool extension: wecan adda plugin implementingthe new analysis technique 47 Dually: Putting in Synergy UML 2.0 and ADLs» A frameworkwhich i) identifies a core set of architectural elements always required (A0), ii) creates an UML profile for A0, iii) provides extensibility mechanisms to add modeling concepts needed for specific analysis. iv) describes how semantic links mechanisms can be kept between different notations. 48 It is impossible to identify a unique 2005 language by H. Muccini for / representing ECI 2005 Course SAs Different Analysis require different notations 16

17 Notation Extension: General Idea Model Transformation = + = = = 49 Notation Extension: DUALLY [TR_May05] Identify a core set of architectural elements always required Create an UML profile able to model the core architectural elements previously identified 50 Describe how semantic links mechanisms can be kept between different notations Provide extensibility mechanisms to add modeling concepts needed for specific analysis The roles of ADL, UML and XML ADL xarch & ACME ADLs Experience UML DUALLY Profile for A 0 DUALLY modeling extensions XML xarch DUALLY XML XML extensions 51 17

18 Tool Extension: Charmy Studio 52 Wish List: Tool one: integration of analysis techniques» Integration of manyanalysis techniques UML/SA profile for Model- Checking UML/SA UML/SA profile profile for for Performance Testing Input filters Other 53» Integration of manyumlbased notations» Integration of different analysistools DUALLY standard editing and analysis tools Testing Performance feedback notation feedback notation feedback Model- Performance Checking tool Testing tool tool Modelchecking notation DUALLY Profile for A 0 Modeling extensions DUALLY XML Semantic Relations XML extensions Plugged into DUALLY Editing tools Analysis tools Other 18