Using Models of Partial Knowledge to Test Model Transformations

Size: px

Start display at page:

Download "Using Models of Partial Knowledge to Test Model Transformations"

Kory Montgomery
5 years ago
Views:

1 Using Models of Partial Knowledge to Test Model Transformations Sagar Sen 1, Jean-Marie Mottu 2, Massimo Tisi 1, Jordi Cabot 1 1 ATLANMOD, Ecole des Mines, Nantes 2 AeLoS, Universite de Nantes, France 1

2 Outline Introduction: Model Transformation Testing Case Study: Class2RDBMS Problem: Tediousness of Creating Test Models Approach: Creating Partial Models and Automatically Completing them. Effective? Experiments based on Mutation Analysis Conclusion 2

3 Introduction Invariants Effective test models! Model Transformation Testing 3

4 Outline Introduction: Model Transformation Testing Case Study: Class2RDBMS Problem: Tediousness of Creating Test Models Approach: Creating Partial Models and Automatically Completing them. Effective? Experiments based on Mutation Analysis Conclusion 4

5 Case study: class2rdbms simple UML Class Diagrams class2rdbms RDBMS Models 1. Object persistence benchmark proposed in the MTIP workshop, MoDELS Input domain spec. covers all major metamodelling concepts such as inheritance, composition, finite and infinite multiplicities. 3. Invariants are both first-order and high-order, contains also transitive closure invariants 4. Transformation exercises most major model transformation operators such as navigation, creation, and filtering 5. Available in many transformation languages Kermeta, ATL, VIATRA, QVT 5

6 Input MM + Invariants is_primary: Boolean name: String 0..1 Attribute 1..* attrs Ecore Meta-model Class Association name: String type is_persistent: Boolean 1 dest 1 src 1 * Classifier name: String PrimitiveDataType association (a) classifier * ClassModel context Class OCL Invariants inv nocyclicinheritance: not self.allparents()->includes(self) inv uniqueattributesname: self.attrs->forall(att1, att2 att1.name=att2.name implies att1=att2) context ClassModel inv uniqueclassifiernames: self.classifier->forall(c1, c2 c1.name=c2.name implies c1=c2) inv uniqueclassassociationsourcename : self.association->forall(ass1, ass2 ass1.name=ass2.name implies (ass1=ass2 or ass1.src!= ass2.src)) (b) (a) Input metamodel MMi : Simplified UML CD (b) A subset of all invariants on MMi 6

7 Outline Introduction: Model Transformation Testing Case Study: Class2RDBMS Problem: Tediousness of Creating Test Models Approach: Creating Partial Models and Automatically Completing them. Effective? Experiments based on Mutation Analysis Conclusion 7

8 Tediousness of Creating Test Models 8

9 Tediousness of Creating Test Models : ClassModel classifier :Class classifier : Class :Class classifier : Attribute = A is_persistent = false name = B is_persistent = true name = D is_persistent = true is_primary = false name = attd attrs attrs src : Association : Attribute is_primary = true name = atta : Attribute is_primary = false name = attb dest : Class name = assodc classifier association classifier type type : PrimitiveDataType name = String type type name = C is_persistent = false attrs : Attribute is_primary = false name =attc A Human-made Test Model Problems 1. Must conform to metamodel MMi 2. Must satisfy MMi invariants 3. Must satisfy pre-conditions pre(mt) on model transformation (class2rdbms in our case) 4. Must contain (partial) knowledge to find bugs 9

10 Outline Introduction: Model Transformation Testing Case Study: Class2RDBMS Problem: Tediousness of Creating Test Models Approach: Creating Partial Models and Automatically Completing them. Effective? Experiments based on Mutation Analysis Conclusion 10

11 Partial Models to Capture Testing Intention :Class is_persistent=true :Class :Class src :Association dest (a) Expressing pure testing ideas! Hit Enter to Generate Complete Test Model? 11

12 Complete Test Model! ClassModel type type 5:PrimitiveDataType type 7:Class attrs -3:Attribute is_persistent=true is_primary=true 6:Class is_persistent=true -8:Class is_persistent=true attrs 5:Attribute is_primary=true 6:Attribute is_primary=true attrs src 4:Association dest 12

13 Question 1: Where to find partial knowledge for testing? 13

14 Where to find partial models of testing knowledge? Humans specify partial testing knowledge as model fragments Humans specify partial testing knowledge as constraints Static analysis of (faulty regions) a model transformation Analyzing (manually/automatically) existing large (human-made) test models 14

15 Question 2: How to express partial knowledge about testing? 15

16 Relaxing the Input Domain (Ramos et. al. Models, 2007) Metamodel MM Metamodel Relaxation Metamodel MM_r conforms To Partial Model m_p (a) (b) (a)relaxation of MMi to specify partial models (b) Relaxed MMi Some Properties of Relaxed MM (a) Class can have 0..* attrs instead of 1..* (b) dest 0..1 and src 0..1 instead of 1 (c) type 0..1 instead of 1 Partial Models are Graphs of Typed Objects that need not to conform to MMi 16

17 Relaxed Input MM Partial Test Model :Class is_persistent=true :Class :Class conforms to src :Association dest (a) Classes have no name or attribute (wouldn t be valid in the real input domain) 17

18 Question 3: How to complete partial models of testing knowledge? 18

19 Approach : Partial Model Completion Invariants /pre-conditions rewritten to Alloy facts Meta model MM Pramana Transformation to Alloy Executable Alloy Model A f Alloy Analyzer Partial Model m_p rewritten to an Alloy predicate Scoping Strategy No Solved? Yes Complete Model(s) m_c Operation MM to Alloy Invariant/Pre-condition to Alloy Partial Model to Alloy Completion of Alloy Model A_f Scoping Strategy Degree of Automation Automatic Manual Manual/Automatic Now Automatic Default Minimal/Manually Tunable 19

20 Metamodel + Inv. to Alloy Source Element Classes to Alloy Signatures Implicit constraints to Alloy Facts Invariants/pre-conditions to Alloy Facts Target Alloy Construct sig Class extends Classifier { is_persistent: one Bool, : lone Class, attrs : some Attribute } fact associationcontainment { all a:association a in ClassModel.association } fact nocyclicinheritance { no c: Class c in c.^ } More information: Sen S., Ph.D. Thesis: Automatic Effective Model Discovery, 2010 Sen S., Baudry B., Mottu Jean-Marie. On Combining Multi-formalism Knowledge to Select Models for Model Transformation Testing. ICST'08 20

21 Approach : Partial Model Completion Invariants /pre-conditions rewritten to Alloy facts Meta model MM Pramana Transformation to Alloy Executable Alloy Model A f Alloy Analyzer Partial Model m_p rewritten to an Alloy predicate Scoping Strategy No Solved? Yes Complete Model(s) m_c 21

22 Partial Model to Alloy Partial Model Rewritten Alloy Model :Class src :Class is_persistent=true :Association dest :Class pred PartialModel { some c1 : Class, c2: Class, c3:class c1!=c2 and c2!=c3 and c1.is_persistent=true and c2. = c1 and c3.=c1 and c2!=c3 and c2!=c1 and some a1: Association a1.src =c2 and a1.dest=c3 } Manually Specified Scoping Strategy: run partialmodel for exactly 1 ClassModel, 4 int, 3 Class, 3 Attribute, 1 PrimitiveDataType, 1 Association 22

23 Approach : Partial Model Completion Invariants /pre-conditions rewritten to Alloy facts Meta model MM Pramana Transformation to Alloy Executable Alloy Model A f Alloy Analyzer Partial Model m_p rewritten to an Alloy predicate Scoping Strategy No Solved? Yes Complete Model(s) m_c 23

24 Automatically Completed Partial Model ClassModel type :Class is_persistent=true type 5:PrimitiveDataType type 7:Class attrs -3:Attribute is_persistent=true is_primary=true :Class src :Association dest :Class 6:Class is_persistent=true -8:Class is_persistent=true attrs 5:Attribute is_primary=true 6:Attribute is_primary=true attrs src 4:Association dest Satisfying MM+invariants+pre-conditions at least one primary Attribute in a Class.. Classes inheriting from persistent are also persistent.. Classes and Associations are contained in ClassModel 24

25 Outline Introduction: Model Transformation Testing Case Study: Class2RDBMS Problem: Tediousness of Creating Test Models Approach: Creating Partial Models and Automatically Completing them. Effective? Experiments based on Mutation Analysis Conclusion 25

26 Experimental Setup 1. Mutation operators applied to class2rdbms create 200 mutants (6 equivalent mutants) 2. Mutant operators on filtering, navigation, and creation operations (Mottu et. al. ECMDA 06) 3. 8 human-made models were available from Mottu 06 to kill 100% mutants concise partial models were manually deduced from these 8 human-made models. (To be sure that they have the same testing knowledge and are comparable) non-isomorphic test models were generated for each partial model using Alloy symmetry breaking scheme. Total = 10 x 14 = 140 test models 26

27 Size of Partial Models partial model P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11 P12 P13 P14 from reference model M1 M1 M1 M3 M5 M6 M3 M4 M7 M3 M4 M2 M1 M8 M1 M2 M3 M4 M5 M6 M7 M8 #ClassModel #Class #name attr #is_persistent attr # relation #Association #name attr #src relation #dest relation #Attribute #is_primary attr #name attr #type relation #PrimitiveDataType #name attr Total #concept Total #concept per set of partial models: 109 of models: 231 Partial models are compressed versions of existing human-made test models! 27

28 Mutation Analysis: Detecting injected bugs Human made model Mutation Score of the model in transformations Partial models matching the human made model Mutation Score of the completed partial models Mutant killed by model also killed by completed partial models M1 57.2% P2, P3, P % 100% M2 57.2% P1, P % 50.5% M3 49% P1, P4, P7, P10, P % 89.3% M4 57.2% P1, P8, P11, P % 98.2% M5 35.6% P1, P5, P % 100% M6 61.3% P1, P2, P4, P6, P8, P12, P13, P14 83% 100% M7 47.4% P1, P9, P % 100% M8 58.2% P1, P12, P % 100% 8 models 100% 14 partial models 100% 100% Smaller partial models when completed detect 100% of the bugs detected by large meticulously designed test models. 28

29 Outline Introduction: Model Transformation Testing Case Study: Class2RDBMS Problem: Tediousness of Creating Test Models Approach: Creating Partial Models and Automatically Completing them. Effective? Experiments based on Mutation Analysis Conclusion 29

30 Conclusion We present a scientific experiment to demonstrate that partial models are sufficient and effective to specify test intentions Partial models are compact, clearly capture testing intentions, and easy to read and understand Completion of partial model is automatic. Transformation to Alloy is currently semi-automatic. Partial models when completed give 100% mutation score w.r.t. humanmade models with the same knowledge. We would like to automate transformation of a certain class of partial models to Alloy. Promote specification of partial testing knowledge that can be automatically completed. 30

31 Thank you. Pleased to Address Your Questions. 31

Automatic Model Generation Strategies for Model Transformation Testing

Automatic Model Generation Strategies for Model Transformation Testing Sagar Sen 1, Benoit Baudry 1, and Jean-Marie Mottu 1 INRIA, Centre Rennes - Bretagne Atlantique, Campus universitaire de beaulieu,