A Template-Based Approach to Describing Metamorphic Relations

Transcription

1 A -Based Approach to Describing Metamorphic Relations Sergio Segura, Amador Durán, Javier Troya, Antonio Ruiz-Cortés Applied Software Engineering Group University of Seville Buenos Aires, MET 2017 Conclusions 1

2 Conclusions Number of Metamorphic Relations (MRs) presented in research papers 2

3 Lack of standards to describe MRs Most MRs are described informally using natural language. Key information about the MRs is sometimes omitted. Great variability in the way MRs are described. Let s use tables! 3

4 D-OSE Name Online search engines Description Online search engines Version 1 License URL Input 1 Input 2 Output 1 Name Domain Description Mandatory Name Domain Description Mandatory Name Domain Description Inputs Q String Search query Yes D String Site domain filter (introduced as a part of the query) No Outputs P List List of search results Program name Program description Program version Program license Program URL Parameter name Parameter domain Parameter description Parameter optionality Naming rules for subindexes Preconditions Examples MR-MPSite Context ID D-OSE Description Test search engine s reliability when retrieving Web pages that contain an exact word or phrase. Version 1 Source test case Inputs P A Q A Outputs Follow-up test case Inputs Q B, D B Outputs P B Preconditions 1 < P A 20 and 1 < P B 20 Input relation Q B = Q A and D B =d i being d i the domain of P A [i] with 1 i P A Output relation P A [i] in P B Example For example, let us test Google by issuing the following source query: "side effect of antibiotics in babies" Conclusions 4

5 User story As a <role>, I want <goal/desire> so that <benefit> Example: As a user, I want to indicate folders not to backup so that my backup drive isn't filled up with things I don't need. Goal Question Metric (GQM) Goal: <purpose, issue, object, viewpoint>, Question: <question(s)> Metric: <metric(s)> Example: Goal: Improve the timeliness of change request processing from the project manager's viewpoint. Question: What is the current change request processing speed? Metric: Average cycle time. 5

6 Empirical experiments Analyze <Object(s) of study>, for the purpose of <Purpose> with respect to <Quality focus> from the point of view of <Perspective> in the context of <Context> Example: Analyze the practice of mindfulness, for the purpose of evaluating its effects with respect to the performance of students in conceptual modelling from the point of view of the experimenters in the context of second-year students in the Degree in SE. These templates have something in common 6

7 Simplicity Flexibility Conclusions 7

8 In the domain of <application domain>, [where <context definition>] [assuming that <constraints>] the following metamorphic relation(s) should hold <metamorphic relation name 1 >: if <relation among inputs/outputs> then <relation among inputs/outputs> <metamorphic relation name n >: if <relation among inputs/outputs> then <relation among inputs/outputs> Let s see some examples 8

9 Examples Tsong Yueh Chen, Fei-Ching Kuo, Wenjuan Ma, Willy Susilo, Dave Towey, Jeffrey Voas, and Zhi Quan Zhou., "Metamorphic Testing for Cybersecurity," in Computer Examples Z. Q. Zhou, S. Xiang and T. Y. Chen, "Metamorphic Testing for Software Quality Assessment: A Study of Search Engines," in IEEE Transactions on Software Engineering

10 Examples Mikael Lindvall, Dharmalingam Ganesan, Sigurthor Bjorgvinsson, Kristjan Jonsson, Haukur Steinn Logason, Frederik Dietrich, and Robert E. Wiegand. Agile metamorphic model-based testing. In Proceedings of the 1st International Workshop on Metamorphic Testing (MET '16). Sergio Segura, Amador Durán, Ana B. Sánchez, Daniel Le Berre, Emmanuel Lonca and Antonio Ruiz-Cortés. Automated Metamorphic Testing of Variability Analysis Tools Software Testing, Verification and Reliability Validation Year Authors Title Domain #MRs 2002 T.Y. Chen et al. Metamorphic Testing of Programs on Partial Differential Equations: a Case Study Numerical programs T.H. Tse et al. Testing Context-Sensitive Middleware-Based Software Applications Embedded systems W.K. Chan et al 2010 K.Y. Sim et al X. Xie et al F.-C. Kuo et al. Finding failures from passed test cases: improving the pattern classification approach to the testing of mesh simplification programs Detecting Faults in Technical Indicator Computations for Financial Market Analysis Testing and validating machine learning classifiers by metamorphic testing Testing Embedded Software by Metamorphic Testing: a Wireless Metering System Case Study Computer graphics 3 Financial software 2 Machine learning 2 Embedded software S. Segura et al. Automated metamorphic testing of variability analysis tools Software variability Z.Q. Zhou et al. Metamorphic Testing for Software Quality Assessment: A Study of Search Engines Search database T.Y. Chen et al. Metamorphic Testing for Cybersecurity Cybersecurity M. Lindvall et al. Agile Metamorphic Model-based Testing Search database 2 Total 17 10

11 specification Versioned technical report LaTeX template Conclusions 11

12 Conclusions Thanks! 12