Existing Model Metrics and Relations to Model Quality Parastoo Mohagheghi, Vegard Dehlen WoSQ 09 ICT 1
Background In SINTEF ICT, we do research on Model-Driven Engineering and develop methods and tools: MODELWARE project finished in 2006. MODELPLEX project started in 2006 with the goal of extending MDE to development of complex systems. QiM (Quality in MDE) in 2007-2008. Other projects on SOA, embedded systems, variability modelling etc. We face the same question in many projects: What is a good model and how can we measure this goodness? ICT 2
Literature review on model quality and model metrics In 2007, we searched a number of publication channels to answer: What does model quality mean? How can it be measured? 40 sources (articles, books, one PhD thesis) were identified regarding model quality. From the analysis, 6 quality goals were identified. A paper is under publication in IST (Information and Software Technology Journal). Regarding model metrics, the number is the lower. The final list depends on how we decide to include or exclude papers. We will publish more details in future. ICT 3
Model quality MDE is model-centric: -> Quality of generated software depends on the quality of models and performing transformations that preserve quality or even improve it. Assessing quality has two parts: Measuring: what metrics and how? Judging: based on the intended function and some baseline data. Models are developed for various purposes: Visualizing design; As sketches for coding; Code and model co-exist and are synchronized; Full executable models. Thus model quality depends on the purpose. ICT 4
The 6C (model quality) goals based on literature review; precise definition depends on the context Completeness; as having all the necessary information and being detailed enough for the purpose of modelling Real World (domain and organization) Correctness; as including correct elements and correct relations and not violating rules and conventions correctness Modelling language completeness correctness <perceives> <uses> confinement changeability Modeller <uses> Modelling tool Changeability; as being easy to modify (since real world changes) comprehensibility Model correctness <uses> <uses> Confinement; <elicits & as being in agreement with the purpose develops> of modelling and the type of system Rules & guidelines Analysis & generation tools <generates> consistency comprehensibility <uses> Consistency; as no contradictions in the model Human users (customers, developers, etc.) Code <develops> Comprehensibility; as being understandable by the intended users ICT 5
Differences between measuring model and code quality 1. Models and source code often differ in abstraction level, precision, completeness, consistency and correspondence to the ultimate system. Even in MDE approaches where models are the primary artifacts of software development and source code is generated from them, some details are added during transformations. 2. If modeling is performed for capturing, abstracting and communicating domain knowledge or Not requirements, all source code model metrics should focus on characteristics can important easily earlier be transferred in the development to models; life cycle. 3. Same quality goals mean differently for different models. but a significant amount For example completeness of a domain model means including all the necessary elements of the domain while can at be the reused. design level it means including all the details necessary for code generation. 4. Often a system is modeled in several diagrams from multiple viewpoints and it is necessary to define which diagram contains the right information for evaluating a quality goal. 5. Metrics collected from source code are often language dependent while models offer the possibility of evaluating some characteristics independent of the implementation language. ICT 6
State of the art regarding model metrics Three classes are identified: Model size metrics: counting the number of classes, use case, associations and so on. Metrics on design and implementation models: the quality of detailed design and implementation which is often discussed to be important for later maintenance. Most reuse of code metrics. Other more specific model metrics: for example related to aesthetics of models and completeness. Since UML is almost the de facto modeling language in industry, researchers have defined several metrics suites targeting UML models directly. ICT 7
Model size metrics: goals Lange defines the goals of measuring model size as: Comparing models. Comparing the size of models, e.g. different versions of the same model, different models for the same system or models for different systems. Measuring progress. Answering questions like How fast is our model growing?. Prediction. Predicting for example the effort needed for a project or the size of the implementation of the system. Note that size is the main driver in most effort estimation models such as in COCOMO. Description. Describing the characteristics of a model. For example, in empirical studies it is necessary to describe carefully characteristics of the model under study. ICT 8
Model size metrics: types Absolute size the number of elements, like use cases, sequence diagrams or classes in a diagram. Relative size ratios between absolute size metrics, such as the number of sequence diagrams divided by the number of use cases, and can be used to compare proportions of different models with each other and to give an indication about the completeness of the models. Complexity a subset of the absolute and relative size metrics, although no specific metrics is proposed. Functionality Lange suspects that there exist relations between functionality and model size metrics that say something about a model s completeness or the model s level of abstraction. Established metrics for functionality are Function Points and Object Points. Specific Use Case Points are also proposed by Karner, who employs use cases as a representation of system functionality. Reuse According to Lange, A reuse metric can only be applied to a UML model that makes use of a profile to denote reuse (such as OMG s Reusable Asset Specification). A simple metric could be the percentage of reuse. ICT 9
Metrics on design and implementation models Metrics for Object-Oriented (OO) systems applied on models: Coupling is included in most OO metrics suites. Also receiving notable focus is inheritance (for example the number of parents or the depth of inheritance tree), cohesion, polymorphism, information hiding and complexity. Reijers describes a cohesion metric that can be used to evaluate operations in a workflow and take decisions on whether to split or combine them, which is related to the quality of design ICT 10
Other work Berenbach has developed a tool called DesignAdviser that reports defects such as missing associations and class not instantiated. Also other tools that can monitor models and give warnings. Metrics for aesthetics of diagrams and their understandability such as the number of elements in a diagram or the number of crossing lines. These are also size metrics but related to other characteristics of models. ICT 11
Observations Counts of elements and relations do not often reveal substantial information by themselves unless linked to other attributes. As an example, counting the number of interactions in a class, can say something about coupling. For models developed and used early in the development life cycle Comprehensibility, relative completeness and consistency (in order to avoid misinterpretations) are most important. Few metrics here; mostly evaluated by inspections. Models developed and used later for detailed design and implementation: should be correct, complete, consistent and changeable, especially if they are used for generating source code. Size metrics can be used for evaluating completeness. Tools can detect or prevent inconsistencies and some violation of rules (related to correctness). Design metrics such as the OO ones are proposed to be applied on models to evaluate the quality of design. Changeability can for example be measured by evaluating the structure of a model (number of packages etc.). ICT 12
Related work on quality assurance or verification techniques Special inspection techniques such as OORT (Object- Oriented Reading Techniques). Verifying consistency between diagrams, for example class vs. state diagrams, manually or by tools. Defining formal syntax and semantic for different views and verifying consistency by analysis or monitoring. Using OCL to define constraints. Using formal methods such as graph transformation rules. ICT 13
What quality goals are especially important in MDE? Solheim and Neple (2006) emphasize two characteristics: Transformability as completeness (correct according to the domain), relevance (containing no extra elements), precision for transformation, well-formedness or compliance to the model s metamodel. Modifiability as traceability of model elements, and well-designedness or not being too complex. They do not specify metrics related to these caharcteristics. Liu et al. (2004): Consistency between views at same abstraction level (horizontal consistency) Consistency between model and its refinements (semantically) (vertical consistency) Traceability: changes should propagate Integration of different views before code generation Other required characteristics: modularity and easily extensible (related to changeability). ICT 14
Future work Visualizing metrics can help in analysis (there are a few tools), and tools detecting and reporting defects can improve the quality of models. Gaps: identifying metrics for early models, linking model metrics to quality goals by using theories or best practices (why and to what degree a metrics can measure if a given model fulfills a quality goal), collecting empirical data that helps in interpreting metrics. Using guidelines for developing high-quality UML models for specifying new metrics. ICT 15