Abstract- This article presents an innovative approach to the determination of a core set of Quality Measure Elements

Transcription

1 Enhancing ISO/IEC Quality Measure Elements for wider application within ISO series Dominique St-Louis, Witold Suryn ÉTS École de Technologie Supérieure Abstract- This article presents an innovative approach to the determination of a core set of Quality Measure Elements (QME), and a comparative analysis aimed at enhancing applicability of ISO/IEC 25021, which are critical to the ISO/IEC SQuaRE series of standards. The research presents the methodology, allowing for moving from an outdated and extensive set of measures, to a core list of measures, which can be further diversified through contexts of use, parameters, and considerably simplify the development of derived measures. Keywords- Software Quality Measure Elements, Base Measure, QME Categorization, Knowledge Base Repository. I. INTRODUCTION This research identifies three major opportunities for improvement of the current ISO/IEC standard: better accessibility for the software development industry, consistency through abidance to ISO software and system engineering vocabulary [19] and ISO metrology standard [6], and finally the design of the standard itself. Complexity of ISO/IEC 25021, a similar concern to the one expressed about ISO/IEC 9126, makes using this standard difficult. It is also the reason why software quality measures remain a vague or non-existent concept for many companies to this day. Making the quality measurement-related part of ISO/IEC SQuaRE series of standards readily available, useful, inexpensive to implement, and easy to use, requires the development of a concise and scientifically sound set of basic measurement construction modules, as well as Quality Measurement Elements or QMEs. Consistent adherence to the measurement theory [6] would improve the credibility of the whole series of standards, while making it scientifically correct, increasing its reliability, and enhancing its trustworthiness amongst the scientific community. As ISO/IEC makes the basis for the development of several other standards in the SQuaRE series (ISO/IEC replace [2], [3] and [4]) the importance of its correct design is crucial. The objective of the presented research was to determine a different (than in ISO/IEC 25021) approach to defining a precise set of quality measures, which would serve as a basis for further development of the majority of derived quality measures. The results obtained provide such set of measures that are easy to understand by the software industry professionals, compliant with ISO/IEC standard and propose a design that allows for intuitive construction of derived measures. This paper describes the development of the process allowing for selecting quality measures from an existing catalogue and the method used to determine a core set of QMEs. Also, the development of a digital knowledge for software quality measures proposed over the course of this research would be one of its most important results, as it potentially leads to opportunities to further develop the SQuaRE series. II. LITERATURE REVIEW Several existing scientific articles present critical judgments of ISO/IEC software quality and measurement standards. The most renowned analyses are briefly discussed below. A. ISO/IEC 9126 and its Technical Reports In 1991, ISO/IEC 9126 was published as the first consensus on the terminology for quality characteristics of software evaluation. Between 2001 and 2004, ISO published an expanded version containing the quality model and the measures proposed for this model [8]. The current version of the ISO/IEC 9126 standard [1][2][3][4] consists of one normative reference, the Quality models [1], and three technical reports: External metrics [2], internal metrics [3], and Quality in use metrics [4]. Researchers have found that ISO/IEC 9126 standard has some major flaws, even if it seems the best available so far. For one, the terminology of ISO/IEC 9126 is not consistent with the classical terminology of measurement science and engineering, which causes a real concern for the validity of proposed measures. Also, measures in ISO/IEC 9126 in many cases are not identified and/or classified as or derived measures, which stays in conflict with ISO/IEC measurement information model [8]. ISO standards [19] and both refer to ISO/IEC [6] definition of terms for and derived measures which are: Base measure: measure defined in terms of an attribute and the method for quantifying it. A measure is functionally independent of other measures. Derived measure: measure that is defined as a function of two or more values of measures. Abran et al. [8] analyzed the ISO/IEC 9126 standard and suggested ways to improve it: The standard should comply with the ISO and general terms of metrology [10][17].

2 Measures in [1][2][3] and [4] are merged and should be reorganized into and derived measures to comply with ISO/IEC [10]. Map the information to the measurement model provided by ISO/IEC [6]. Measurement results described by ranges such as [0..1] could be presented qualitatively. A score of 100% could be described as excellent, and a score of 80% as very good [11][12]. Other researchers have also demonstrated the impairments in the structure of the ISO/IEC 9126 quality model. Jung et al. argued that the use of the six sub-characteristics of Compliance leads to two kinds of results: (1) each compliance sub-characteristic belongs to its characteristic as defined in ISO/IEC 9126, (2) the six compliance subcharacteristics form a separate dimension [13]. The results of their research showed an ambiguity in the structures of the characteristics and sub-characteristics in ISO/IEC As for Abran et al. [15] they suggest reviewing the structure of the Usability model by adding Learnability and Security, d on their analysis of ISO/IEC SQuaRE. Chua and Dayson [14] discovered that the Feasibility characteristic adds specific factors such as Consistency, Simplicity, Legibility and Color use. They also suggest adding the Help sub-characteristic to Usability characteristic and the global characteristic of User satisfaction. In the subject of the implementation of the measures in industry, Boegh [9] indicated that the measures listed in ISO/IEC 9126 could not be applied directly because they entailed a degree of imprecision. In addition, the author mentioned that engineers in the industry should devote considerable efforts to redefine the measures and implement formalized approaches. B. ISO/IEC SQuaRE In 1991, ISO/IEC 9126 was published as the first consensus on the terminology for quality characteristics of software evaluation. Between 2001 and 2004, ISO published an expanded version containing the quality model and the measures proposed for this model [8]. The current version of the ISO/IEC 9126 standard [1][2][3][4] consists of one normative reference, the Quality models [1], and three technical reports: External metrics [2], internal metrics [3], and Quality in use metrics [4]. Being the part of this new series ISO/IEC has the purpose to define and/or design an initial set of QME In its 2007 edition of ISO/IEC [5] ISO/IEC JTC1 SC7 addressed several of the issues identified in ISO/IEC For example, the standard is now consistent with the theory of metrology as well as the software and system engineering vocabulary. Concerning the reorganisation of the measures, ISO/IEC has identified and classified a set of measures in addition to the current development of other standards concerning derived measures. Moreover, one of its key goals was the harmonization and coordination with ISO/IEC [8]. The standard also provides a set of rules for the design of QMEs and for verification of the design of existing QMEs. Its content provides the link between ISO/IEC 9126, and the subsequent SQuaRE series. ISO/IEC 2502n-Quality Measurement Division is made of six standards: 25020: Measurement reference model and guide 25021: Quality measure elements 25022: Measurement of quality in use (revision of ) 25023:Measurement of system and software product quality (revision of & 3) 25024: Measurement of data quality 25025: Measurement of IS service quality III. RESEARCH APPROACH 1. Selection of a Repository of measures The initial research approach was to propose a set of measure elements that would be at the same time scienced (i.e. following metrology principles), useful and accessible to the software industry and utilized as an international reference. The catalog of measures was to be built with all the existing quality measures known in international reference set of ISO/IEC JTC1 documents [2][3][4][5] and [7]. Besides of the SQuaRE series of standards no other internationally recognized software measurement standards referring to ISO/IEC principles were found. The measures published in ISO/IEC 9126 standard [2][3][4] were taken as the basis of this research because they make the foundation for the development of the SQuaRE series and at the same time the only available complete set of measures. Due to the extensive list of measures that were found, (including duplicates and synonyms) a turn to a more simplistic approach was taken. Only measures in ISO/IEC 9126 parts 2, 3 and 4 [2][3][4] were considered for the experiment. ISO/IEC was kept in the repository but only for the final research comparison analysis as it was considered the control group, while [2][3][4] made the experiment group. Although ISO/IEC has already identified a set of measures, thus dramatically reducing the number of QMEs proposed by ISO/IEC 9126, QMEs defined in ISO/IEC were not used for the experiment. This decision was taken in order to differentiate the experiment and the control group of QMEs as well as obtaining unbiased experiment results for the selection of QMEs from ISO/IEC 9126 repository. Only the final results of this research, the core set of QMEs generated by the experiment, was to undergo a comparison analysis with the line QMEs from ISO/IEC Selection of a Repository of measures The measure elements, even if they primarily apply to the software quality measurement, must be d on the standard metrology for systems engineering and software, ISO/IEC Metrology is the science of measurement

3 and its applications [17]. It includes all theoretical and practical aspects of measurement, regardless of the measurement uncertainty and field of application... [17]. The entire list of measures published in ISO/IEC 9126 parts 2, 3 and 4 were identified, then stored in a repository, called QME Reference Base [18]. As Abran mentioned in [17] the technical reports of ISO/IEC 9126 contain over 250 derived measures and 80 measures. After the development of the repository [18], the research team identified 403 measures including derived and measures, while ISO/IEC had identified and defined more than 67 measures. Of these 403 measures, only eight were clearly identified as measures, and five more were ambiguous. The numbers presented above (403) are unrefined and, as there was no discrimination of the terminology, it included a large number of repetitive measures with no classification in term of or derived measures. The process presented in Figure 2 summarizes the approach used in this research for selecting a core set of QMEs from the ISO/IEC 9126 standard. Note that step 6 of the approach was theoretically verified but not practically experimented due to absence of published data. The repository tool itself played a major part in the success of this experiment. It eased the search and comparison of text defining the QMEs. Although inexpensive, easy to use, readily available, and understood by all team members, the choice of this particular data storage, Excel data, had its limitations. It was not strong enough to make a systematic analysis of the data it contained. A relational data management system (RDBMS) would have allowed for more volume and capacity, supported more complex data structures and processing, data evolution and development. Information in ISO/IEC 9126 could have been superposed to the terminology from ISO [19] to assist in the interpretation of the text, the organization and the differentiation of the QMEs and the mapping of the information to the measurement model of ISO/IEC (see Figure 8), thus improving the analytical capabilities. In essence, the choice of Excel over a RDBMS was motivated strictly by time restriction. Nevertheless, a RDBMS would have been incontestably a superior repository tool option. 4. Core QME determination The measures input into the research reference were classified, categorized and sorted according to the vocabulary used in ISO/IEC 9126 standard. Following this step, QME contexts of use were deducted using the metrics information from ISO/IEC 9126 technical reports. The metrics information encompassed the QME names, purpose, application and measurement method, formula, metric scale, measure type, and target audience. The resulting inventory allowed for determining and contrasting measures from derived measures (see example in Figure 3). Fig. 2. QME selection 3. Choice of storage and analysis tool The creation of the repository, called the QME reference was developed with the Microsoft Excel [18]. The and derived measures proposed in ISO/IEC 9126 parts 2, 3, and 4, as well as those identified in ISO/IEC were all listed in the reference or repository. The reference was also populated with more information relevant to inventory. For example, elements of traceability allowing locating the source of the information from the technical reports [2][3][4] were added. Information often referenced was added to the data, thus avoiding searching the paper versions of the standard. To enhance the communication, follow-up among the research team, rationale, questions, comments about classification or organisation were recorded in distinctive fields of the. Fig. 3. Example of determination during preliminary analysis of a QME As the next step, harmonization of terms was performed: quotation marks, hyphens and spaces were removed, cases were replaced with upper or lower case when appropriate, but words were not changed. Excluding duplicates and using identical words, the reference contained: a set of 121 measures a set of 170 derived measures After these adjustments, the reference underwent several reviews by the research team, each time, thoroughly

4 examining its information. Below the detailed steps followed during the reference development. Step 1: Context determination After each metrics was dissected to extract the QME, it had to be determined if this QME had contexts of use. For example, the QME Number of failure has been identified as a measure. Its contexts were numerous; it was referred several times as a data element of metrics found in ISO/IEC Below, some exemplary contexts for that QME: Number of failures detected during defined trial period Number of failures emerged after failure is resolved Number of failures related to transmission Step 2: Grouping of identical and/or similar QMEs Once all contexts identified and reviewed, identical QME were grouped. This grouping, however, called for some minor QME names adjustment in cases where contexts were similar. For example, Number of message for I/O related errors QME and Number of messages for Mean I/O fulfillment ratio QME were all harmonized with a plural term. A number of words in the reference were found to use the singular and others the plural form. After these initial groupings, there were still several small groups of QME remaining. Groupings extension was created according to the terminology used in ISO/IEC 9126 standards in order to involve more contexts in the existing QMEs. ISO/IEC [19] was used as discerning judgment tool as it is the terminological dictionary known and broadly accepted in software engineering domain. Step 3: QMEs Categorization The grouping easily identified a trend: some QMEs were original and other QMEs, without being secondary, could be extricated from the original QMEs. Secondary QMEs could be related to primary QMEs with reference contexts. A QMEs categorization (see Table 1) was developed by using the same reference, a field allowing differentiation of the QME called roots from the other QME. The objective was to clearly identify the primary QME on which ISO/IEC measurement model could be applied. QME Root Contextualized Singleton Obsolete TABLE I QME CATEGORIZATION Description Original Caused by other QME. They are not derived, but QME whose existence is due to the presence of other QME (e.g. number of errors is a QME and number of error messages is also a QME, but an error is needed to create an error message). Listed only once in the reference. No longer relevant due to technology evolution. Derived Derived QME Note: At this step of the research, derived QME were not supposed to be found since they had already been broken down into QME. Step 4: Validation the core with ISO/IEC The previous steps greatly reduced the amount of validation work using the templates of ISO/IEC by focusing on a small core group of QMEs. Validation work covered options from selecting Root QME with the largest number of contexts to creating the Singleton QME (measures less often referred to or sparsely represented). The research team created a visual model prior to the measurement model validation (with ISO/IEC 15939), to better evaluate the work achieved. The visual template that maps the Root QMEs and their various contexts has been called Contextualized QME model and is presented in Figure 4. As for Figure 5, it presents a concrete example of the QME number of failures and its different contexts of use as described in ISO , 3 and 4. Base QME Name Context 1 Context 2 Context 3 Context 4 Context 5 Fig. 4. Base and Contextualized QME model view Sub-Context 1 Sub-Context 1 Sub-Context 2 Fig. 5. QME Number of failures and its different contexts of use. In the following step, for the validation of these Root QMEs, measurement models provided by ISO/IEC were used to prove that the selected measures met indeed the requirements of this standard. In order to formally define and assess the Root QMEs the adjusted template d on standards was created. The ISO/IEC template had to be adjusted to the context of this research and since many sections of the original template were linked to derived measures rather than to QME, they were removed, e.g. measurement function. The practical validation of developed QMEs was the experimental creation of several derived measures. Figure 6, presents the outcome for the derived measure Access Auditability. Validation of the Root QMEs by testing the

5 measures on existing projects in the industry were not completed due to confidentiality agreements with partners and in consequence due to time constraints.... Analysis Model (Compute mean and standard deviation of all project access auditability values) Derived measure 1 (Access auditability) Measurement Function (Divide A by B) NOTE: Computed confidence limits d on the standard deviation indicate the likelihood that an actual result close to the average Access auditability will be achieved. Very wide confidence limits suggest a potentially large departure and the need for contingency planning to deal with this outcome. ISO/IEC WG6 (Work Group 6). The resulting core set of QMEs is also different from this proposed in ISO/IEC Some of the measures identified by this research were not presented in ISO/IEC and vice versa, however, the goal of reducing the core set of QMEs from 67 to much smaller number of 21 valid QMEs has been achieved. The original measures reference data, which data was extracted from ISO/IEC , contained 410 lines, one line per measure. By limiting the validation work to Root QME category and QMEs with numerous contexts 14 QME were identified (coverage of 245 lines or 60% of the reference, see Figure 7). QME 2 (B= Number of user accesses to the system and data done during evaluation) QME 1 (A= Number of user accesses to the system and data recorded in the access history data) Measurement Method (count) Measurement Method (count) Context 3 (COntrollability) Context 3 (Helping) Context 2 (USer) Context 1 (Logging types) Fig. 7. Core QME chart Number of access Fig. 6. QME Number of access and with development of Access auditability as a derived measure. Step 5: Comparison of the developed core set of QMEs with ISO/IEC The research was concluded by a comparative analysis of the Root QMEs identified by this research with those listed in ISO/IEC The existence of each QME found by the research was verified in ISO When possible, information such as definition, measurement method, and so on was compared. See Table 2 for details. TABLE 2 SOME COMPARISON ELEMENTS OF THE CORE SET OF QMES Findings This research ISO Core size 21 measures 67 measures (22 initial measures plus extensions) Number of failures Number of failures Number of system Failures Number of faults Number of faults (code) Number of faults (design) Number of faults (requirements) Base Number of errors Number of messages measure Number of error messages Error context comparison Number of messages Number of internal functions N/A Number of external functions feature N/A N/A Duration IV. RESULTS The presented research applied a different method of determination of the core set of QMEs than the one used by It was estimated that around 70 lines still need to be validated, what explains the 40% from Figure 7. The 165 remaining lines were rarely referred to by derived measures; they often applied to a single derived measure, for example: I/O Utilization, Number of instances of data corruption prevention, Size of software. Another interesting finding was that 111 derived measures could be created from the 14 identified QMEs. Considering the total of 178 measures found in ISO/IEC , this gave 62% of coverage (111/178) of all the measures found in this standard. Unfortunately, the coverage ratio could not be verified against SQuaRE list of derived measures since standards are still under development. V. RECOMMENDATIONS A few enhancements could be provided by the 2502n division standards to make its use more widespread. Distribution of 2502n division standards in a digital format, like a knowledge data, would attract a larger population of software quality engineers, and increase its use by the industry. Consistent use of ISO/IEC vocabulary. Systematic use of ISO measurement templates, providing the mapping of each QME to strengthen the scientific and the design of the standard. The standards provided by ISO are prepared on paper or as PDF files. This is considered as a limitation for users, since they must go through several documents in order to find the necessary measure elements for their projects. The recommendation of providing a digital knowledge (see Figure 8) to the industry is one of the most important findings of this research.

6 ISO ACKNOWLEDGMENT Internet QME Knowledge ISO ISO ISO ISO ISO ,3,4 Knowledge ISO/IEC Knowledge ISO/IEC This article has been developed in the context of a research project realized at École de Technologie Supérieure, Montreal, Canada. The authors would like to thank Guillaume Bellosguardo, Salah-Eddine Benbrahim and Charles Hébert for their contribution to this research. Finally, the authors are grateful to fellow students of class MGL842 (2011) for their valuable input. REFERENCES Fig. 8. QME Knowledge. Web Interface MeasurementGraphical Tool Server Creation Tool Server Server With this recommendation, some practical issues could be addressed: Gather all software measurements information on a single storage platform. Provide an embedded process to create new measures, or derived, associated to a specific business domain or project, software or component. Develop graphical engines to build appropriate measurement models. Provide secure and remote access to all the information, thus, letting ISO be the guardian of the data, while allowing organisations to globally share or protect their own measurement information to other ISO users. Other directions for the refinement of this research are: The application of the same analysis method to the measures listed in ISO/IEC in order to determine other contexts of measurement since the measures provided in ISO are also d on ISO 9126-technical reports. Adding new QMEs that reflect current development trends such as Agile programming. Although this issue came up as a side effect of this research, it seems logical that the standard evolves in this direction. Removing QMEs that are no longer used due to technology evolution. Adding the definition of QME context as a parameter complementing measurement method and measurement unit. The 2502n division will surely provide the derived measures, but adding the method linking derived measures and QMEs would supplement the information already in the standards. CONCLUSION Other servers To improve the ISO/IEC 2502n division standards, the use of a digital content, as proposed in this paper, would be beneficial to the IT industry. A data support and userfriendly applications that allows quality engineers to quickly navigate and research would be a major improvement to these standards. The tools, such as an automatic calculation of derived measures could also be considered. Furthermore, software quality measurement activities could easily be integrated to existing development processes. Hence, making it a great tool for improving software quality overall. [1] ISO/IEC : Software Engineering - Product Quality - Part 1: Quality Model, Geneva, Switzerland: International Organization for Standardization, [2] ISO/IEC TR : Software Engineering - Product Quality - Part 2: External Metrics, Geneva, Switzerland: International Organization for Standardization, [3] ISO/IEC TR : Software Engineering - Product Quality - Part 3: Internal Metrics, Geneva, Switzerland: International Organization for Standardization. [4] ISO/IEC TR : Software Engineering - Product Quality - Part 4: Quality in Use Metrics, Geneva, Switzerland: International Organization for Standardization. [5] SC7, ISO/IEC DTR : Software Engineering - Software Product Quality Requirements and Evaluation (SQuaRE) - Quality Measure Elements. [6] ISO/IEC IS : Software Engineering - Software Measurement Process, Geneva, Switzerland: International Organization for Standardization. [7] IEEE Std , Standard for a Software Quality Metrics Methodology, revision, IEEE Standards Dept., Piscataway, N.J., 1998 [8] Abran, R. E. Al-Qutaish, J. M. Desharnais, and N. Habra, "An Information Model for Software Quality Measurement with ISO Standards," in Proceedings of the International Conference on Software Development (SWDC-REK), Reykjavik, Iceland, pp [9] J. Boegh, Certifying Software Component Attributes, IEEE Software, Vol. 23, No. 3, 2006, pp [10] A. Abran, R. E. Al-Qutaish, and J.-M. Desharnais, Harmonization issues in the updating of the ISO standards on software product quality, Metrics News: Journal of the Software Metrics Community, Vol. 10, No. 2, 2005, pp [11] A. Abran, R. E. Al-Qutaish, and J. Cuadrado-Gallego, Analysis of the ISO 9126 on Software Product Quality Evaluation from the Metrology and ISO Perspectives WSEAS Transactions on Computers, Vol. 5, No. 11, 2006, pp [12] A. Abran, R. E. Al-Qutaish, and J. Cuadrado-Gallego, Investigation of the Metrology Concepts in ISO 9126 on Software Product Quality Evaluation, Proc. Of the 10th International Conference on Computers (ICComp 06), Vouliagmeni, Athens, Greece, 2006, pp [13] H.-W. Jung, S.-G. Kim, and C.-S. Chung, Measuring Software Product Quality: A Survey of ISO/IEC 9126, IEEE Software, Vol. 21, No. 5, 2004, pp [14] B. B. Chua and L. E. Dyson, Applying the ISO 9126 Model to the Evaluation of an e-learning System, Proc. Of the 21st Conference of the Australasian Society for Computers in Learning in Tertiary Education (ASCILITE 04), Perth, Australia, 2004, pp [15]. Abran, A. Khelifi, W. Suryn, and A. Seffah, Consolidating the ISO Usability Models, Proc. Of 11th International Software Quality Management Conference, Glasgow, Scotland, UK, [16] ISO/IEC, International Vocabulary of Basic and General Terms in Metrology (VIM), Geneva, Switzerland: International Organization for Standardization, [17] A. Abran, Software Metrics and Software Metrology, IEEE Computer Society, John Wiley & Sons Inc., New Jersey, [18] QME Reference Base V2.0, 2011 (not published). [19] ISO/IEC FDIS : Systems and software engineering Vocabulary, Geneva, Switzerland: International Organization for Standardization.