ISO/IEC JTC1/SC7 /N2975

Transcription

1 ISO/IEC JTC1/SC7 Software and Systems Engineering Secretariat: CANADA (SCC) ISO/IEC JTC1/SC7 /N Document Type Title Source Comment Disposition Report Comment Disposition Report - CD : Software and Systems Engineering Software quality requirements and evaluation (SQuaRE) - Quality measurement Measurement reference model and guide WG06 Project Status Reference Action ID Final SC 7 N2767, N2830 FYI or ACT Due Date Distribution AG No. of Pages 8 Note Address reply to: ISO/IEC JTC1/SC7 Secretariat École de technologie supérieure Département de génie électrique 1100 Notre Dame Ouest, Montréal, Québec Canada H3C 1K3 secretariat@jtc1-sc7.org

2 ID (TH,TL,E,G) Clause, Sub-clause Table Comment and rationale Proposed new text Disposition BRA 1 E all Check the spelling of "subcharacteristics" BRA 2 E all Check the use of capital letters when referring to parts of ISO/IEC Standards, as well as to "External", "Internal", "Measures", etc. BRA 3 TL Introduction last paragraph "It is strongly recommended that the user refers to ISO/IEC 2501n, It is strongly recommended that the user refers to ISO/IEC ISO/IEC 2503n and ISO/IEC- 2504n series of standards..." 2500n, ISO/IEC 2501n, ISO/IEC 2503n and ISO/IEC 2504n This sentence should also refer to ISO/IEC 2500n, since ISO/IEC series of standards contains all common definitions and ISO/IEC address a "measurement plan" BRA 4 TL Introduction figure 1 What's the meaning of the lines above each series number? BRA 5 TL Introduction figure 1A the figure does not represent documentation of evaluation modules insert in the figure BRA 6 E Scope "a" bullet "ISO/IEC (metrics for external quality), ISO/IEC " ISO/IEC (metrics for external quality) and ISO/IEC BRA 7 TL Scope bullets from "a" to "h" The text referring to intended users of SQuaRE series should be standardized, so that the same information have the same format. This comment is extensible to all other SQuaRE standards. BRA 8 TL 4.1 to 4.5 these definitions are also in C.2.7. Please check if it is necessary to keep different definitions for the same terms. per ISO Directives Part 2, Annexes C & D ed in part. The standard for documentation of evaluation modules has been moved to the evaluation division, so the figure is now. SQuaRE is intented to be used as a complete set, and since ISO/IEC is the general overview, it will be used as the standard. - will rationalise the definitions - as per Directives Part 2, all definitions will be placed in Clause 4. BRA 9 TH 4.7 measurement primitives the definition is different from ISO/IEC Use definition from ISO/IEC per discussion. Please replace "generally" in with may. BRA 10 E 6.1 1st paragraph "... And ISO/IEC Quality in use Metrics)..."... And ISO/IEC Metrics for quality in use... BRA 11 E 6.1 2nd paragraph "These give methods for measurement, assessment and evaluation of software product quality" an ordered sequence of processes could be more didactic. These give methods for measurement, evaluation and assessment of software product quality in principle. Deleted "assessment" since it has the same meaning as "evaluation" BRA 12 E 6.1 5th paragraph, last sentence "the user should first select the measurement primitives that are needed to collect all required data for evaluation". This sentence is valid not only for quality in use measures. BRA 13 TL 6.1 6th paragraph "Measurement primitives are used throughout the software product lifecycle to construct Internal, External and Quality in Use Measures. In essence, they represent the operational definition of the various quality measures which in turn represent the quality characteristics of the quality model." What is "operation definition of the various quality measures"? They are not necessarily quality measures BRA 14 E 6.1 8th paragraph, 2nd sentence "The user of these standards must then select appropriate measures for these quality characteristics." "must" means "shall" in this case? BRA 15 TL 6.1 last paragraph "Having valid and reliable measures is important for developing a quality model that relates internal quality characteristics with external quality characteristics." the concept of this "quality model" seems different from "quality model" referred in ISO/IEC Probably this sentence wants to mean a "predictive model" BRA 16 TL Figure 3 "direct influence" and "indirect influence" are not either defined or explained Put this sentence in a new paragraph. Delete the second sentence, or rephrase it. Having valid and reliable measures is important for developing a predictive model that relates internal quality characteristics with external quality characteristics. Explain "direct" and "indirect influence".. This is part of the process, and will be extracted as part of the response to. Will rephrase.. Will be reviewed in response to... The figure will be revised to make the distinction clearer. 07N2975 ISO stCD-Dispositions R.xls Page 1 of 8

3 BRA 17 TH all paragraph "Measurement primitives represent quality attributes of the software product (either executable or non-executable). Base measures (resulting from applying measurement primitives) are used as components for the composition of internal, external or Quality in use measures. Every internal, external or quality in use measure formula has to be composed from one or more Base Measures. Base Measures are also used as a common denominator (eg. size in KLOC) while comparing two different software products. The primary purpose of Measurement primitives is to serve as a defined base for software quality evaluation, ensuring that direct raw measurement data are correctly collected and that quality measurement results based on these data are reliable. Examples of Measurement primitives are: number of functions, number of failures, product size in KLOC etc." This paragraph seems to confuse the concepts of "Base Measure" and "Measurement primitives". According to 25021, Measurement primitives can be composed of base or derived measures and they refer not only to quality attributes. Internal measures, external measures and quality in use measures are obtained applying measurement primitives. Measurement primitives represent attributes of the software product (either executable or non-executable). Measurement primitives are used as components for the composition of internal, external or Quality in use measures. Every internal, external or quality in use measure formula has to be composed from one or more measurement primitives. Measurement primitives are also used as a common denominator (eg. size in KLOC) while comparing two different software products. The primary purpose of Measurement primitives is to serve as a defined base for software quality evaluation, ensuring that direct raw measurement data are correctly collected and that quality measurement results based on these data are reliable. Examples of Measurement primitives are: number of functions, number of failures, product size in KLOC etc. Delete EXAMPLE, since the sentence refers to the same examples.. However, Directives Part 2 require that the example be presented separately, so the last sentence will be removed and the example will remain. BRA 18 TL st paragraph "Evaluating quality in use validates software product quality in specific user-task scenarios." This sentence is not precise, since that evaluating quality in use does not validate the software quality, but the quality of the user task. BRA 19 TL nd paragraph "Once measures of the specified quality characteristics have been defined, they must be implemented in the development or operational context. Some measures may directly capture a quality characteristic. In other instances, measures made for other purposes such as progress measures for project management may be combined with quality related measurement primitives to produce a measure of a quality characteristic. Yet another possibility is to combine multiple quality-related primitive measures to capture a quality characteristic as in the case of constructing a quality index." The to quality characteristics seem misplaced, since it is not feasible to measure directly a quality characteristic. May be the t i t d t f t lit tt ib t Evaluating quality in use validates the extent that the software product contributes to the user task quality in specific user-task scenarios. Once measures of the specified quality characteristics have been defined, they must be implemented in the development or operational context. Some measures may directly capture a quality attribute. In other instances, measures made for other purposes such as progress measures for project management may be combined with quality related measurement primitives to produce a measure of a quality attribute. Yet another possibility is to combine multiple quality-related primitive measures to capture a quality attribute as in the case of constructing a quality index. Quality subcharacteristics and characteristics are obtained by applying a composition model i th ti lit tt ib t d b i t l in principle. Will work on new phrasing. 07N2975 ISO stCD-Dispositions R.xls Page 2 of 8

4 BRA 20 TH figure 5 There are some questions related to figure 5: What is the meaning of the large arrow connecting Int/Ext/QIU Quality measure to Quality subcharacteristic? It is not possible to measure a subcharacteristic directly. It is obtained by summarizing the quality attributes. Referring to the "evaluation process", according to , "assess results is the last step, when quality attributes are summarized to obtain quality subcharacteristics, quality characteristics, and finally a statement of the extent to which the software product meets quality requirements. Another problem is that "indicator" has a different meaning in SQuaRE model (at least in series). Indicators are used to di t t l lit i i t l Delete the arrow connecting Int/ext/QIU to quality subcharacteristic; Replace "quality indicator" by "quality summarization". BRA 21 E A.3.1 last paragraph "that is obtained the during testing stages..." that is obtained during testing stages.. accept BRA 22 E A.3.2 first paragraph correct "an quality attribute" a quality attribute. accept BRA 23 E A.3.2 bullets The title refers to "estimation" and the bullets to "prediction". Check the proper use, remembering that prediction is used as a way to anticipate possible values of external quality from internal attributes. accept BRA 24 E B.1 check the reference to ISO/IEC Should it use ISO/IEC 25041? BRA 25 E B nd paragraph check the reference to ISO/IEC Should it use ISO/IEC 2504n?. The diagram will be modified to more correctly reflect the relationships. Reject. The reference will be placed in the bibliography, since it is not normative. Only can be referenced at this stage, since does not exist. Reject. The reference will be placed in the bibliography, since it is not normative. Only can be referenced at this stage, since 2504n does not exist. BRA 26 E C.2.7 1st paragraph "... commonly defined:5... commonly defined: accept CA 1 E Introduction, second line Instead of ISO/IEC (Fig.1) betted would be ISO/IEC accept Quality model and guide CA 2 E 1. Scope second line line should end by : not by full stop reject. Directives Part 2, clause require that a period be used to terminate a full sentence that introduces a list. CA 3 E 3. Normative CA 4 E 3. Normative CA 5 E title According to WG6 decision the part "quality mesurement" of the title should be removed CA 6 E 3. Normative CA 7 E 4. Terms and definitions CA 8 E 6.1 relationship between types and measures CA 9 E 6.1 relationship between types and measures CA 10 E 6.1 relationship between types and measures First sentence would be easier to read if reworded accept. Will be reworded to conform to the Directives. are numbered reference should not be numbered ed. This clause will be modified to conform to the Directives.. The Directives require that Standard names have no more than three elements. all terms and definitions second para titles of series of should not contain a middle part such as "quality mesurement" or "quality evaluation" the text should be revised from a point of view of diferentiaiting between the core body of a definition, notes (explanations) and examples. In the existing form often all make one part of text should be revised from readibility and consistency point of view (measures mixed with metrics etc) fifth para change "realistic" to "real". sixth para plus bullets below (talking about text should be rephrased to allow better understanding/ Bullets should be claasic black points. Refer CAN-E05. This clause will be modified to conform to the Directives.. This clause will be revised in in part. However, Directives state to use hyphens for lists where numbering is not required. 07N2975 ISO stCD-Dispositions R.xls Page 3 of 8

5 CA 11 E all text needs simplelfication and rewriting. In actual form the text creates too much of confusion CA 12 E all as above. Too much of a mix up between measurement primitives, base measures etc CA 13 E all as above. It would be recommended to use a model or a diagram to illustrate the concept CA 14 TH fig.4 the model is unclear both in form and intent and should be revised. The "QC" in "QC Indicator" is a very common term, and would often be mis-interpreted as "Quality Control" CA 15 E all what is the purpose of this clause? It does not present any applicable, necessary or subject-critical conclusion. Remove. CA 16 TL 7.2 second and third para paragraph is difficult to read and comprehend and uses many terms that are NOWHERE defined. Rewrite and add proper definitions to clause 4.. This clause will be revised in. This clause will be revised in. This clause will be revised in. "Quality Indicator" will be used to match Figure 5.. This clause is informative, and will be moved to an informative. This clause will be revised in CA 17 E last sentence should be revised from readibility point of view. This clause will be revised in CA 18 TL all the contents of this clause asks for diagram, then the text as the conclusieve explanation. Recommended revision to add a diagram. A flowchart will be added to clarify the clause. CA 19 E A.2.1 names of properties some are in Bold font some not. Should be unified. Will be reformatted to of measures conform to the Directives. CA 20 E A.3.2 first line "an quality" should be replaced by "a quality" "which are" by. "which is" CA 21 E A.4 title on space too much in the title of the clause. CA 22 E C all both figure and all to it should be C1 instead of A1.. CA 23 E Foreword second para. " rules given in the ISO/IEC Directives, Part 3." It is my understanding that Part 3 no longer exists, but was merged into Part 2, with the release of the fourth edition of the Directives, in change to " ISO/IEC Directives, Part 2.". CA 24 E 3. Normative CA 25 TL 3. Normative 15. "ISO/IEC 15939:2001 Process Framework". This document was released in 2002, and the title listed in the JTC1/SC7 web site does not include "Framework" in the title. 16, 17 & 18 "The following normative documents through reference in this document.. " includes ISO , ISO & ISO/IEC , yet none of these appear anywhere else in the document. Further, ISO/IEC and ISO/IEC series are both identified in Annex B, yet are not listed in Clause 3. CA 26 E 4.7 second line " non-executable).internal, External ". Should be a space between the 2 sentences. CA 27 E third sentence the wording is unclear in: " then defined and specified to plan to finally achieve " CA 28 TL A.1.1 (a) each point item does the term "performance" mean just timing, load handling, etc. or does it also include "functionality"? In (b), "functionality" is identified separately, which implies that it is not included in "performance". When discussing the differences between the test and user environments, it is not clear that this encompasses restrictions/limitations in the functionality of the test environment. For example, in large, complex systems, the test environment is often a simulation of the end-use environment, and may not simulate the full functionality of the real environment. I have not seen the definition for "performance", within the context of the series, so maybe this was understood to cover "functionality", but it is often t id d t b t f it If t th "f ti lit " h ld b change to "ISO/IEC 15939:2002 Process" delete ISO , ISO & ISO/IEC , and add ISO/IEC series change to " non-executable). Internal, External.".. Will rreview the list of normative.. uncertain as to what was meant here; possibly: " specified in a. This clause will be revised in plan, in order to achieve the required " include, or reference, a definition of "performance". If it doesn't include "functionality", then add some example(s) in (a). This is different than the "coverage of functionality in test environment" mentioned in testing execution, in (b) Rejected. The context of use of the word "performance" in the examples clearly indicates its meaning. It may be that the wording of the question has caused some confusion, and this will be corrected. 07N2975 ISO stCD-Dispositions R.xls Page 4 of 8

6 ID (TH,TL,E,G) Clause, Sub-clause Table Comment and rationale Proposed new text Disposition CA 29 E A.1.1 (b) first sentence ends with " user environment.?". Should remove the period ".". CA 30 TL A.1.2 c) point item as part of results data validation, this should also cover the add another point item to c) [or maybe it should be added to (a). This clause will be reworded validation of tools or facilities used for automatic collection of? ] "validation of tools or facilities used for automatic to make the intent clearer. evaluation data (as indicated in (a) ). Certainly the validation of any collection of evaluation data" tools used, will affect the confidence we have in the validity of the results. This is often overlooked, and is worth stating here. CA 31 E B.3.3 Table B.3 (b) Under "Portability", I don't think the sub-characteristic "Instability". External & c) is appropriate :-) It should be "Installability" CA 32 G general the terms "quality in use", "Quality in use" and "Quality In Use" are Chose one variant of a term, and apply it consistently.. "quality in use" will be used all used. The usage should be consistent, throughout the document. to conform to the Directives. FR 1 Introduction Figure 1 Also, variations are found with "Quality In Use Measure", "Quality is use measure", "measures for quality in use" and "quality measures in use". In particular, the last term, could mean "the quality measures we are using to collect Internal, External and Quality In Use measurements", so we need to be consistent in the use of terms, to make it clear to the reader what was meant Text is unreadable. Will simplify the diagram. FR 2 A.2.2 "Discriminative" is not a noun: inconsistency (e) Discriminativity. Will use "discrimination". JPN 1 GH Contents of "25021 Measurement Primitives" is still not matured enough. We should wait for forwarding this document to the next stage until it matures and conformance between them can be will be balloted at least once more at the CD stage. JPN 2 E Introduction 3rd Para. (07N2767 P.4) JPN 3 E Introduction 4th Para. Line 3 (07N2767 P.4) The description order queues up in the order such as 25010, 25021, 25023, 25022, and We had better describe the standard in numerical order. We had better describe, "user" than "reader". We had better correct it as written by the following BOLD character. ISO/IEC defines measurement primitives, ISO/IEC defines Internal software product quality measures, ISO/IEC defines External software product quality measures and ISO/IEC defines Quality in use measures We had better correct it as written by the following BOLD character. It is strongly recommended that the user refers to ISO/IEC 2501n, ISO/IEC 2503n, and ISO/IEC-2504n series of standards prior to using this International standard and the associated metrics standards, particularly if the user is not familiar with the use of software measures for product specification and JPN 4 E Introduction Fig.1A In Fig.1A, there is no description concerning ISO/IEC 25025: Documentation of evaluation modules should also be explained as one of division document. JPN 5 E 3 JPN 6 E 3 (07N2767 P.7) (07N2767 P.7) JPN 7 E 6.1 5th Para. Line 3 (07N2767 P.11) In NOTE, some have the description about a corresponding old standard (ex: #5,#6 and #7), and some don't have the description about a corresponding old standard. The expression does not unite it. The expression of the standard name (in Normative ) does not unite it. We had better remove the descriptions about a corresponding old standard We should describe it based on Directives(part.3). ed in part. The standard for documentation of evaluation modules has been moved to the evaluation division, so the figure is now accept We had better change the paragraph (change the line) ahead of. the following sentence The user should first select the measurement primitives that are needed to collect all required data for the evaluation. 07N2975 ISO stCD-Dispositions R.xls Page 5 of 8

7 JPN 8 E th Para. Line 1 (07N2767 P.12) JPN 9 E st Para. Line 8 (07N2767 P.12) JPN 10 E st Para. Line 6 JPN 11 E Fig.5 (07N2767 P.12) (07N2767 P.15) JPN 12 TL 7.1 1st Para. Line 1 (07N2767 P.16) External property, Internal quality attribute and external quality attribute, which is written by the following BOLD character, are undefined. external properties (such as suitability, accuracy, fault tolerance or time behaviour) will influence the observed quality. A failure in quality in use (such as the user cannot complete the task) can be traced to external quality attributes (such as suitability or operability) and the associated internal quality attributes which The last one sentence and EXAMPLE mean the same thing. We had better use the word defined by Terms & Definition. We had better remove the EXAMPLE. in principle. Do not need separate definitions of quality attributes, but rather product and process attribute. May need to add definitions or rewrite in terms of subcharacterisitcs. So, quality measures are only those used to represent quality char and quality subchars but not attributes We had better describe, "evaluate intermediate software product" than "evaluate software product". There are two "Assesment" in the SQuaRE model at the left of Fig.5. The meaning of the following sentence is not understood easily. 7.1 Measure to meet information needs Measures used for the construction of quality measures shall be valid and reliable. When we examine the use of measures, it have not been understood yet whether there were effectiveness and reliability of measures. We might use it by way of experiment, to confirm effectiveness and the reliability of measures. Therefore, I think that it is necessary to inform effectiveness and li bilit JPN 13 TL 7.1 In the following sentence, it is not understood easily what item that Quality Measures should conform Additionally, the specific data used should come from measurement processes that conform to ISO/IEC and the measures shall be made to meet information needs in accordance with ISO/IEC Annex A provides a detailed informative guide on factors to consider when using measures JPN 14 TH 7.2 2nd Para. Meanings of the terms "construct validity","external validity","internal validity", and "face validity" are not clear. JPN 15 TL 8.1 1st Para. The relation to Quality Requirements in 8.1 is not understood easily. We had better correct it as written by the following BOLD character. measures provide users, evaluators, testers, and developers with the benefit that they are able to evaluate intermediate software product quality and address quality issues early before the software product becomes executable We had better change the expression into "Analysis Model" from "Assesment",which is in the SQuaRE model, corresponding to "(Analysis) Model" of models. We had better correct it as written by the following BOLD character. 7.1 Measure to meet information needs Measures include Measurement Primitives used for the construction of quality measures shall inform their validity and reliability. We had better described the example of the item of ISO/IEC15939 which should be conformed with Quality Measures in Annex A. (For example, we had better show what Quality Measure should fill according to Base Measure shown by ISO/IEC15939.) They should be explained. Because we should more explain the relation to Quality Requirements, we had better insert the sentence "(See in Quality Requirements)" in the 2nd line or the 3rd line... The higher bubble will be changed to "Evaluate", and the lower bubble to "Predict". Reject. Cannot use shall in this sentence as proposed. Could be addressed by better explanation of types of validity and reliability. in principle. Use to identify info needs. Use measures in this series as recommendations for what to measure. Use as the normative ref for a measurement process. in principle. SPAIN 1 E 4.1 Current definition includes the defined tern Identification of a measurement when a... in principle. Will rewrite or use other defns. SPAIN 2 TL 4.4 Current definition excludes inverse scales In M' = am + b ( a > 0 ). in principle. Will rewrite or use other defns. SPAIN 3 TL 4.5 Same as ID-2 In M' = am ( a > 0 ). in principle. Will rewrite or use other defns.. 07N2975 ISO stCD-Dispositions R.xls Page 6 of 8

8 ID (TH,TL,E,G) Clause, Sub-clause Table Comment and rationale Proposed new text Disposition SPAIN 4 TL first Include different user roles (eg: Admin, Client,...) the various subtypes of end user for whom.. in principle. Will rewrite or use other defns. USA 1 TL Introduction Figure 1 Figure 1 is incomplete Complete the figure USA 2 TL 3 1st sentence The 1st sentence does not agree with the prescribed text in the ISO/IEC Directives, Part 2 (4th Edition, 2001), subclause Change the text to agree with the Directives.. The following referenced documents are indispensable for the application of this document. For dated, only the edition cited applies. For undated, the latest edition of the referenced document (including any amendments) applies USA 3 TH 3 The lengthy list of normative mostly contains documents List only those documents that are publicly available and that are not yet published and thus are not publicly available. This is include a Note about the others that are in development. violation of the ISO/IEC Directives, Part 2, subclause USA 4 TL 3 All of the standards listed are not referenced in the document. List only those standards that are referenced in the document.. USA 5 TH 6, 7, 8 It is confusing in Clauses 6, 7, and 8 (especially Clause 6) to identify Rewrite the text of these Clauses to make a clear distinction the requirements that one is obliged to satisfy. Most of the text is in between requirements, statements, and recommendations. Use the indicative mood rather than the imperative mood. See ISO/IEC correct terminology per the Directives. Directives, Part 2, subclause 6.6.1, subclause 6.3.3, and Annex G, including tables G.1 through G.4. For example, the word "may" is used throughout this document to indicate possibility rather than permission. The word "can" should be used instead. Also, the terms "must" and "has to" are used in a few instances where it is unclear whether "shall" or "should" was meant. So, the present USA 6 TL st paragraph, 1st verbal forms for the expression of provisions is in violation of the The phrase "well defined quality model" is not well defined. What Provide clarification. sentence are the criteria for a "well defined quality model?" USA 7 TL nd paragraph, 1st sentence USA 8 G All The USA remains concerned about the total number of documents Condense the SQuaRE series to fewer documents. planned for the SQuaRE series could be a hinderence to its usability by intended users as well as its marketability by ISO. USA 9 TH All The SQuaRE series of measurement documents promotes measurement as a bottom-up discovery process rather than one guided by indentified information needs. This leads to conceptual confusion between quality attributes and the measures that ZAR 1 E 4 I find the way these definitions are expressed quite confusing and difficult to understand. A great deal more effort needs to be devoted to producing readable and understandble definitions. Because the definitions are unclear, all to these in the body of the document will be be misunderstood. ZAR 2 G Is this an IS or TR? If it is an IS, can we accommodate 'shalls' and 'shoulds' in the same breath? ZAR 3 4 This duplicates the definitions in where they are better expressed. ZAR 4 2 Replace 'state the rationale for their exclusion.' with 'state the validity of the rationale for their exclusion.'. We will review for appropriate use of shalls, shoulds, mays, and cans. in principle. Does "need" mean "should?" See comment #5. in part. Changed to shall. Rewrite the documents to promote a top-down view of measurement to be in agreement with ISO/IEC Reject. For the purposes of this document. Will await further analysis of other documents to see if normative information can be consolidated and simplified. in principle. in principle. Need to coordinate with other authors on standard definitions and will adopt those.. We will review for appropriate use of shalls, shoulds, in principle ed in part, however, there cannot be any justification for non conformance. ZAR 5 F4 Q - C is unclear - Quality Characteristic or Quality Control. Quality Characteristic 07N2975 ISO stCD-Dispositions R.xls Page 7 of 8

9 ZAR Is Valid and Reliable the only two characteristics one would need to refer back to (See 25000)? ZAR Is there a link into ISO 9001 ito product quality measure Reject ZAR 8 F7 What analysis model is this - is it the same as in 4.2 in 25000? Is it the same as analysis model in Figure 5? Rejected. In clause 6.2.1, ISO/IEC lists some desirable characteristics for measures. However, these are a subset of the characteristics of "validity" and "reliability" that we get from measurement theory ed. This should be titled as an example. 07N2975 ISO stCD-Dispositions R.xls Page 8 of 8