ISO/IEC JTC1/SC7 N2830,

Transcription

1 ISO/IEC JTC1/SC7 Software & Systems Engineering Secretariat: CANADA (SCC) ISO/IEC JTC1/SC7 N2830, Document Type Letter Ballot Summary Title Letter Ballot Summary CD 25020: Software and Systems Engineering Software quality requirements and evaluation (SQuaRE) Quality measurement Measurement reference model and guide. Source JTC1/SC7 Secretariat Project ISO Status CD References N2767 Action Information or action Due Date Distribution SC7_AG; P, O & L Members Medium Acrobat No. of Pages 15 Note TABLE OF VOTING AND COMMENTS RECEIVED P members Approve Disapprove Abstain Comments Not voting Australia Belgium Brazil Canada China Czech Republic Denmark Finland France Germany Hungary Ireland Israel Italy Japan Korea Netherlands Portugal Romania Singapore South Africa Spain Sweden Switzerland UK Ukraine USA Thailand TOTAL Address reply to: ISO/IEC JTC1/SC7 Secretariat École de technologique supérieure Alain Abran 1100 rue Notre-Dame Ouest Montréal (Québec) Canada H3C 1K3 sc7@ele.etsmtl.ca Phone.: +1 (514) Fax: +1 (514)

2 LETTER BALLOT SUMMARY O and L members commenting: Country / Liaison YES Comments Russia Country Comments 1

3 Balloted document: ISO/IEC JTC 1/SC 7 N2767 CD Vote: Approve BRA 1 E all Check the spelling of "subcharacteristics" BRA 2 E all Check the use of capital letters when referring to parts of ISO/IEC Standards, as well as to "External", "Internal", "Measures", etc. BRA 3 TL Introductio n last paragraph BRA 4 TL Introductio n figure 1 BRA 5 TL Introductio figure 1A n BRA 6 E Scope "a" bullet "ISO/IEC (metrics for external quality), BRA 7 TL Scope bullets from "a" to "h" "It is strongly recommended that the user refers to It is strongly recommended that the user refers to ISO/IEC 2501n, ISO/IEC 2503n and ISO/IEC- 2504n ISO/IEC 2500n, ISO/IEC 2501n, ISO/IEC 2503n series of standards..." and ISO/IEC 2504n series of standards... This sentence should also refer to ISO/IEC 2500n, since ISO/IEC contains all common definitions and ISO/IEC address a "measurement plan" What's the meaning of the lines above each series number? the figure does not represent documentation of insert in the figure evaluation modules ISO/IEC (metrics for external quality) and ISO/IEC " ISO/IEC The text referring to intended users of SQuaRE series should be standardized, so that the same information have the same format. This comment is extensible to all other SQuaRE standards. BRA 8 TL 4.1 to 4.5 these definitions are also in C.2.7. Please check if it is necessary to keep different definitions for the same terms. BRA 9 TH 4.7 measurement the definition is different from ISO/IEC Use definition from ISO/IEC primitives BRA 10 E 6.1 1st paragraph "... And ISO/IEC Quality in use Metrics)..."... And ISO/IEC Metrics for quality in use... BRA 11 E 6.1 2nd paragraph "These give methods for measurement, assessment and evaluation of software product quality" an ordered sequence of processes could be more didactic. These give methods for measurement, evaluation and assessment of software product quality 07N2830 Brazil comments.xls Page 1 of 5

4 BRA 12 E 6.1 5th paragraph, last sentence "the user should first select the measurement primitives that are needed to collect all required data for evaluation". This sentence is valid not only for quality in use measures. Put this sentence in a new paragraph. BRA 13 TL 6.1 6th paragraph "Measurement primitives are used throughout the Delete the second sentence, or rephrase it. software product lifecycle to construct Internal, External and Quality in Use Measures. In essence, they represent the operational definition of the various quality measures which in turn represent the quality characteristics of the quality model." What is "operation definition of the various quality measures"? They are not necessarily quality measures. BRA 14 E 6.1 8th paragraph, 2nd sentence "The user of these standards must then select appropriate measures for these quality characteristics." "must" means "shall" in this case? BRA 15 TL 6.1 last paragraph "Having valid and reliable measures is important for developing a quality model that relates internal quality characteristics with external quality characteristics." the concept of this "quality model" seems different from "quality model" referred in ISO/IEC Probably this sentence wants to mean a "predictive BRA 16 TL Figure 3 "direct influence" and "indirect influence" are not either defined or explained Having valid and reliable measures is important for developing a predictive model that relates internal quality characteristics with external quality characteristics. Explain "direct" and "indirect influence". 07N2830 Brazil comments.xls Page 2 of 5

5 BRA 17 TH all paragraph "Measurement primitives represent quality attributes of the software product (either executable or nonexecutable). Base measures (resulting from applying measurement primitives) are used as components for the composition of internal, external or Quality in use measures. Every internal, external or quality in use measure formula has to be composed from one or more Base Measures. Base Measures are also used as a common denominator (eg. size in KLOC) while comparing two different software products. The primary purpose of Measurement primitives is to serve as a defined base for software quality evaluation, ensuring that direct raw measurement data are correctly collected and that quality measurement results based on these data are reliable. Examples of Measurement primitives are: number of functions, number of failures, product size in KLOC etc." This paragraph seems to confuse the concepts of "Base Measure" and "Measurement primitives". According to 25021, Measurement primitives can be composed of base or derived measures and they refer not only to BRA 18 TL st paragraph "Evaluating quality in use validates software product quality in specific user-task scenarios." This sentence is not precise, since that evaluating quality in use does not validate the software quality, but the quality of the user task. Measurement primitives represent attributes of the software product (either executable or nonexecutable). Measurement primitives are used as components for the composition of internal, external or Quality in use measures. Every internal, external or quality in use measure formula has to be composed from one or more measurement primitives. Measurement primitives are also used as a common denominator (eg. size in KLOC) while comparing two different software products. The primary purpose of Measurement primitives is to serve as a defined base for software quality evaluation, ensuring that direct raw measurement data are correctly collected and that quality measurement results based on these data are reliable. Examples of Measurement primitives are: number of functions, number of failures, product size in KLOC etc. Delete EAMPLE, since the sentence refers to the same examples. Evaluating quality in use validates the extent that the software product contributes to the user task quality in specific user-task scenarios. 07N2830 Brazil comments.xls Page 3 of 5

6 BRA 19 TL nd paragraph "Once measures of the specified quality characteristics have been defined, they must be implemented in the development or operational context. Some measures may directly capture a quality characteristic. In other instances, measures made for other purposes such as progress measures for project management may be combined with quality related measurement primitives to produce a measure of a quality characteristic. Yet another possibility is to combine multiple qualityrelated primitive measures to capture a quality characteristic as in the case of constructing a quality index." The references to quality characteristics seem misplaced, since it is not feasible to measure directly a quality characteristic. May be the sentence intends to refer to quality attributes. BRA 20 TH figure 5 There are some questions related to figure 5: What is the meaning of the large arrow connecting Int/Ext/QIU Quality measure to Quality subcharacteristic? It is not possible to measure a subcharacteristic directly. It is obtained by summarizing the quality attributes. Referring to the "evaluation process", according to , "assess results is the last step, when quality attributes are summarized to obtain quality subcharacteristics, quality characteristics, and finally a statement of the extent to which the software product meets quality requirements. Another problem is that "indicator" has a different meaning in SQuaRE model (at least in series). Indicators are used to predict external quality using internal measures. Once measures of the specified quality characteristics have been defined, they must be implemented in the development or operational context. Some measures may directly capture a quality attribute. In other instances, measures made for other purposes such as progress measures for project management may be combined with quality related measurement primitives to produce a measure of a quality attribute. Yet another possibility is to combine multiple quality-related primitive measures to capture a quality attribute as in the case of constructing a quality index. Quality subcharacteristics and characteristics are obtained by applying a composition model using the respective quality attributes measured by internal, external and quality in use measures. Delete the arrow connecting Int/ext/QIU to quality subcharacteristic; Replace "quality indicator" by "quality summarization". BRA 21 E A.3.1 last paragraph "that is obtained the during testing stages..." that is obtained during testing stages.. BRA 22 E A.3.2 first paragraph correct "an quality attribute" a quality attribute. 07N2830 Brazil comments.xls Page 4 of 5

7 BRA 23 E A.3.2 bullets The title refers to "estimation" and the bullets to "prediction". Check the proper use, remembering that prediction is used as a way to anticipate possible values of external quality from internal attributes. BRA 24 E B.1 check the reference to ISO/IEC Should it use ISO/IEC 25041? BRA 25 E B nd paragraph check the reference to ISO/IEC Should it use ISO/IEC 2504n? BRA 26 E C.2.7 1st paragraph "... commonly defined:5... commonly defined: 07N2830 Brazil comments.xls Page 5 of 5

8 Balloted document: ISO/IEC JTC 1/SC 7 N2767 CD Vote: Approve with comments CA 1 E Introductio n, second line CA 2 E 1. Scope second line line should end by : not by full stop CA 3 E 3. First sentence would be easier to read if reworded Normative references Instead of ISO/IEC (Fig.1) better would be ISO/IEC Quality model and guide CA 4 E 3. references references are numbered reference should not be numbered Normative references CA 5 E title According to WG6 decision the part "quality measurement" of the title should be removed CA 6 E 3. Normative references CA 7 E 4. Terms and definitions references all terms and definitions CA 8 E 6.1 relationship between types and measures second para CA 9 E 6.1 fifth para relationship between types and measures CA 10 E 6.1 sixth para plus relationship bullets below between (talking about types and primitives) measures titles of series of references should not contain a middle part such as "quality measurement" or "quality evaluation" the text should be revised from a point of view of diferentiating between the core body of a definition, notes (explanations) and examples. In the existing form often all make one part of text should be revised from readability and consistency point of view (measures mixed with metrics etc) change "realistic" to "real" text should be rephrased to allow better understanding/ Bullets should be classic black points 07N2830 Canada comments.xls Page 1 of 4

9 CA 11 E all text needs simplification and rewriting. In actual form the text creates too much of confusion CA 12 E all as above. Too much of a mix up between measurement primitives, base measures etc CA 13 E all as above. It would be recommended to use a model or a diagram to illustrate the concept CA 14 TH fig.4 the model is unclear both in form and intent and should be revised. The "QC" in "QC Indicator" is a very common term, and would often be mis-interpreted as "Quality Control" CA 15 E all what is the purpose of this clause? It does not present any applicable, necessary or subject-critical conclusion. Remove. CA 16 TL 7.2 second and third para paragraph is difficult to read and comprehend and uses many terms that are NOWHERE defined. Rewrite and add proper definitions to clause 4. CA 17 E last sentence should be revised from readibility point of view CA 18 TL all the contents of this clause asks for diagram, then the text as the conclusieve explanation. Recommended revision to add a diagram CA 19 E A.2.1 names of some are in Bold font some not. Should be unified properties of measures CA 20 E A.3.2 first line "an quality" should be replaced by "a quality" "which are" by "which is" CA 21 E A.4 title on space too much in the title of the clause CA 22 E C all both figure and all references to it should be C1 instead of A1. CA 23 E Foreword second para. " rules given in the ISO/IEC Directives, Part 3." It is change to " ISO/IEC Directives, Part 2." my understanding that Part 3 no longer exists, but was merged into Part 2, with the release of the fourth edition of the Directives, in CA 24 E 3. Normative references 15. "ISO/IEC 15939:2001 Process Framework". This document was released in 2002, and the title listed in the JTC1/SC7 web site does not include "Framework" in the title. change to "ISO/IEC 15939:2002 Process" 07N2830 Canada comments.xls Page 2 of 4

10 CA 25 TL 3. Normative references 16, 17 & 18 "The following normative documents through reference in this document.. " includes ISO , ISO & ISO/IEC , yet none of these appear anywhere else in the document. Further, ISO/IEC and ISO/IEC series are both identified in Annex B, yet are not listed in Clause 3. CA 26 E 4.7 second line " non-executable).internal, External ". Should be a space between the 2 sentences. CA 27 E third sentence the wording is unclear in: " then defined and specified to plan to finally achieve " CA 28 TL A.1.1 (a) each point item does the term "performance" mean just timing, load handling, etc. or does it also include "functionality"? In (b), "functionality" is identified separately, which implies that it is not included in "performance". When discussing the differences between the test and user environments, it is not clear that this encompasses restrictions/limitations in the functionality of the test environment. For example, in large, complex systems, the test environment is often a simulation of the enduse environment, and may not simulate the full functionality of the real environment. I have not seen the definition for "performance", within the context of the series, so maybe this was understood to cover "functionality", but it is often not considered to be part of it. If not, then "functionality" should be included in the examples. CA 29 E A.1.1 (b) first sentence ends with " user environment.?". Should remove the period "." CA 30 TL A.1.2 c) point item as part of results data validation, this should also cover the validation of tools or facilities used for automatic collection of evaluation data (as indicated in (a) ). Certainly the validation of any tools used, will affect the confidence we have in the validity of the results. This is often overlooked, and is worth stating here. delete ISO , ISO & ISO/IEC , and add ISO/IEC series change to " non-executable). Internal, External." uncertain as to what was meant here; possibly: " specified in a plan, in order to achieve the required " include, or reference, a definition of "performance". If it doesn't include "functionality", then add some example(s) in (a). This is different than the "coverage of functionality in test environment" mentioned in testing execution, in (b) add another point item to c) [or maybe it should be added to (a)? ] "validation of tools or facilities used for automatic collection of evaluation data" 07N2830 Canada comments.xls Page 3 of 4

11 CA 31 E B.3.3 Table B.3 (b) External & c) Internal Under "Portability", I don't think the sub-characteristic "Instability" is appropriate :-) It should be "Installability" CA 32 G general the terms "quality in use", "Quality in use" and "Quality In Use" are all used. The usage should be consistent, throughout the document. Also, variations are found with "Quality In Use Measure", "Quality is use measure", "measures for quality in use" and "quality measures in use". In particular, the last term, could mean "the quality measures we are using to collect Internal, External and Quality In Use measurements", so we need to be consistent in the use of terms, to make it clear to the reader, what was Chose one variant of a term, and apply it consistently. 07N2830 Canada comments.xls Page 4 of 4

12 Balloted document: ISO/IEC JTC 1/SC 7 N2767 CD Vote: Approve, Disapprove, Abstain FR 1 Introductio n Figure 1 Text is unreadable FR 2 A.2.2 "Discriminative" is not a noun: inconsistency (e) Discriminativity 07N2830 France comments.xls Page 1 of 1

13 WG Mem Balloted document: Vote: CD Reject ZAR 1 E 4 I find the way these definitions are expressed quite confusing and difficult to understand. A great deal more effort needs to be devoted to producing readable and understandble definitions. Because the definitions are unclear, all references to these in the body of the document will be be misunderstood. ZAR 2 G Is this an IS or TR? If it is an IS, can we accommodate 'shalls' and 'shoulds' in the same breath? ZAR 3 4 This duplicates the definitions in where they are better expressed. ZAR 4 2 Replace 'state the rationale for their exclusion.' with 'state the validity of the rationale for their exclusion.' ZAR 5 F4 Q - C is unclear - Quality Characteristic or Quality Control ZAR Is Valid and Reliable the only two characteristics one would need to refer back to (See 25000)? ZAR Is there a link into ISO 9001 ito product quality measure requirements ZAR 8 F7 What analysis model is this - is it the same as in 4.2 in 25000? Is it the same as analysis model in Figure 5? 07N2830 South Africa comments.xls Page 1 of 1

14 Balloted document: ISO/IEC JTC 1/SC 7 N2767 CD Vote: DISAPPROVE with comments Figure, Table USA 1 TL Introduction Figure 1 Figure 1 is incomplete Complete the figure USA 2 TL 3 1 st sentence The 1st sentence does not agree with the prescribed Change the text to agree with the Directives. text in the ISO/IEC Directives, Part 2 (4th Edition, 2001), subclause USA 3 TH 3 The lengthy list of normative references mostly List only those documents that are publicly contains documents that are not yet published and thus available and include a Note about the others that are not publicly available. This is violation of the are in development. ISO/IEC Directives, Part 2, subclause USA 4 TL 3 All of the standards listed are not referenced in the document. List only those standards that are referenced in the document. USA 5 TH 6, 7, 8 It is confusing in Clauses 6, 7, and 8 (especially Clause 6) to identify the requirements that one is obliged to satisfy. Most of the text is in the indicative mood rather than the imperative mood. See ISO/IEC Directives, Part 2, subclause 6.6.1, subclause 6.3.3, and Annex G, including tables G.1 through G.4. For example, the word "may" is used throughout this document to indicate possibility rather than permission. The word "can" should be used instead. Also, the terms "must" and "has to" are used in a few instances where it is unclear whether "shall" or "should" was meant. So, the present verbal forms for the expression of provisions is in violation of the Directives. Rewrite the text of these Clauses to make a clear distinction between requirements, statements, and recommendations. Use correct terminology per the Directives. USA 6 TL st paragraph, 1 st The phrase "well defined quality model" is not well Provide clarification. sentence defined. What are the criteria for a "well defined quality model?" USA 7 TL nd paragraph, 1 st sentence Does "need" mean "should?" See comment #5. USA 8 G All The USA remains concerned about the total number of Condense the SQuaRE series to fewer documents. documents planned for the SQuaRE series could be a hinderence to its usability by intended users as well as its marketability by ISO. 07N2830 USA comments.xls Page 1 of 2

15 USA 9 TH All The SQuaRE series of measurement documents promotes measurement as a bottom-up discovery process rather than on guided by indentified information needs. This leads to conceptual confusion between quality attributes and the measures that operationalize those attributes. Rewrite the documents to promote a top-down view of measurement to be in agreement with ISO/IEC N2830 USA comments.xls Page 2 of 2