ISO/IEC JTC1/SC7 /N3652

Transcription

1 ISO/IEC JTC1/SC7 Software and Systems Engineering Secretariat: CANADA (SCC) ISO/IEC JTC1/SC7 /N Document Type Title Source Meeting Minutes Meeting Minutes, JTC1/SC7/WG6: Evaluation and Metrics, Salvador, Brazil, November 23 (Monday) to 27 (Friday), 2006 WG6 Convener Project Status Final Reference Action ID Distribution FYI or ACT AG No. of Pages 18 Note Address reply to: ISO/IEC JTC1/SC7 Secretariat École de technologie supérieure Departement of Software and IT Engineering 1100 Notre Dame Ouest, Montréal, Québec Canada H3C 1K3

2 TITLE: SC7/WG6 The 31 st SC7/WG6 Meeting Salvador, Brazil November 23 (Monday) to 27 (Friday), 2006 SC7/WG6 N585 ISSUED DATE: SOURCE: M. Azuma (WG6 Convener) and T. Komiyama (WG6 Secretariat) WORK ITEMS: ISO/IEC series: Software product Requirements and Evaluation (SQuaRE) REFERENCES: 6/N-569 (7N3559): SC7/WG6 Bangkok meeting minutes 6/N-572 (7N3582): Calling notice and draft agenda for SC7/WG6 Salvador meeting DOCUMENT TYPE: Meeting Report ACTION: FYI Convener Prof. AZUMA, Motoei (JISC, Japan) Department of Industrial and Management Systems Engineering Waseda University 3-4-1, Okubo, Shinjuku-ku, Tokyo , Japan FA: Secretariat Mr. KOMIYAMA, Toshihiro Consulting Business Division NEC Corporation 5-7-1, Shiba, Minato-ku, Tokyo , Japan FA:

3 1. SUMMARY WG6 had a very co-operative and successful meeting from November 23 (Monday) to 27 (Friday), 2006 at Sol Barra Express Hotel in Salvador, Brazil. Twenty-five members from fourteen countries participated the meeting and contributed. WG6 is struggling for standardization of Software product Requirements and Evaluation, called SQuaRE. SQuaRE project consists of totally seventeen ISO/IEC series documents. WG6 currently focuses on ten sub-projects, i.e., ISO/IEC 25001: Planning and management, ISO/IEC 25010: model, ISO/IEC 25012: Data quality model, ISO/IEC 25020: Measurement reference model and guide, ISO/IEC 25021: Measurement primitives, ISO/IEC 25022: Measurement of internal quality, ISO/IEC 25023: Measurement of external quality, ISO/IEC 25024: Measurement of quality in use, ISO/IEC 25030: requirements, ISO/IEC 25040: Evaluation reference model and guide. Also, the draft study group report of recoverability evaluation module is tabled and discussed. According to the resolutions of the Bangkok Meeting, the revision of ISO/IEC was forwarded for FIDS ballot and ISO/IEC for 2 nd CD ballot. WG6 internal review was done for and in advance to forwarding for FIDS ballot, and for 1 st CD ballot. The revision of and the draft study group report on recoverability evaluation module were circulated for WG6 internal review before the meeting. In the Salvador meeting, no actions were taken for ISO/IEC and ISO/IEC because both were in the FIDS ballot stage. As to ISO/IEC 25010, revision points from the ISO/IEC were discussed. The revised document will be circulated for WG6 internal review and the further revision will be forwarded for CD ballot after the next meeting. As to ISO/IEC 25012, 2 nd CD ballot comments were disposed. Data quality model concept and architecture were refined during the meeting. WG6 decided to forward the revision for 3 rd CD ballot. As to ISO/IEC 25021, the final check for publication was done and WG6 agreed to forward it for TR publication. As to ISO/IEC 25022, and 25024, common format for table of measures was discussed and agreed. The draft to forward for WD and CD Registration ballot will be prepare by the editors and circulated for WG6 internal review. As to ISO/IEC 25030, WG6 internal review comments were disposed. WG6 decided to forward the revision to FDIS ballot just after the Salvador meeting. As to ISO/IEC 25040, WG6 internal review comments were disposed. WG6 agreed to forward the revision for 1 st CD ballot after WG6 internal review. As to evaluation module for recoverability, draft NWI proposal was reviewed. WG6 agreed to propose this project as NWI at the next plenary meeting. 2. PARTICIPANTS 25 members, 13 countries:

4 Name Country 1 Vipula Godamunne Australia 2 Danilo Scalet Brazil 3 Regina Colombo Brazil 4 Maria Furtado Brazil 5 Jean-Marc Deshrnais Canada 6 Peter Shum Canada 7 Ke Han China 8 Wang Baoai China 9 Yuan yuyu China 10 Jiri Vanicek Czech Republic 11 Jogen Boegh Denmark 12 Markku Tukiainen Finland 13 Enrico Viola Italy 14 Motoei Azuma Japan 15 Toshihiro Komiyama Japan 16 Yukio Tanitsu Japan 17 Atsushi Yamada Japan 18 Keum-Suk Lee Korea 19 Seok-kyoo Shin Korea 20 Chang-Sun Song Korea 21 Juan Carlos Granja-Alvarez Spain 22 Nigel Bevan UK 23 Michael Kress USA 24 Dean Baker USA 25 David Zubrow USA

5 3. MEETING SCHEDULE AM1 8:30 9:00 AM2 9:00 10:30 AM3 11:00 12:30 PM1 13:30 15:00 PM2 15:30 17:00 PM3 17:00-18:00 Nov. 23 (Mon) WG6 Opening Plenary WG6 Opening Plenary -2n Division n Division New EM Nov. 24 (Tue) WG6 Plenary & Nov. 25 (Wed) WG6 Plenary , 23 & , 23 & Social Event Nov. 26 (Thu) WG6 Plenary & & , 23 & , 23 & New EM Nov. 27 (Fri) WG6 Closing Plenary WG6 Closing Plenary WG6 Closing Plenary 4. DOCUMENTS 4.1 Base documents Category SC7 Doc # WG6 Doc # Document Title admin 3520 WG6 Convener Report, Closing Plenary, Bangkok, admin n 3538 Resolutions JTC 1/SC7 Plenary Meeting, Bangkok, Thailand, to 19 Comment disposition report - FCD Software engineering: Software product Requirements and Evaluation (SQuaRE) Planning and management FDIS Software engineering: Software product Requirements and Evaluation (SQuaRE) Planning and management. Sent to ITTF for FDIS processing Result of Voting on New Work Item Proposal - System and software product - Requirements and Evaluation (SQuaRE). Common industry Format for Usability - General framework for usability-related information

6 2506n 3539 Calling Notice of Meeting, JTC1/SC7-TC159/SC4 JWG on Common Industry Format for Usability, Gaithersburg, Maryland, USA, August 24 (Thursday) to 25 (Friday), 2006 Letter Ballot Summary - N CD25012 Software engineering: Software Requirements and Evaluation (SQuaRE). Data Model Letter Ballot Summary - N FCD Software engineering - Software quality requirements and evaluation (SQuaRE) - requirements Meeting Minutes, WG06, Evaluation and Metrics, Bangkok, Admin Thailand, to 19 Comment Disposition Report, N3435, CD Software engineering: Software Requirements and Evaluation (SQuaRE) Data Model CD Software engineering: Software Requirements and Evaluation (SQuaRE) Data Model Joining Instructions, SC7/WG6 Interim Meeting, Oct. Admin , Salvador, Brazil Admin Calling Notice and Draft Agenda for Interim Meeting of SC7/WG06, Oct. 2006, Salvador, Brazil Letter Ballot Summary,SC7 N3574, CD Software engineering: Software Requirements and Evaluation (SQuaRE) Data Model Comment Dispositions FDIS draft_ ISO v FDIS for Ballot3-Figure4Rev Abstract Disposition-FCD_ _Requirements_v7_kap comment disposition reports _Evaluation_Guide_CD_30_08_2006 NP 581 New Work Item Proposal for WG (SQuaRE) NP 582 Evaluation Module for Recoverability - Study Group Report V Comment disposition report of DTR measure elements Ver JTC1 N8143: Summary of voting on N 8072 Text for DTR Software Product Requirements and Evaluation (SquaRE). Measurement Primitives 4.2 Salvador meeting documents: Document Document title # 7/6-Sal-01 WG6 Project Status (revised) 7/6-Sal-02 Document List for WG6 Salvador Meeting (revised) 7/6-Sal-03 Participants list of WG6 Salvador meeting 7/6-Sal-04 WG6 Salvador Meeting Plan 7/6-Sal-05 SC7/WG6 Evaluation Module for Recoverability Study Group Report (ppt) 7/6-Sal-06 Guidance for writing 25022, 25023, and 25024

7 7/6-Sal-07 Notes from discussion of the Model 7/6-Sal-08 Proposed revision of ISO/IEC REPORT OF WG6 SALVADOR MEETING : Planning and management No discussion because of under FIDS ballot period. No remaining issues! FDIS Ballot Closed => 2006/10/30! To discuss based on the FDIS ballot result in St. Petersburg : model should explain the relationship between software quality and data quality. Functionality: promote the subcharacteristics security and probably interoperability to characteristics. Revise the definition of functionality: The capability of the software product to provide functions which meet stated and implied needs when the software is used under specified conditions. to reflect the remaining subcharacteristics. Clarify the difference between: 1. Maturity: The capability of the software product to avoid failure as a result of faults in the software, and 2. Fault tolerance: The capability of the software product to maintain a specified level of performance in cases of software faults or of infringement of its specified interface. in use

8 quality in use effectiveness productivity safety satisfaction Revise the model into characteristics and subcharacteristics such as: Context of Use User compatibility Task compatibility Environmental compatibility Usability in use Effectiveness in use Productivity in use Satisfaction in use o o o Likability (cognitive) Emotionality (pleasure etc) Operator health and safety Adaptability in use Learnability in use: usability in use measured for the task of achieving adequate performance, for example, by completing a training course or through the use of learning materials Universality in use: usability in use measured in a range of different contexts and cultures. Accessibility in use: usability measured for users with particular disabilities Risk in use People, property and environmental safety in use Commercial risk in use

9 Technical risk in use Security in use Liaise with the JWG to ensure consistency with the CIF Check how usability is measured in six sigma External measures can be measured: automatically, by an expert, or by user evaluation Obtain definitions and subcharacteristics for security and interoperability. Rewrite the explanatory clauses. CD after St Petersburg Send initial premature document => 2006/11/30 WG6 comments FB => 2006/12/31 Revise the document and Distribute to WG6 => 2007/1/30 WG6 feedback => 2007/2/28 Re-revise Forward CD after St. Petersburg => 2007/4/E : Data quality Major discussion points are: 1. define context of use of DQ Model within SQUARE in a system perspective, 2. define DQ characteristics categories (internal, external, in use DQ), 3. clarify relationship between and 25012, 4. analyze the comments received. Resolutions 1: and Data " Change of definition of ISO/IEC 2501n - Model Division. ACTUAL: The standard that forms this division presents a detailed quality model including characteristics for internal, external and quality in use. Furthermore, the internal and external software quality characteristics are decomposed into subcharacteristics. Practical guidance on the use of the quality model is also provided.

10 PROPOSED: The standards that form this division present detailed quality models for software and data including characteristics for internal, external and quality in use. Furthermore, for software the internal and external software quality characteristics are decomposed into subcharacteristics. Practical guidance on the use of the quality models is also provided. SQUARE QUALITY MODELS Internal Software External Software Software In Use Software Model Data Model Internal Data External Data Data In Use System Information system Computer system Computer hardware Other Software Target Software Data Human Business process Mechanical system Computer System Model Internal System External System System In Use Resolution 2: DQ characteristics categories INTERNAL DATA : Data stored in a physical support INTERNAL Data : the capability of a set of coded and stored data to satisfy stated and implied needs when the data are used under specified conditions. NOTE 1: From this perspective data quality characteristics refer to data itself; they provide the criteria to ensure and verify the quality of : data values data domains data meaning rules (i.e.: domain restriction derived by business rules) relationships between data (i.e.: integrity constraints at conceptual level)

11 taking account of documentation, files, and the real world Example : from the internal point of view currency refers to required status of the data independently of computer system processing it. ETERNAL DATA: Data communicated to a user ETERNAL Data : the capability of the communicated data to satisfy the receiving user stated and implied needs when the data are communicated to a user under specified conditions within a computer system. External data quality depends on those specific needs and on those specific conditions of use. NOTE 1: From this perspective data quality characteristics provide the criteria to ensure and verify the quality of : data input data presentation software Example: security depends also on HW and SW capabilities, that are applicable to certain data depending on user requirement. NOTE 2: Ways of viewing internal, external, in use data quality are different from the ones used to view software quality (ISO/IEC Software Engineering-Software Product Requirements and evaluation (SQuaRE). Guide to SQuaRE.2005 ) NOTE 3: Internal Data is not able to predict external data quality; NOTE 4: measures are different for the same characteristic for internal/external data quality. External Data may be considered (and evaluated) at three stages a) External Data of input data: Data is evaluated before software processing. Data quality characteristics will be evaluated by software, users or experts. b) External Data of presented data: Data is evaluated after software processing. Data quality characteristics will be evaluated by experts or automatic tools. c) External Data due to the implementation of computer system capabilities: Data is evaluated independently from target software, by using HW/SW

12 computer system capabilities. Data quality characteristics will be evaluated by experts or automatic tools. DATA IN USE: data communicated to a user as perceived and interpreted by the user Data in Use: the capability of the data to enable users to achieve specific goals with success and satisfaction in specific contexts of use. NOTE 1: From this perspective data quality characteristics provide the criteria to ensure and verify the quality of data for user information needs and job objectives. NOTE 2: Data in use characteristics express the user s point of view about data he is working with, in terms of how such data satisfies his information needs and how that data help him to perform his job. Resolution 3: and Data Categories DQ characteristics INTERNAL DATA QUALITY (value, domain, meaning, rules, relationships) DATA INTRINSIC LEVEL ETERNAL DATA QUALITY (DATA & SYSTEM INTERACTION) Computer input (1) presentation(2) System(3) DATA QUALITY IN USE (user needs, job performances) Consistency (consistent input format) (consistent representation) Currentness Completeness (completeness of input) Precision Accuracy Security Availability Recoverability Understandability Manageability Efficiency Changeability Portability Productivity Safety Credibility Accessibility Regulatory compliance Resolution 4: & A. The computer system does not include target software for SW Model (i.e: if CS includes only COTS or when only data quality evaluation is required). Data model

13 is fully independent. B. The computer system include target software of SW Model. Each Model contributes to CS providing requirements and measures from its own perspective. Resolution 5: comment disposition 94 comments received 83 OK 11 Rejected with reasons or overcome by events To explain better relationships with and To introduce cost considerations Editorial work Revise the document and Distribute to WG6 => 2006/11/30 WG6 feedback => 2006/12/15 Revise document and send it to WG6 Convener and Secretariat => 2007/01/15 Revise document and send it to SC7 Secretariat => 2007/01/22 SC7 secretariat send it for 3rd CD Ballot => 2007/01/29 Ballot Closed => 2007/04/29 Letter Ballot Summary => 2007/05/ : Measurement reference model and guide No action because the final version for publication has already sent to SC7 secretariat.. The project editor will inquire the status for publication to SC7 secretariat by , attaching the abstract, final revision and comment disposition report for confirmation.! End of project : measure elements Review all the comments disposed Alignment with (definitions, SPQM-RM, document structure)

14 Reformat the cross reference tables according to Japanese comments. Agreement on the following issues Changing Recommended QMEs to Initial set of Recommended QMEs Changing QM in Annex A to Recommended QMs Addition of QMEs for Product size in next version Preparing for publication Revise the document and Distribute to WG6 => 2006/11/15 WG6 feedback => 2006/12/15 Revise document based on feedback and send it to publication => 2006/12/ : Measurement of internal quality Discussion Points Reviewed Table format for documenting quality measures and produced some examples and definitions Identified content to be covered in text Began review of quality measures to include in Resolution Add Song Changsun (Korea) as co-editor Editorial assignments were decided as below: General Text Dr. Zubrow Efficiency and Maintainability Dr. Yuan Reliability and Portability Mr. Song

15 Functionality and Usability Mr. Tanitsu Resolve how to handle missing QMEs for Efficiency and Maintainability Need to complete definitions of fields Need to decide how to address text components Review measures and QMEs for inclusion Decide whether to retain provisionally included fields Draft new content Editorial team share and review initial drafts of sections => 2006/12/5 Revise the document and Distribute to WG6 => 2007/1/22 WG6 feedback => 2007/2/15 Revise document and send it to WG6 Convener and Secretariat => 2007/03/15 Revise document and send it to SC7 Secretariat => 2007/03/22 SC7 secretariat send it for WD and CD Registration Ballot => 2007/03/29 WG6 bring national body comments to SC7 Plenary => 2007/05/20 Ballot Closed => 2007/06/29 Letter Ballot Summary => 2007/07/ : Measurement of external quality Clarification of table format for external quality measures are discussed combined with and Preparation of initial WD. No specific plan : Measurement of quality in use Revise to include measures for the new model for quality in use in

16 Cannot be completed until the model for quality in use is finalised. No specific plan : requirements Comments disposition report was reviewed and comment resolutions were confirmed Minor additional editorial issues were discussed and resolved None Revise document and send it to SC7 Secretariat => 2006/11/15 SC7 secretariat send it for FDIS Ballot => 2006/11/22 Ballot Closed => 2007/01/22 Letter Ballot Summary => 2007/02/ : Evaluation reference model and guide During this meeting the WD and CD Registration ballot results were discussed. There were 40 comments to be addressed (28 from USA (6TH), 10 from Japan and 2 from Canada (1TH); All the technical comments were discussed and addressed by the editor and the countries members; Other points were raised during the discussion, such as: " Scope of each document of the 2504n division; " Evaluation reference model (how to represent) " style (show only the activities) " style (show Input, Output and activities) " a different style; " The need to prepare an example of application of evaluation process " The quality in use evaluation process;

17 " Possible implication if necessary to address risk analysis (project and implications of use) " How to address Safety taking into account the safety community work? " Could be a free document? The following resolutions were adopted: " The work on continues considering that all the division documents will be prepared according to the original definition as stated in ISO/IEC 25000; " The diagram representing the evaluation reference model will be revised in order to possibly represent input, output, control and technical support for each activity; " An example for the evaluation process will be included in the document. Some points remain pendent for further discussion, such as: Possible standard structure for documents to 25044; Use of normative statements in ISO/IEC and in to 25044; The quality in use evaluation process; Possible implication if necessary to address risk analysis (project and implications of use) How to address Safety taking into account the safety community work? Could be a free document? Revise the document and Distribute to WG6 => 2006/11/30 WG6 feedback => 2006/12/15 Revise document and send it to WG6 Convener and Secretariat => 2007/01/15 Revise document and send it to SC7 Secretariat => 2007/01/22 SC7 secretariat send it for 1st CD Ballot => 2007/01/29 Ballot Closed => 2007/04/29 Letter Ballot Summary => 2007/05/ Other Topics Study group on evaluation module for recoverability Discussion points and resolutions: Customers are interested in recoverability and degree of self-management of this sub-characteristic in the software products and thereby the systems they run on.

18 Study group investigated the possibility of documenting an Evaluation Module for Recoverability under ISO/IEC to provide a standard to measure and report results on: The availability of a system when facing externally injected disturbances, And how well the system detects, analyses, and recovers from these disturbances on its own. SG identified quality measures presentation for details) and methodology for evaluation (see report and SG recommends to propose new work item for this EM to SC7 plenary Remaining issues: Revise study group report to fix up terminology and minor clarification E.g. Use quality measure, not metric Plan for next stage and milestones: Revise the document and distribute to WG6 => 2006/10/30 WG6 feedback => 2006/11/30 Revise document and send it to convener and secretariat of WG6 => 2007/01/22 CIF joint working group The 1 st JWG between ISO/IEC JTC1/SC7 and ISO TC159/SC4 was held in August, 2006 and Dr. Bevan made a brief report on it. Participation to the JWG was encouraged, especially to the countries expressing the intention with the NP. Appointment of editor and co-editor The following new co-editors is appointed: Song Changsun (Korea) as co-editor of ISO/IEC 25022: Measurement of internal quality Markku Tukiainen (Finland) as co-editor of ISO/IEC 25024: Measurement of quality in use The following new editors and co-editors will be appointed formally when NP of evaluation module for recoverability are accepted: Peter Shum (Canada) as editor. Vipula Godamunne (Australia), Danilo Scalet (Brazil) and Jorgen Boegh (Denmark) as co-editor. 6. MEETING PLAN The following meeting plan was discussed and approved.

19 2007, May 21-25, St. Petersburg => Moscow, Russia 2007, October 7-12, Pittsburgh, USA 2008, May, Berlin, Germany (To be confirmed) 2008, October, Japan <if not in 2010> (To be confirmed, ) 7. APPRECIATION SC7/WG6 extends its appreciation to the Brazilian Member Body for hosting and supporting the thirty-first WG6 Meeting of ISO/IEC JTC1/SC7 in Salvador, Brazil.