EUROPEAN MIDDLEWARE INITIATIVE

EUROPEAN MIDDLEWARE INITIATIVE DSA2.3.1 - PERIODIC QA REPORTS EU DELIVERABLE: D4.3.1 Document identifier: EMI-DSA2.3.1-QAReport-Final.doc Date: 31/07/2010 Activity: Lead Partner: Document status: Document link: SA2 CERN Final http://cdsweb.cern.ch/record/1277600?ln=en Abstract: The Periodic QA Reports contain a description of the compliance with and results of the Software Quality Process. They are created once per year in PM3, PM12, PM24 and PM36. INFSO-RI-261611 2010 Members of EMI collaboration PUBLIC 1 / 18

DSA2.3.1 - PERIODIC QA REPORTS Doc. Identifier: EMI-DSA2.3.1-1277600-Periodic_QA_Reports_v1.0.docx Date: 31/07/2010 Copyright notice: Copyright (c) Members of the EMI Collaboration. 2010. See http://www.eu-emi.eu/about/partners/ for details on the copyright holders. EMI ( European Middleware Initiative ) is a project partially funded by the European Commission. For more information on the project, its partners and contributors please see http://www.eu-emi.eu. This document is released under the Open Access license. You are permitted to copy and distribute verbatim copies of this document containing this copyright notice, but modifying this document is not allowed. You are permitted to copy this document in whole or in part into other documents if you attach the following reference to the copied elements: "Copyright (C) 2010. Members of the EMI Collaboration. http://www.eu-emi.eu ". The information contained in this document represents the views of EMI as of the date they are published. EMI does not guarantee that any information contained herein is error-free, or up to date. EMI MAKES NO WARRANTIES, EXPRESS, IMPLIED, OR STATUTORY, BY PUBLISHING THIS DOCUMENT. EMI RI-261611 Members of the EMI collaboration PUBLIC 2 / 18

Delivery Slip Name Partner / Activity Date Signature From Maria Alandes Pradillo CERN/SA2 03/09/2010 Reviewed by Claudio Cacciari Morris Riedel CINECA/SA1, JUELICH/JRA1 15/10/2010 Approved by PEB 18/10/2010 Document Log Issue Date Comment Author / Partner 1 03.09.2010 First Version Maria Alandes 2 22.09.2010 Changes after internal review in SA2 Maria Alandes 3 07.10.2010 Changes after official review by SA1 Maria Alandes 4 11.10.2010 Changes after official review by JRA1 Maria Alandes 5 15.10.2010 Minor changes after second round of official reviews. Maria Alandes Document Change Record Issue Item Reason for Change 1 2 3 INFSO-RI-261611 2010 Members of EMI collaboration PUBLIC 3 / 18

TABLE OF CONTENTS 1. INTRODUCTION... 5 1.1. PURPOSE... 5 1.2. DOCUMENT ORGANISATION... 5 1.3. REFERENCES... 5 1.4. DOCUMENT AMENDMENT PROCEDURE... 7 1.5. TERMINOLOGY... 7 2. EXECUTIVE SUMMARY... 8 3. OVERVIEW OF THE SQA PROCESS... 9 4. STATUS OF THE MINIMUM DOCUMENTATION REQUIREMENTS... 10 5. STATUS OF DOCUMENTATION... 13 5.1. TECHNICAL DEVELOPMENT PLAN... 13 5.2. SOFTWARE RELEASE PLAN... 13 5.3. SOFTWARE MAINTENANCE AND SUPPORT PLAN... 13 5.4. QA TOOLS AND DOCUMENTATION... 13 5.5. CONTINUOUS INTEGRATION AND CERTIFICATION TESTBEDS... 13 6. STATUS OF GUIDELINES... 14 6.1. CONFIGURATION AND INTEGRATION... 14 6.2. PACKAGING AND RELEASING... 14 6.3. CHANGE MANAGEMENT... 15 6.4. METRICS GENERATION... 15 6.5. CERTIFICATION AND TESTING... 16 7. CONCLUSIONS... 17 INFSO-RI-261611 2010 Members of EMI collaboration PUBLIC 4 / 18

1. INTRODUCTION 1.1. PURPOSE The purpose of the Periodic QA Reports is to inform about the progress and implementation of the software quality assurance process within the EMI project. This report provides an overview of the current status of relevant QA documents and guidelines after the first months of the EMI project. 1.2. DOCUMENT ORGANISATION The document contains a series of reports of different elements of the SQAP. It is organised as follows: Overview of the SQA process. Status of the Minimum Documentation Requirements for the EMI software components. Summary of QA activities carried out during the first three months of the project. o Status of the documents that are part of the SQAP. o Conclusions Recommendations 1.3. REFERENCES Status of the guidelines to be prepared by SA2. R1 R2 R3 R4 R5 R6 R7 EMI Software Quality Assurance Plan https://twiki.cern.ch/twiki/pub/emi/deliverabledsa21/emi-dsa2.1-1277599-qa_planv1.2.doc SQAP Review Schedule https://twiki.cern.ch/twiki/bin/view/emi/sqap#sqap_review_schedule Technical Development Plan https://twiki.cern.ch/twiki/bin/view/emi/deliverabledna131 Software Release Plan https://twiki.cern.ch/twiki/bin/view/emi/deliverabledsa12 Software Maintenance and Support Plan https://twiki.cern.ch/twiki/bin/view/emi/deliverabledsa11 Continuous Integration and Certification Test beds https://twiki.cern.ch/twiki/bin/view/emi/deliverabledsa24 Service Reference Card Template https://twiki.cern.ch/twiki/bin/view/emi/emtsrctemplate INFSO-RI-261611 2010 Members of EMI collaboration PUBLIC 5 / 18

R8 R9 R10 R11 R12 R13 R14 R15 R16 R17 R18 R19 R20 R21 Summary of Minimum Required Documentation status for QA report https://twiki.cern.ch/twiki/bin/view/emi/qar1#minimum_documentation_requirem en Configuration and Integration Guidelines https://twiki.cern.ch/twiki/bin/view/emi/emisa2configurationintegrationguidelines Packaging and Releasing Guidelines https://twiki.cern.ch/twiki/bin/view/emi/emisa2packagingreleasingguidelines Change Management Guidelines https://twiki.cern.ch/twiki/bin/view/emi/emisa2changemanagementguidelines Metrics Generation Guidelines https://twiki.cern.ch/twiki/bin/view/emi/emisa2metricsgenerationguidelines Certification and Testing Guidelines https://twiki.cern.ch/twiki/bin/view/emi/emisa2certtestguidelines JIRA http://www.atlassian.com/software/jira/ EPEL http://fedoraproject.org/wiki/epel Debian http://www.debian.org/ Fedora http://fedoraproject.org/ EGEE http://www.eu-egee.org/ ETICS http://etics.web.cern.ch/etics/ YUM http://yum.baseurl.org/ APT http://wiki.debian.org/apt INFSO-RI-261611 2010 Members of EMI collaboration PUBLIC 6 / 18

1.4. DOCUMENT AMENDMENT PROCEDURE This document can be amended by the authors further to any feedback from other teams or people. Minor changes, such as spelling corrections, content formatting or minor text re-organisation not affecting the content and meaning of the document can be applied by the authors without peer review. Other changes must be submitted to peer review and to the EMI PEB for approval. When the document is modified for any reason, its version number shall be incremented accordingly. The document version number shall follow the standard EMI conventions for document versioning. The document shall be maintained in the CERN CDS repository and be made accessible through the OpenAIRE portal. 1.5. TERMINOLOGY APT EGEE EMT EPEL PEB PT PTB QA SQA SQAP YUM Advanced Packing Tool Enabling Grids for e-science Engineering Management Team Extra Packages for Enterprise Linux Project Executive Board Product Team Project Technical Board Quality Assurance Software Quality Assurance Software Quality Assurance Plan Yellow Dog Updater INFSO-RI-261611 2010 Members of EMI collaboration PUBLIC 7 / 18

2. EXECUTIVE SUMMARY The first Periodic QA Report presents the status of different elements that are part of the SQAP [R1] after the first three months of the project. The SQAP is a document that specifies tasks that are needed to ensure quality, responsibilities for quality monitoring, documentation required and procedures that will be followed to manage the software process. In practice the SQAP specifies the manner in which the EMI project is going to achieve its quality goals. At the moment of writing this report, no quality monitoring activities have been carried out yet (SQA reviews) according to the review schedule. It s foreseen to report about the review activities in further reports. This document is mainly focused on the status and availability of documentation and metrics. The following items will be presented: Overview of the SQA Process. A report on how the SQA Process has been organised in the SA2 activity will be presented. Status of the minimum documentation requirements for each software component. As described in the SQAP, a set of documents have been identified to describe the development, verification, validation, use and maintenance of the software. This report will evaluate the availability of the documents for each software component. Status of the documents that are part of the SQAP. In the first months of the project, the documents governing the software lifecycle of the EMI releases are produced. This report will give an overview on the current status of those documents. Status of the guidelines to be followed by JRA1 and SA1 to develop software for EMI, as described in SQAP. It was agreed that SA2 will provide a set of guidelines describing best practices for different aspects of the software development, like integration, configuration or change management. The guidelines were not ready for the first version of the SQAP and they will be included as annexes in a future update of the SQAP. Announcements on the status of the guidelines will be done in the EMT. The report also presents some conclusions. Documents that are key elements in the software lifecycle process are still missing and SA2 considers this situation should be improved as soon as possible. Documents may not be ready in 100% but at least should describe the basics so PTs can start working. The situation with the guidelines is very similar although important decisions had already been made, like choosing ETICS as the build and test tool for the project. Technical discussions are happening and this has shown that harmonising the way of working of different middlewares is sometimes difficult. Guidelines are also believed to evolve throughout the lifetime of the project and QA activities should guarantee up to date documentation and involvement of PTs in the definition of best practices. The report also contains some recommendations to improve the situation from the QA perspective. The importance of documentation describing the software lifecycle has to be stressed by SA2. SA2 needs to create an awareness that it s very important to prepare a first version of the documentation early in the project even if updates and improvements can be added later on. QA activities should also put effort on monitoring the status of the guidelines so that documentation is up to date and changes are communicated to PTs at the EMT. INFSO-RI-261611 2010 Members of EMI collaboration PUBLIC 8 / 18

3. OVERVIEW OF THE SQA PROCESS The SQAP [R1] is a document that specifies tasks that are needed to ensure quality, responsibilities for quality monitoring, documentation required and procedures that will be followed to manage the software process. In practice the SQAP specifies the manner in which the EMI project is going to achieve its quality goals. The SQAP defines which documents are needed to describe the software development process [R3], [R4], [R5] and [R6] as well as best practices called guidelines that developers should follow to produce quality software [R9], [R10], [R11], [R12] and [R13]. It also defines a set of minimum required documentation for software components as explained in section 4. The SQAP also defines SQA reviews. The reviews are one of the main activities in SQA. It monitors the status all the documents mentioned above and whether they actually meet the quality factors defined in the SQAP. The schedule of the reviews is summarised in the SQAP Review Schedule [R2]. Review templates are available and accessible from [R2]. The person responsible for each review is also defined in [2]. Up to now no reviews have been performed yet as defined in the schedule. Due to this, the SQA process is at this point mainly focused on writing the guidelines that define in detail the software development process. The status of this activity is described in more detail on section 6. INFSO-RI-261611 2010 Members of EMI collaboration PUBLIC 9 / 18

4. STATUS OF THE MINIMUM DOCUMENTATION REQUIREMENTS As explained in the SQAP, a set of documents have been identified to describe the development, verification, validation, use and maintenance of the software. This section includes the results of evaluating the availability of the documents for each software component, in particular: Software Requirements Specifications: description of the essential requirements of the software. Software Design Description: description of the major components of the software including data bases and internal interfaces. Software Verification and Validation Plan: overall plan for the verification and validation of the software including the methods that have to be used. Software Verification and Validation Report: results of the execution of the Software Verification and Validation Plan. User Documentation: all the documentation needed by the users of the software so that they can use the system. Installation Guide: all the necessary instructions to be able to install and configure the software. Troubleshooting guide: compilation of the most common error situations which can occur and how to react. In order to get the information on the availability of the documentation, either the documents were found using the existing middleware distribution release information, or by getting it directly from the product teams. In most cases, there are no requirement specification documents. According to the SQAP, this document should be now provided by the PTB for each EMI major release. A new document, the Service Reference Card, has been requested by the EMT and it will be added to the set of the Minimum Required Documentation in the SQAP. The Service Reference Card contains information useful for system administrators like daemons running when a service has started, ports that need to be opened, cron jobs and log files, etc. It is a document already used by glite during the EGEE [R18] project and was proven to be very useful for system administrators. EMT has therefore requested to create Service Reference Cards also for ARC, UNICORE and dcache as well. A Service Reference Card should be defined for each software component. A template for Service Reference Cards has been already created by SA1 [R7]. Product Teams have already started to create their Service Reference Cards. However the status of the Service Reference Cards per software component is not included in this particular QA report. It will be included in the upcoming QA reports. The approach now is to contact product teams to inform them about the lack of documentation where applicable. In the table below, the current status is presented. The legend is: Y: The document exists. N: The document doesn t exist. INFSO-RI-261611 2010 Members of EMI collaboration PUBLIC 10 / 18

In progress: The document is being created now.?: document couldn t be found or the product team didn t provide any information. A summary of the status with links to the documents are available in [R8]. Find a summary with the status in the table below. EMI components Software Requirements Specifications Software Design Description Software Verification and Validation Plan Software Verification and Validation Report User Documentation Installation Guide AMGA N N Y N Y Y N APEL N Y Y Y Y N Y ARGUS N Y Y Y Y Y Y BDII N Y Y N Y N N CREAM N Y Y Y Y Y Y DPM N Y Y Y Y Y N FTS Y Y Y Y Y Y N Hydra N Y Y N Y Y N LB N N Y Y Y Y N LFC N N Y Y Y N N VOMS N Y Y Y Y Y Y WMS N Y Y Y Y Y Y A-Rex Y Y N Y Y Y Y ARC Compute Elements ARC Data Clientlibs ARC Infosys ARC Security utils ARC Container Unicore TSI Unicore XNJS Unicore UAS Unicore Service Registry Unicore UAS Y Y N N Y N N Y Y N N N N N Y Y N N Y N N Y Y N N Y N N Y Y N Y N N N N Y N N Y Y N N Y N N Y Y N N Y N N Y Y N N Y N N Y Y N N Y N N Y Y N Troubleshooting guide INFSO-RI-261611 2010 Members of EMI collaboration PUBLIC 11 / 18

(Data) dcache N N Y N Y Y N DGAS N Y N N Y Y N MPI In In N N Y Y Y Progress Progress SAGA Y N Y Y Y N N StoRM N N N N Y Y Y INFSO-RI-261611 2010 Members of EMI collaboration PUBLIC 12 / 18

5. STATUS OF DOCUMENTATION The following documents are considered to have a key role in driving the software lifecycle and in the delivery of quality software. They have been included in the SQAP and will be reviewed regularly to ensure they are up to date and that they meet the general goals of the project as far as software quality is concerned. 5.1. TECHNICAL DEVELOPMENT PLAN The progress of the Technical Development Plan can be tracked in [R3]. The document is still a working draft not yet submitted for internal review. SA2.2 has received comments about the possibility of better tracking the status of the Technical Work Area plans. The Technical Development plan is indeed a higher level document which is not supposed to change with every quarter. Therefore, it s been argued that there s no added value of reviewing it in every QA report. This will imply a change in the SQAP that hasn t been implemented yet at the time of writing this document. For future QA reports, it s therefore foreseen to include reviews of the Technical Work Area plans. 5.2. SOFTWARE RELEASE PLAN The progress of the Software Release Plan deliverable can be tracked in [R4]. The document is in progress. The last version contains the description of the current status of the release process of the different middleware distributions. Once the document is finished it will be considered in the next QA report. 5.3. SOFTWARE MAINTENANCE AND SUPPORT PLAN The progress of the Software Maintenance and Support Plan can be tracked in [R5]. The document is still a working draft but with complete contents for all the sections. It contains the feedback and discussions collected during the definition of the Change Management Guidelines. 5.4. QA TOOLS AND DOCUMENTATION The QA Tools and Documentation deliverable Plan has not started yet. It s a document that hasn t been considered as essential for the first months of the project and therefore its definition has been postponed. 5.5. CONTINUOUS INTEGRATION AND CERTIFICATION TESTBEDS The progress of the Continuous Integration and Certification Testbeds deliverable can be tracked in [R6]. The deliverable has been submitted for review. Minor changes may be integrated regarding the way changes in the testbed will be communicated to PT community. INFSO-RI-261611 2010 Members of EMI collaboration PUBLIC 13 / 18

6. STATUS OF GUIDELINES The following guidelines are defined as part of the SQAP to help development teams to follow a set of best practices addressing different aspects of the software. Experts from ARC, Unicore, dcache and glite are involved in the definition of the guidelines. Once the guidelines are finished, they will be included in a future update of the SQAP. They will be reviewed as part of the formal review process of the SQAP. 6.1. CONFIGURATION AND INTEGRATION The Configuration and Integration Guidelines aim at defining the following: Integration of the ARC, Unicore, dcache and glite in a single unified build process from source. Integration of external dependencies. Selection of common sources for external dependencies. Usage of a single tool for building/integrating. Unified and standardized configuration of software. The following has been agreed: A single tool will be used to perform integration builds from source (nightly builds). All build tools will be supported and integrated via this unified integration system. ETICS [R19] is the only tool able to handle an integration build from source of all the four middleware distributions. Therefore it will be used to produce a first unified integration system. ETICS Worker Nodes will be configured to use a minimal VM image with only the set of packages required to install other ones. Enabled YUM [R20] and APT [R21] repositories will be OS base + EPEL [R15] or Debian [R16] UNSTABLE if needed. Everybody will use the latest default OS version installed using YUM or APT-GET. If a package not available in OS or EPEL is required, the requester will be responsible of creating the proper packages building the package from source and making it available as an ETICS external. Configuration templates which define how to add software of different languages/build systems in ETICS. The guidelines are now completed and can be found in [R9]. 6.2. PACKAGING AND RELEASING The Packaging and Releasing Guidelines aim at defining the following: How to produce packages which will be acceptable for linux distributions. INFSO-RI-261611 2010 Members of EMI collaboration PUBLIC 14 / 18

How to minimize the effort for producing packages compliant with Fedora [R17], EPEL and Debian guidelines. What packaging formats will be supported by EMI for what platforms How to produce metapackages, source packages, and buildable source packages What tools to use and how to configure them for packaging There hasn t been much progress on these guidelines since it s not considered to be as urgent as the other ones. The available material can be consulted in [R10]. 6.3. CHANGE MANAGEMENT The Change Management Guidelines aim at defining the following: A process to describe how to manage software changes in the EMI middleware. The contents and life cycle of a Request for Change. The roles involved in the management of Changes. The Release Candidate format and the tools to manage them. After collecting feedback from the different middleware distributions, a first version of the guidelines was proposed taking into account the different scenarios. The guidelines have been also defined based on the contents of the Software Maintenance and Support Process [R5]. The guidelines are general and tool independent so that changes can be implemented using any tracking tool. More details will be added for the JIRA [R14] objects that will represent the release candidates once the release manager starts preparing EMI 1 release. The guidelines were presented to the EMT and after a period of one week were considered accepted. After some feedback from the four middleware distributions it was clarified that the guidelines implied changes in the tracking tools. This wasn t very welcomed by the middlewares and after a discussion (summarized in http://indico.cern.ch/getfile.py/access?resid=minutes&materialid=minutes&confid=106263), a mapping between the proposed Request for Change and Change Status was done for each middleware so no changes are needed in the tracking tool. The mappings are now added to the guidelines. It s forseen to update the guidelines with more specific details once the product teams start to prepare the first EMI release. The guidelines can be found in [R11]. 6.4. METRICS GENERATION The Metrics Generation Guidelines aim at allowing ARC, UNICORE, dcache and glite to obtain a set of metrics that can be defined consistently for all of them. This means that software and bugtracking related metrics should be specified identically in each middleware, where possible. The Configuration and Integration Guidelines state that ETICS should be used to build EMI releases. This means that it is possible to define any software related metric inside ETICS. Hence, there should INFSO-RI-261611 2010 Members of EMI collaboration PUBLIC 15 / 18

be no availability issue within each middleware for producing these software related metrics. For a complete list of defined metrics, please refer to the SQAP. Similarly, the bug-tracking related metrics should use a common set of definitions. However, glite is using Savannah with a much smaller granularity of bug-tracking states, bug detection areas and severity levels than the other bug-tracking systems used by ARC, dcache and UNICORE. The initially defined set of bug-tracking related metrics contain too fine granularity, so this must be coarsened to suit all middlewares. This item is work in progress, requiring feedback from each middleware partner to come to a consensus. No interoperation or reusability metrics can be defined until a realistic software development plan for the combining of EMI-0 into EMI-1 is produced. Nevertheless, JRA1.7 - Interoperability and Integration works on identifying related pieces of information such as which components are interoperable with other components and such like. Whether these results are of relevance for this activity needs to be explored during the course of the project. Language specific bug density metrics will be defined to supplement the currently defined total bug density metric. Project wide metric thresholds need to be defined. The guidelines can be found in [R12]. 6.5. CERTIFICATION AND TESTING The Certification and Testing guidelines aim at defining a set of mandatory tests that all software components must develop as part of their test plans and also a set of testing reports to be used by product teams when doing the certification of their products. The guidelines are still in an early stage. A first proposal for testing guidelines and certification report has been defined and it s now been evaluated by the experts to gather feedback from all the middleware distributions. The guidelines are available in [R13]. INFSO-RI-261611 2010 Members of EMI collaboration PUBLIC 16 / 18

7. CONCLUSIONS The evaluation of the availability of the Minimum Documentation Requirements shows that some effort is needed to complete the missing documents. Most of the software components are missing requirements specifications and design description. This is expected to be defined after the Area Work Plans are ready. Support will be given to the different product teams so that they can provide the missing documentation. The support will be made through the definition of templates that product teams will have to fill in for each software component. In the case of the Validation and Verification Plans and Reports, the Certification and Testing guidelines will be the main source of information to guide product teams on defining these documents. Most of the documents monitored by the SQAP are still in progress and in some cases discussions are held and consensus has to be reached among the different middleware distributions to be able to finish the documents. This is a long and complex process and this is the reason why some documents haven t been able to meet their deadlines. The definition of the guidelines is also a key activity to establish best practices across the whole software lifecycle process. Some of the deliverables are indeed connected to the guidelines and there s an interdependency among different documents, which makes it also more complicated to finish preparing the documentation. The guidelines have triggered interesting discussions and in some cases they have successfully reached agreements among the different middlewares. Some guidelines are still in progress. Practice has shown that it was difficult to have 100% of the guidelines completed by the time PTs started to work in the first EMI release. It was also clear that once PTs actually started to follow the guidelines, they were giving useful feedback for changes and improvements based on practical experience. This means that guidelines should evolve throughout the lifetime of the project based on real experiences and adapting to the needs of the project. One of the goals of the QA activities will be to keep the guidelines up to date, gathering feedback from PTs and involving them in the process. INFSO-RI-261611 2010 Members of EMI collaboration PUBLIC 17 / 18

8. RECOMMENDATIONS QA activities should put more effort on following up the definition of requirements specifications and design description together with work package leaders and with the help of the EMT. We consider these documents, as it was understood when preparing the SQAP, are key elements in the description of what needs to be developed by each software component. In order to improve the quality assurance process, an effort is needed to make sure the missing deliverables described in this document will be provided as soon as possible. A list of missing documents will be presented to the SA2 activity leader every week so that this is reported at the PEB meeting. We believe that failing to provide these deliverables in the short term will have negative impact on many areas of the project affecting the quality of the software. We consider it s more useful to have the deliverable as soon as possible always taking into account that it may change and evolve in time. INFSO-RI-261611 2010 Members of EMI collaboration PUBLIC 18 / 18