EUROPEAN MIDDLEWARE INITIATIVE

Transcription

1 EUROPEAN MIDDLEWARE INITIATIVE DSA2.4 - CONTINUOUS INTEGRATION AND CERTIFICATION TESTBEDS EU DELIVERABLE: D4.4 Document identifier: EMI-DSA Integration_Testbeds_v1.0.doc Activity: Lead Partner: Document status: Document link: SA2 INFN Final Abstract: This document describes the distributed certification testbeds for internal and acceptance certification and its access and usage requirements. INFSO-RI Members of EMI collaboration PUBLIC 1 / 40

2 Copyright notice: Copyright (c) Members of the EMI Collaboration See for details on the copyright holders. EMI ( European Middleware Initiative ) is a project partially funded by the European Commission. For more information on the project, its partners and contributors please see This document is released under the Open Access license. You are permitted to copy and distribute verbatim copies of this document containing this copyright notice, but modifying this document is not allowed. You are permitted to copy this document in whole or in part into other documents if you attach the following reference to the copied elements: "Copyright (C) Members of the EMI Collaboration. ". The information contained in this document represents the views of EMI as of the date they are published. EMI does not guarantee that any information contained herein is error-free, or up to date. EMI MAKES NO WARRANTIES, EXPRESS, IMPLIED, OR STATUTORY, BY PUBLISHING THIS DOCUMENT. INFSO-RI Members of EMI collaboration PUBLIC 2 / 40

3 . Delivery Slip Name Partner / Activity Date Signature From Danilo N. Dongiovanni INFN / SA2 23/09/2010 Reviewed by Oliver Keeble, Morris Riedel, Balazs Konyua CERN/SA1, JUELICH/JRA1, LU/NA1 03/12/10 Approved by PEB 03/12/10 Document Log Issue Date Comment Author / Partner /07/2010 First draft to the attention of SA2 participants Danilo N. Dongiovanni /08/ /09/ /09/2010 Integrated feedbacks from Cristina Aifitimiei, Jozef Cernak and Tomasz Wolak. Added Executive Summary. Added EMI Testbed twiki automatic updates notifications (Sec ). Added use case of debugging information accessibility for tester users (Sec ). Added the notification of new Release Candidate product versions to EMI Testbed team (Sec ). Integrated feedbacks from Jozef Cernak on amendment procedure (Sec 1.5). Added reference on testing guidelines (Sec ) Danilo N. Dongiovanni Cristina Aiftimiei Jozef Cernak Tomas Wolak Danilo N. Dongiovanni Danilo N. Dongiovanni Jozef Cernak /09/2010 Reviewed Alberto Aimar /09/2010 Changed References. Highlighted Open issues in the Executive Summary. Mentioned in sec. 5 the testing VO created in the EMI testbed. Added the possibility of direct support requests to SA2.6 through savannah. Added the plan to setup a shift infrastructure for incoming support requests (sec ) Danilo N. Dongiovanni /09/2010 Reviewed Balazs Konya INFSO-RI Members of EMI collaboration PUBLIC 3 / 40

4 0.8 07/10/2010 Added some terms definition in terminology table. Added new section 4.1 to clarify about the usage of EMI Testbed and subset testbeds terms throughout the document. Added section reporting current implementation status of Testbed. Added EMI Testbed operation solutions and practice reference table. Changed Executive Summary. Danilo N. Dongiovanni /10/2010 Reviewed Morris Riedel /10/2010 Added paragraph 5.5 defining inputs expected from JRA1.7 Task. Danilo N. Dongiovanni /11/2010 Reviewed Oliver Keeble /11/ /11/2010 Minor corrections to text or presentation layer. Added integration testing definition. Mentioned EGI staged rollout procedure. Added reference on Release Management process Minor changes. Added reference to EMI products coverage table in section 7.1 Danilo N. Dongiovanni Danilo N. Dongiovanni /12/2010 Final version accepted by all reviewers Danilo N. Dongiovanni /12/2010 Final version for submission Alberto Di Meglio Document Change Record Issue Item Reason for Change INFSO-RI Members of EMI collaboration PUBLIC 4 / 40

5 TABLE OF CONTENTS 1. INTRODUCTION PURPOSE DOCUMENT ORGANISATION APPLICATION AREA REFERENCES DOCUMENT AMENDMENT PROCEDURE TERMINOLOGY EXECUTIVE SUMMARY EMI INTEGRATION TESTBED REQUIREMENTS IDENTIFICATION SA2.6 TASK DEFINITION SURVEY OF TESTING RESOURCES AND PROCEDURES CONVERGING INTO EMI REQUIREMENTS IDENTIFICATION Envisaged integration testing scenarios Effort definition, Procedures, Communication and Documentation needs EMI TESTBED IMPLEMENTATION MODEL EMI TESTBED OR TESTBEDS? EMI INTERNAL INTEGRATION TESTBED SOLUTION Hardware Resource dimensioning Product Teams members effort contribution Testbed monitoring solutions Resources access policy Testing Certificates handling Usage Monitoring and Concurrent usage Requirements Testbed update and evolution procedure Activity tracking, communication channels and procedures Public documentation LARGE-SCALE TESTBED IMPLEMENTATION PLAN Implementation plan OPEN ISSUES COMMON AUTHENTICATION FRAMEWORK ACROSS MIDDLEWARES SOFTWARE PRODUCT VARIABLES TO CONSIDER RESOURCE DISCOVERY AND PUBLISHING OVER DIFFERENT MIDDLEWARES TOOLS AND AUTOMATION INPUTS FROM JRA EMI TESTBED EVOLUTION AND FUTURE REVISIONS OF THE MODEL EMI TESTBED IMPLEMENTATION STATUS INSTALLED SERVICES INVENTORY ARC glite UNICORE dcache TESTBED OPERATIONS QUICK REFERENCE CONCLUSIONS INFSO-RI Members of EMI collaboration PUBLIC 5 / 40

6 INFSO-RI Members of EMI collaboration PUBLIC 6 / 40

7 1. INTRODUCTION 1.1. PURPOSE This document describes the implementation strategy of EMI Integration Testing testbeds, taking into account integration testing needs and resources made available from task participants and envisaged future evolutions. Apart from the testbed implementation model, also documentation, coordination procedures, communication channels and activity tracking tools are defined both for the EMI internal testbed and for the large-scale testbed in collaboration with external partners (less accurate level for this initial stage of the project) DOCUMENT ORGANISATION Section 1 and 2 are the introductory and the executive summary section respectively. Section 3 focuses on the collection and definition of requirements and targets for the EMI integration testbed. Input from the EMI description of work and testbed customers (survey and meeting with developers representatives) have been collected and summarized. These inputs have then been matched with SA2.6 task targets and actual resources and consolidated testing procedures are available across three middlewares (survey and meeting with ARC, glite, UNICORE representative task participants). Section 4 is the key section, presenting EMI internal integration testing testbed implementation plan including: testbed scenarios implementation model, documentation, coordination procedures, communication channels and activity tracking tools. Also a preliminary model for a large scale testing testbed is proposed. Section 5 focuses on the definition of some key issues having a strong impact on testbed implementation which need inputs or discussions from/with other EMI work packages. Section 6 reports about the testbed model evolution and revision APPLICATION AREA The document refers to the EMI software integration testbed implementation that is all resources and coordination/effort needed to setup an infrastructure on which software products integration tests can be performed. Since the test infrastructure implementation, update and management strictly depends on the way software products certification and release process are defined, SA2.6 expect a continuous collaboration and interaction with JRA1and SA1 activities REFERENCES R1 R2 R3 EMI DoW SA2.6 Survey about testing procedures and testbed status across middleware projects ARC project testing references INFSO-RI Members of EMI collaboration PUBLIC 7 / 40

8 R4 R5 R6 R7 R8 R9 R10 R11 R12 R13 R14 R15 R16 R17 R18 R19 R20 R21 R22 Atlassian bamboo for UNICORE project Yaimgen utility webpage at CERN ETICS project homepage EGEE project pre-production service homepage CERN VNode utility NAGIOS Alarming system project homepage EGEE project Operations Automation Team homepage CERNS s Savannah bug/task tracking tool instance EMI Product Teams list EGEE project pre-production service: Pilot Service Testing procedure Jira issue and project tracking tool Twiki automated notification tool EMI Certification and Testing Guidelines ARC Project glite Project UNICORE Community Web page dcache Project Global Grid User Support European Grid Initiative INFSO-RI Members of EMI collaboration PUBLIC 8 / 40

9 R23 Release Process Guidelines DOCUMENT AMENDMENT PROCEDURE Amendments, comments and suggestions should be sent to or to the author of this document. This document can be amended by the Quality Assurance team (SA2) further to any feedback from the other teams. Minor changes, such as spelling corrections, content formatting or minor text reorganisation not affecting the content and meaning of the document can be applied by SA2 without previous review. Other changes must be peer reviewed and submitted to the PEB for approval. When the document is modified for any reason, its version number shall be incremented accordingly. The document version number shall follow the standard EMI conventions for document versioning. All versions of the document shall be maintained using the document management tools selected by the EMI project TERMINOLOGY Work Package (WP) Product Team (PT) Software Product Validation and Testing Integration Testing The EMI project is composed of two Networking Work Packages (NA1 and NA2), two Service Work Packages and one Joint Research Work Packages (JRA1). Product Teams are teams of software developers and testers fully responsible for the successful release of a particular software product (or group of tightly coupled related products) compliant with agreed sets of technical specification and acceptance criteria. The Process responsible for Validation and Testing of a new or Changed IT Service. Service Validation and Testing ensures that the IT Service matches its Design Specification and will meet the needs of the Business. Testing activity performed by PT involves different classes of tests: Unit tests: test the correctness of individual units or a group of related units. Deployment tests: verify that the component can be properly installed and configured on all the supported platforms Functionality tests: cover the majority of the tests done on a component to verify it's correct (according to specifications) behavior. Regression tests: tests that are meant to verify specific bug fixes. Performance tests: tests that aim at verifying the performance of a component, which in many cases involves the measurement of the response time for specific service requests. Performance tests should verify how well the service behaves with nominal workloads. Scalability tests meant to verify that the component behaves according to its specifications when varying one of the variables that can affect its performance. Load and stress tests are included. Integration refers to a setup where one software component inherently uses INFSO-RI Members of EMI collaboration PUBLIC 9 / 40

10 Certification of Software Product another component to provide a dedicated functionality. The way how a component is integrated into another varies including the use of communication protocols (i.e. Web services via SOAP) or simply the usage of using libraries. From definition above, integration tests are functionality tests involving the interaction among different components, generally developed by different Product Teams and which may be deployed on different hardware instances or as modules on the same instance. The action done by a Product Team on a release candidate in order to "certify" that the software has been verified (the software has been correctly built) and validated (the right software has been built). Verification and validation involve testing. INFSO-RI Members of EMI collaboration PUBLIC 10 / 40

11 2. EXECUTIVE SUMMARY The present document describes the EMI continuous integration and certification testbed implementation model. To define an effective strategy of implementation SA2.6 tried to integrate into a homogeneous framework, inputs, ideas and requirements from the main middleware projects, taking into account: i) the EMI Description of Work targets for the task; ii) integration testbed customer requirements (collected through meetings and surveys with developer representatives) and iii) task participant resources, established procedures and experiences (collected through meetings and surveys and grouped per middleware: ARC, glite and UNICORE). Therefore in section 3 SA2.6 identifies all needed requirements to setup a suitable EMI integration testbeds for a middle-term perspective. In particular: the envisaged testbed scenarios (testing for minor releases, major releases, for crossmiddleware integration and for large scale); the needed infrastructural supplies (accessible hardware resources, connectivity, certificates handling etc.); the inter-project communication and effort coordination needs. Taking into account both requirements identified in section 3 and the three middleware (ARC, glite, UNICORE) pre-emi approaches to integration testing and testbed implementation, in section 4 SA2.6 define the EMI internal integration testing testbed implementation plan including: what SA2.6 actually provide to implement the testbed scenarios previously identified; the public documentation solution implemented (EMI Testbed web page), coordination procedures (for testbed installation, modification, configuration or generic support requests), communication channels and activity tracking tools (mailing lists defined and Savannah tools groups and squads defined). Also a preliminary model for large a scale testing testbed is proposed, even though that will be the object of the second quarter of the project. Even though the initial implementation of testbeds will treat the four middleware distribution separately, effort was spent into the harmonization of testbed access procedures, documentation and resources monitoring. Open Issues to the attention of readers. Even though, at the present time SA2.6 has a limited knowledge of all possible EMI testbed use cases, section 5 was dedicated to the identification of main issues expected to have a strong impact on testbed implementation and which need inputs or discussions from/with other EMI work packages. In particular: cross-middleware resource discovery and publication; hardware dimensioning relevant variables (releases, platforms, etc); common authentication/authorization framework for testbed accessibility; automated testing evolutions (ex. post build testing and on-demand testbed infrastructure creation). A section is then devoted to the testbed model evolution and revision, envisaging three main phases: the current phase of early convergence among middleware distributions; a second phase approaching the first EMI release, when the focus will move towards cross-middleware integration and a third phase in which external partners and integration testing for new user communities or Linux distributions will become more and more important. A final section reports about current testbed implementation status and a how-to quick reference table about EMI Testbed operational aspects and implemented solutions. INFSO-RI Members of EMI collaboration PUBLIC 11 / 40

12 3. EMI INTEGRATION TESTBED REQUIREMENTS IDENTIFICATION The EMI SA2.6 task is in charge of providing testbeds for integration testing within the EMI project. Purpose of this section is to identify what are the requirements and possible scenarios for an integration testing testbed in a project merging different grid computing middleware distributions, like EMI. First step is to define a common dictionary concerning integration testing and testbed: Integration testing: In the present document SA2.6 calls integration testing the phase and related activities in which individual software modules are combined and tested as a group. So SA2.6 expects this phase to come after an earlier certification phase in which software module building, installation, configuration and functionality testing in insulation has been carried out on Product Teams hardware resources. Inputs on the definition of integration testing procedures are expected from EMI task JRA1.7 Integration and Interoperability and will be integrated in the document when made available (see section 5 for details). Integration testing testbed: a platform for testing EMI software products in a working environment. This definition implies a set of hardware resources deploying a variety of operative systems and software products with proper configuration to allow services intercommunication and grant concurrent access to a realistic production environment set of testing users. Software product, i.e. the unit object of integration testing, interacting with each other software in a production like environment. Testing operations on a software product require its installation and configuration on dedicated hardware resource on which operative systems included in the predefined set of supported platforms are deployed. A software product is typically available in several versions associated to a middleware release. A release can be of two main categories: minor and major, depending on the possibility of breaking backward compatibility (allowed only in major releases). For each middleware release SA2.6 assumes to have two available versions per software product: a Release Candidate (RC) version and a Production version, respectively the version which is going to be tested and the last version passing the acceptance certification process for that release. From an integration point of view a software product is then defined by a version tag and the associated release and platform. Other implementation differences not affecting the considered product Application Programming Interfaces (API) are not taken into account for testbeds. For example a given service version changing only at underlying database implementation level (ex. Mysql or Oracle), with no changes at the level implementing integration with other services, can be considered a unique software product from an integration testing point of view, implying that just one version will be effectively made available in the testbed. The rest of the section is organized as follows: a first subsection reporting the SA2.6 task definition as stated in EMI DoW [R1]; then a subsection reporting the result of a survey conducted among task participants to collect information about current testing procedures and testbed resources and effort organization for the various middleware converging into EMI; then a final subsection translating task goals into a list of requirements, infrastructural resources and procedures to be made available. INFSO-RI Members of EMI collaboration PUBLIC 12 / 40

13 3.1. SA2.6 TASK DEFINITION The EMI Description of Work states [R1]: Task SA2.6: This task consists in the setup and maintenance of distributed testbeds for the project continuous integration and testing operations and the coordination and provision of large-scale testbeds from collaborating resource providers. Success Criteria: Availability and reliability metrics of the execution nodes Partners: INFN, CERN, CESNET, JUELICH, UPJS Key Performance Indicators for SA2 Testbeds is shown below CODE KPI Description Method to Measure Estimated Targets KSA2.1 SA2 Services Reliability % uptime dependent only on the SA2 services themselves (individual KPIs for test beds, repository, etc) Participating sites monitoring tools 99.00% KSA2.2 SA2 Services Availability Total % uptime including the underlying suppliers (individual KPIs for test beds, repository, etc) Participating sites monitoring tools 97.00% KSA2.3 Distributed Testbed Size Number of CPUs available for distributed testing through collaborations with external providers (NGIs, sites, commercial providers, other projects, etc) Participating sites monitoring tools Year 1: 50 CPUs Year 2: 200 CPUs Year 3: 500 CPUs 3.2. SURVEY OF TESTING RESOURCES AND PROCEDURES CONVERGING INTO EMI Among initial activities in SA2.6 task SA2.6 conducted a survey among task participants, grouped into different middleware representatives (ARC, glite, UNICORE), in order to collect information about current testing procedures or practices and available integration testbeds. Comparative results of the survey were made available here [R2]. The convergence between middleware on testing procedures is beyond the scope of SA2.6 task. Therefore SA2.6 reports just issues which are relevant for our purposes. Testing procedures: Testing procedures and practices: a very heterogeneous scenario was observed in the three middleware distributions converging into EMI project: ARC, glite and UNICORE. In the survey results document [R2], links to public documentation on currently used procedure can be found. Although the definition of a common testing procedure will be addressed in other EMI Work Packages, from a testbed point of view, a common concern is on the need for a INFSO-RI Members of EMI collaboration PUBLIC 13 / 40

14 clear definition of Release Candidate version of a given software product for each release (minor or major) and when integration testing starts in the certification process. Automated Tools: general requests of keeping and improving currently achieved results regarding definition of certification process automated testing tools. ARC seems to have a well defined framework and an automated testing tool [R3]. For glite the existence of tools may depend on the products (DPM, BDII use yaimgen [R5], VOMS has automated tests in ETICS [R6]). UNICORE uses Atlassian Bamboo for continuous integration [R4]. This topic strongly overlaps with EMI SA2.4 and JRA1 activities, so for what concerns the automatic creation of testing resources needed automatic testing, future updates of this document are foreseen. A typical scenario for integration testing involves a Release Candidate software product version versus other software product released versions. Also rare scenarios involving Release Candidate software product version versus other software product Release Candidate versions can occur (typically when testing for an upcoming major release, when backward compatibility can be broken for several products at the same time). Apart from some structured approaches in glite previous experience (EGEE III European project, Pre-Production activity, Pilots [R7]), a defined framework is missing for users community involvement in the test of upcoming versions of software products, where the setup of a dedicated testbed for stress-test or large scale test is required. In EGI project [R22] a staged-rollout procedure is defined with pre-deployment in production of released software products to a set of early adapters volunteer sites, but it is more a final acceptance YES/NO test than a collaborative testing phase. Testing effort is an issue. All middleware projects (ARC, glite, UNICORE) used to have a tester team carrying on testing activity. Now much will be left to PT efforts, with possible problems to keep current standard of testing. Moving to a distributed infrastructure able to support tests from the various middlewares components will imply additional effort for concurrent (from access to resources point of view) test coordination and testbed maintenance. Testbeds implementation: Number of software components to deploy, not considering cases of service co-hosting: ARC: 11 products glite: 19 products in Release 3.1 (SL4); 17 products in Release 3.2 (SL5) (without considering cases like mentioned glitevoms Mysql/oracle database implementation) UNICORE: 11 products Platforms: ARC (Fedora, Debian, Red Hat, Ubuntu, Windows and Mac OSX), glite (SL4, SL5), UNICORE (Linux, Windows, Mac OSX) Virtual Servers are in general an applicable solution to deploy services but physical servers are required for some services, as well as for performance / stress testing purposes. INFSO-RI Members of EMI collaboration PUBLIC 14 / 40

15 Host certificates are required in ARC and glite. Containers certificate in UNICORE not related to host. Special CA for testing available (ARC, CERN) or for dummy certificate emission for testing purposes, otherwise manual certificates. Currently available hardware resources among the task partners: ARC: ~14 Virtual nodes + 9 physical servers glite: CERN (~15 Virtual + 17 physical), INFN (~25 physical + 10 Virtual), CESNET (~10). Still not clear whether it is covering actual needs (other sites were involved before). UNICORE: Julich (~10 Virtual). Covering actual needs. dcache: 6 dcache servers + glite BDII service User service usage: no policy currently available in any middleware neither for scheduling usage nor for monitoring it. Server Monitoring: available in ARC (Ganglia + GridMonitor and Nagios), glite (Nagios), and no tools within UNICORE. Automated tools for on demand virtual nodes creation: vnode [R8] portal available at CERN. These tools are potentially very interesting since, with the automatic creation and configuration of a pool of virtual servers deploying a set of service talking to each other, the integration testing stage could become a default post-build operation, implemented with batch scripts. There is no established procedure or channels of communication for recruiting users community partners interested in providing resources/effort for testing. Some past experience with glite Pilot services [R13] REQUIREMENTS IDENTIFICATION On the base of the basic terminology reported at the beginning of section 3, task goals are described in section 3.1 and the current testing procedures and testbed implementation across the three middleware (ARC, glite, UNICORE), in this section SA2.6 will translate task goals into procedures, activities, infrastructural and operational resources which SA2.6 must make available to have the EMI integration testbed in place. The collaboration with other WPs ensures that feedback from SA1 and JRA1 contribute to the solution described below Envisaged integration testing scenarios From the definition of software products reported in section 3. and after preliminary discussions concerning testbeds with EMI JRA1/SA1, matching SA2.6 task goals implies providing testbed resources for the following testing scenarios: 1. EMI Internal Testing scenario A: Integration testing within a minor release (no backward compatibility broken), so that a Release Candidate 1 Service Version (RCSV in the following) 1 Here SA2.6 assumes that for each service a single Release Candidate version per Release exists, identified by a patch number or an official build. INFSO-RI Members of EMI collaboration PUBLIC 15 / 40

16 can be tested VERSUS other Production Version Services (PVS in the following). This implies a distributed testbed of production services available for each middleware stack, with possible multiple instances for central services. This could also imply cases of RCSV vs. other RCSV or RCSV vs. (RCSV + PVS): imagine the case of two interacting products in which a common bug is fixed contemporaneously, that would imply RCSV for both of them to be tested together with all other services at production version. Key performance indicators KSA2.1, KSA2.2 will apply to this testbed. 2. EMI Internal Testing scenario B: Integration testing for a major release (where it is allowed to have new features or backward compatibility broken for many services). This implies a testbed of RCSV available for each middleware stack, so basically this means providing hardware with platform installed for Product Teams (PT) to install needed RCSV and allow them for previewing other's PT RCSV. Key performance indicators KSA2.1, KSA2.2 will apply to this testbed. 3. EMI Internal Testing scenario C: Initial cross middlewares (glite<->arc<->unicore) Integration testing. This implies a testbed of PVS for all middleware to be accessible, which will be normally accomplished by testbeds defined at points 1 and 2 above, but could also imply specific testbeds configuration setup especially in the initial phase of the middleware convergence process. An issue specific to this scenario is the resource discovery and information publishing across middlewares (see Section. Key performance indicators KSA2.1, KSA2.2 will apply to this testbed. 4. EMI External Testing scenario D: Large-scale tests, this will involve voluntary partners providing resources for the test. Key performance indicators KSA2.3 will apply to this testbed. Figure 1 below provides a simplified representation of first three testing scenarios above (third scenario can be treated as a normal major release where product X,Y and Z come from one of the three middlewares or dcache respectively. Figure 1: EMI Integration testing scenarios INFSO-RI Members of EMI collaboration PUBLIC 16 / 40

17 As mentioned before, the certification testing resources of a service in insulation before integration is in charge of PT so SA2.6 will focus on making available the testing infrastructure needed by each PT to test the interaction of its product with other PT's software products. The definition of certification process and testing guidelines PT should follow is in charge of other activities. See certification and testing guidelines [R16] for overview of certification process and release management guidelines [R23] for a reference on tests documentation, coordination and communication. From a testbed point of view SA2.6 assumes that testbed available means: Available HW resources: Server with OS installed (one among agreed platforms), network connection and utilities Server host certificate available Server level monitoring tool (e.g. Nagios, Ganglia etc.) to collect statistics about server availability/reliability. Access to the server for PT testers Effort definition, Procedures, Communication and Documentation needs Apart from the testbed infrastructure, SA2.6 has to define the coordination, communication and task tracking procedures needed to put in place and evolve the testbed, following the needs of the EMI projects. Effort: to put in place and maintain the EMI integration testbed, the SA2.6 expect activities on: Hardware setup, OS installation, maintenance Software Product installation, configuration and update Host certificates, users configuration, services inter-communication configuration Coordination of activities, testbed planning, resources rising for large-scale testbed Automation solution identification and setup Testbeds documentation: available server inventory and associated logbooks (twiki or similar) should be maintained and made publicly available: Inventory: a complete list of available instances grouped by testbed purpose (according to description in section 2) Logbooks: a service status page describing installed SW version, configuration (including users who may access it, and other service resources it is connected to). Possibly historic changes should be documented too. Contacts: a list of names and contacts of people responsible for service status maintenance Task tracking and Communication: INFSO-RI Members of EMI collaboration PUBLIC 17 / 40

18 Procedures and defined channels to request a testbed and to involve people in providing HW resource A task tracking system to assign tasks to people and track results (CERN's Savannah [R11] is a good candidate for this). In future a migration to Jira [R14] tracking tool will be evaluated in case that tool is widely adopted within EMI. For large-scale tests, testbed means production environment simulation (services interaction, distributed pool of users, production like usage of services: definition of use cases) Procedures and defined channels to request a testbed and to involve people outside EMI in providing HW resource (EGI, NGIs, etc) As for internal testbed: server monitoring tools, public description of related testbeds, people responsible, task tracking systems INFSO-RI Members of EMI collaboration PUBLIC 18 / 40

19 4. EMI TESTBED IMPLEMENTATION MODEL Moving from requirements identified in section 3, and in agreement with SA1 and JRA1 EMI work package, the implementation model described in the following paragraphs has been adopted by SA EMI TESTBED OR TESTBEDS? To avoid misleading terminology, it is useful clarifying the usage of testbed and testbeds terms in the document. EMI Integration Testbed indicates the whole set of geographically distributed infrastructural (hardware, networking, software) and operational (procedures, tools, documentation, communication tool, support handling etc.) resources made available for integration testing purposes. EMI Integration Testbed is then sub-classified into EMI Internal Integration Testbed and EMI largescale Testbed, to distinguish the set of infrastructural and operational resources provided by EMI partners participating to SA2.6 task from those resources provided by / involving partners external to EMI (e.g. European Grid Initiative project or the various National Grid Initiatives). At the same time, from the pool of EMI Integration Testbed resources, several testbeds can be built, depending on the particular subset aggregation needed for each given testing scenario. From this point of view the testbed for a particular test is ultimately defined by the particular resources view the user can have from the exploited user interface or client, depending on the information system configuration, the middleware under test, the user credentials and authentication/authorization configurations etc. To give a practical example, if a developer of a job management product needs to test job submission from a User Interface to all middleware compute elements, she/he will need to have a properly configured User Interface (api/client required version) from which a set of compute elements and data management services properly configured (published into an accessible information system collecting information from needed resources, authorization/authentication, etc.) can be accessed etc. This will in general require the availability of a set of resources with basic configuration, but also some additional customization effort for specific testing needs. The objective of the SA2.6 task is therefore to provide both an adequate set of infrastructural resources able to reproduce a production environment and a set of operational tools and solutions allowing for the flexible customization of subset testbeds suited for the peculiar integration test a Product Team may need to perform. In this perspective, the classification of testbeds by testing scenarios A, B, C, D in section or by middleware is useful to identify the number and types of services needed and the possible combinations and configurations to be adopted, but it does not imply their formal implementation as separated testbeds EMI INTERNAL INTEGRATION TESTBED SOLUTION The four testing scenarios described in section are expected to cover envisaged integration testing needs EMI software products. Notice that earlier certification testing 2 and product performance testing testbed provision is not considered in charge of SA2.6 task. 2 SA2.6 intends integration testing as the part of certification process when the product is tested versus other products. Before this testing phase certification tests of the product in insulation should be carried out, either as a post-build step or in a proper testing activity carried out by JRA1 or SA1 Product Team members. INFSO-RI Members of EMI collaboration PUBLIC 19 / 40

20 To implement internal integration testbed for scenarios A, B, C partners in the task will provide: Hardware Resources: Dedicated Hardware (physical or virtual), OS from supported platforms installed, grid service ports accessibility and root privileges access for software product installation and configuration from Product Team people. Software Products installation and configuration. As mentioned in previous section products will be differentiated on the base of: Release, Version (Production or Release Candidate), platform, middleware. Notice that full coverage of production versions of EMI products in the testbed is the main goal while the deployment of Release Candidate versions will be done on best effort depending on available HW resources. Information Systems: configuration in order to make the testbed grid resources visible to test users. Initially just intra-middleware information service configuration will be considered, and with this rather disjoint testbeds will be initially implemented for each middleware (ARC, glite, UNICORE). Therefore, cross-middleware resource discovery and information publishing is an issue to be addressed at middleware development and harmonization level (more in sec. 5). Notice that while we have a commitment to make available production version instances of products for each release, Release Candidate versions will be made available and maintained under specific requests. Concerning distribution of testbed service instances among partners, we adopt a Product Team / middleware proximity criterion in the choice of the SA2.6 task partner site deploying software products. That means that, whenever possible, each site contributing to testbed will firstly deploy software products from local product teams or from middleware for which the site has previous experience on. Other than optimizing the exploitation of partner experience on each specific product, this approach will favour communication with PT and result in fewer accesses to testbed servers from outer sites people Hardware Resource dimensioning Input from JRA1 is expected about the complete list of EMI products, platforms supported and foreseen releases. In this section SA2.6 will therefore identify the relevant variables for testbed hardware resources dimensioning: 1. Number of Software Products. Notice that two software products implementation implying no differences at API level will be considered equivalent (glitevoms with oracle or Mysql database) 2. Number of supported releases. Notice that two versions (Production version and Release Candidate version) for each supported release will be generally deployed. 3. Number of supported platform per middleware. Further details on the certification process definition are expected from JRA1.7 task and will be integrated into testbed model once available. INFSO-RI Members of EMI collaboration PUBLIC 20 / 40

21 Concerning current available resources, SA2.6 can rely on roughly 70 servers deploying services from ARC, glite, UNICORE middleware plus additional resources made available by dcache team for data management integration testing purposes. Concerning platforms SA2.6 has currently deployed a subset of the supported one per middleware: ARC (CENTOS 5.5, sl5, Debian Lenny, Ubuntu Hardy), glite (sl4, sl5), UNICORE (which is not dependent on any particular platform, generally deployed on opensuse 11.2), dcache (sl4, sl5). Apart from not deploying all supported platform permanently, the deployed services cover current integration testing needs Product Teams members effort contribution Testbed implementation implies a considerable effort for software product installation, configuration, debugging and update. Moreover installing or configuring Release Candidate product version generally requires expertise on the product. Therefore, in agreement with JRA1 and SA1 leaders SA2.6 expect collaboration from PT members involved in certification concerning: 1. Providing effort for service installation on HW provided by SA Keeping Service Status Logbook documenting all updates, configuration and information useful for the service usage from other PT 3. Providing effort for support and debugging on issues related to the service they are in charge of. The coordination of PT member intervention on server is in charge of SA2.6, which will ask intervention and coordinate different PT interventions on services which need to be put in connection. As stated in section below granting access to PT members for installation or configuration will be left to partners decision according to their local security policy Testbed monitoring solutions Key performance indicators KSA2.1 and KSA2.2 reported in section 3, involving availability and reliability metrics for testbed servers, imply automatic monitoring solutions for resources. Each middleware currently has a monitoring solution deployed: ARC (Ganglia, GridMonitor, Nagios), glite (Nagios), UNICORE (Nagios is going to be installed). Initially SA2.6 will provide availability and reliability statistics just on testbed server instances, not on services given the fact that not all services have availability/reliability metrics defined and tools to measure them. Concerning the adopted tool to monitor instances and produce statistics, all middlewares plan to converge on Nagios solution [R9] for two suitable features for our task purposes: Evolution of Nagios into grid monitoring Nagios [R10], which is expected to provide metrics for service monitoring Solution for geographical distribution: a second level Nagios can implement a central instance, republishing and aggregating data coming from local sites Nagios instances. Initially, availability and reliability statistics periodically produced by local sites Nagios instances will be made available in the testbed public documentation center for SA2.6 described in section INFSO-RI Members of EMI collaboration PUBLIC 21 / 40

22 4.2.4 Resources access policy Three types of access to testbed resources are expected: 1. Tester users access to perform tests. This type of access will require common grid services user configuration, which may include both personal account for PT testers on User Interface services and dummy user accounts to simulate concurrent usage stress test on software products. 2. Tester user access to collect information useful for tests debugging: access to logs, network configuration etc. This use case will not necessarily require root privileges to be granted to testers and could be implemented just making needed files available through https to users with specific certificates, or via gridftp whenever possible. The mentioned approaches would have the advantage of reducing the number of direct access to servers, which would involve matching the local sites (the sites hosting the testbed servers) security access policies for each participant site. 3. Root access to server for service installation and configuration. Since granting root access to remote users poses a security problem, it will be regulated following local security access policies for each participant site. Account requests will be regulated according to communication procedures described in section Testing Certificates handling Testing certificates handling is a key security issue even in a testing environment. As mentioned in section 3.2, participant sites and middleware have different solutions both for host and user certificates as reported in TAB below. TAB ARC GLite UNICORE Host Certificate Policy User Certificate Policy Required by all software products. Possibility to disable certificate checks in WS based services. Grid proxies and glite voms supported. Required by most software products. Grid proxies and glite voms supported. Container certificates not tied to any particular host, i.e. when accessing UNICORE, only the certificate by itself, not the host, is checked. Apart from some interoperability tests, proxies are not a supported feature. SAML VOMS supported, glitevoms not supported. Certificates Generation Instant CA Service for testing purpose temporary user and host certificates generation. Manual generation of official certificates with CA signature. Tools available for fake certificate generation. Manual generation of official certificates with CA INFSO-RI Members of EMI collaboration PUBLIC 22 / 40

23 signature. Notice that the interoperability testing activity among the three middleware instances in EMI integration testbed will rely on the convergence on a common framework for authentication, which is beyond the scope of SA2.6 task (see section 6) Usage Monitoring and Concurrent usage Requirements Neither usage monitoring tools nor resources usage scheduling tools are currently available. So SA2.6 is not able to implement policies regulating concurrent usage of testing resources. On the other hand, the typical use case for integration testing will not imply exclusive usage of resources. If in particular cases exclusive usage is required for testing purposes, an ad-hoc solution will be defined and put in place on request Testbed update and evolution procedure The testing scenarios in section will imply the installation of both production version and Release Candidate version software products for all supported or upcoming releases. After initial setup the following evolution scenarios are expected: 1. New Product versions Released: each time the certification process successfully ends into the release of new versions for each product, updating or configuration activities will be needed to keep testbed aligned to official releases. Just 1 production version and 1 Release Candidate version per product software per release will be deployed at a time. 1. Production versions: updates on testbed servers will be triggered by releases broadcasts or other chosen automated notification tool. Involved PT people will be contacted to support the update activity. Some optimization could derive from the fact that RC instances are expected to become production version instances, while old production version instances can be dismissed to host the new RC version. 2. Release Candidate versions: RC version availability will be notified by responsible PT to testbed administrators or automatically triggered by a Ready for Certification status flag Configuration changes requests: these types of requests may involve information system configuration to publish subsets of resources, grid services configuration for intercommunication, users or Virtual Organization configuration. 3. Special testbed requests: could be needed for peculiar testing requirements at some point in EMI project, for example for testing some security issues across all products or new features All these requests will be taken in charge by SA2.6 participants, who will request the support of PT members if needed, as described in next section. 3 This will depend on the certification process adopted within EMI project. INFSO-RI Members of EMI collaboration PUBLIC 23 / 40

24 4.2.6 Activity tracking, communication channels and procedures Both coordination and installation/configuration activities concerning testbeds require clear channels of communication and a way to track the effort of people involved. The solution adopted: Communication: SA2.6: an emi-sa26@eu-emi.eu was created both for task internal communication and for testbed requests reception. Activity Tracking: SA2.6 and PT activities on testbed will be tracked through Savannah [R11] tasks. An emi-sa2-testbed Savannah squad has been created to submit requests. Product Team squads have been created to track PT activities on testbeds, using contacts in [R12] Procedures: the following support requests cases (from PTs, SW Area Manager, SA1, JRA1) are expected and treated as described below: Requests for configuration support of existing services. These requests may include enabling VO/users, making services to talk to each other, custom bdii setup etc. Procedure: Send either a request by mail to SA2.6 mailing list explaining the user's testing and configuration needs or open a savannah task on the testbed squad. The request will be then evaluated and tracked into a savannah task on the testbed squad. If needed a PT members of services involved in the test will be contacted and his contribute is tracked by savannah tasks. Requests for new services setup (particular RC Service versions): Send a request by mail to SA2.6 mailing list or by savannah task on the testbed squad, explaining: Your testing needs and the type and version of services you need to be installed and the PT producing that service Please also specify if you need the service to be included in the permanent EMI testbed. The request will be then evaluated and tracked into a savannah task on the testbed squad. If needed, the involved PT members will be then contacted and his contribute tracked by savannah tasks. Requests for specific testbed (in this category: performance tests, security tests, data management tests, etc.): Send a request by mail to SA2.6 mailing list or by savannah task on the testbed squad, explaining: Your testing needs and an estimate of HW and SW requirements for your test INFSO-RI Members of EMI collaboration PUBLIC 24 / 40