D1.4v2 Reference Wf4Ever Implementation Phase II

Transcription

1 Wf4Ever: Advanced Workflow Preservation Technologies for Enhanced Science STREP FP7-ICT Objective ICT b) Advanced preservation scenarios D1.4v2 Reference Wf4Ever Implementation Phase II Deliverable Co-ordinator: Raul Palma Deliverable Co-ordinating Institution: Poznan Supercomputing and Networking Center (PSNC) This document presents the Wf4Ever Toolkit, the reference implementation of Wf4Ever architecture, which comprises a set of services providing functionalities exposed via RESTful APIs, alongside client applications providing access to these functionalities. Document Identifier: Wf4ever/2011/D1.4/v2 Date due: 31/07/2013 Class Deliverable: Wf4ever Submission date: 31/07/2013 Project start date: December 1, 2010 Version: V1.0 Project duration: 3 years State: Final Distribution: Public 2013 Copyright lies with the respective authors and their institutions.

2 Page 2 of 34 Wf4Ever STREP FP7-ICT Wf4Ever Consortium This document is a part of the Wf4Ever research project funded by the IST Programme of the Commission of the European Communities by the grant number FP7-ICT The following partners are involved in the project: Intelligent Software Components S.A. Edificio Testa Avda. del Partenón 16-18, 1º, 7ª Campo de las Naciones, Madrid Spain Contact person: Dr. Jose Manuel Gómez-Pérez address: jmgomez@isoco.com University of Manchester Department of Computer Science, University of Manchester, Oxford Road Manchester, M13 9PL United Kingdom Contact person: Professor Carole Goble address: carole.goble@manchester.ac.uk Universidad Politécnica de Madrid Departamento de Inteligencia Artificial Facultad de Informática, UPM Boadilla del Monte, Madrid Spain Contact person: Dr. Oscar Corcho address: ocorcho@fi.upm.es University of Oxford Department of Zoology University of Oxford South Parks Road, Oxford OX1 3PS United Kingdom Contact person: Dr. Jun Zhao / Professor David De Roure address: {jun.zhao@zoo.ox.ac.uk, david.deroure@oerc.ox.ac.uk} Poznań Supercomputing and Networking Center Network Services Department Poznań Supercomputing and Networking Center Z. Noskowskiego 12/14, Poznan Poland Contact person: Dr. Raúl Palma de León address: rpalma@man.poznan.pl Instituto de Astrófísica de Andalucía Dpto. Astronomía Extragaláctica Instituto Astrofísica Andalucía Glorieta de la Astronomía s/n Granada, Spain Contact person: Dr. Lourdes Verdes-Montenegro address: lourdes@iaa.es Leiden University Medical Centre Department of Human Genetics Leiden University Medical Centre Albinusdreef 2, 2333 ZA Leiden The Netherlands Contact person: Dr. Marco Roos address: M.Roos1@uva.nl

3 D1.3v1: Wf4Ever Architecture Phase I Page 3 of 34 Work Package Participants The following partners have taken an active part in the work leading to the elaboration of this document, even if they might not have directly contributed to the writing of this document or its parts: PSNC OXF UNIMAN UPM ISOCO Change Log Version Date Amended by Changes Raul Palma Document Outline Raul Palma Introduction Raul Palma Conclusions Raul Palma Section Raul Palma Section 2 based on input of users Raul Palma Revision Raul Palma Send to QA Raul Palma Update based on QA Raul Palma Final Version 2013 Copyright lies with the respective authors and their institutions.

4 Page 4 of 34 Wf4Ever STREP FP7-ICT Executive Summary This document describes the Wf4Ever toolkit, the reference implementation of Wf4Ever architecture, which consists of a set of services providing functionalities for the realization of a scientific workflow preservation infrastructure, alongside client applications providing access to these functionalities. We present these components in the context of the functional categorization introduced by the Wf4Ever architecture (see D1.3v2), followed by a description of short use case scenarios, which motivates the developed components and illustrate how they can interact with each other to support users in the management and preservation of scientific workflows. Moreover, for each of these components this document provides an information card including: a brief description highlighting the key functionalities provided, the APIs implemented, the services used, the source code repository location, the deployment location of a live instance, and a reference to the corresponding M32 technical WP deliverable where the component implementation and usage is described in more detail. Hence, this document serves as an index for other deliverables. A complete documentation of the API implemented by the services is provided in the Architecture deliverable (D1.3v2), and the user guides for client applications are provided in the Sandbox deliverable (D1.2v3). Note that this is a self-contained document that supersedes D1.4v1. Therefore, some of the content from D1.4v1 has been reused and updated when necessary, in particular for Section 1, Section 3, and Section 5. The main novel contributions of this document consist of the use case scenarios (Section), which have been aligned with the user deliverables with the input from our users, and the updated information cards (Section).

5 D1.3v1: Wf4Ever Architecture Phase I Page 5 of 34 Table of contents Wf4Ever Consortium... 2 Executive Summary... 4 Table of contents... 5 List of Figures Introduction Relations with other deliverables Use case scenarios Scenario: Discover Research Objects Scenario: Inspect Research Objects Scenario: Research Object creation and management Scenario: Annotations Scenario: Quality Scenario: Workflow run evidence Scenario: Recommendations Scenario: Evolution & Preservation Scenario: Manage Research Object using different tools Toolkit components Storage and Lifecycle services Research Object Digital Library RODL RO-enabled myexperiment (repository) Workflow Runner Data Management & Analysis Services Checklist Evaluation Service Quality Evaluation Service Recommender Service Workflow Abstraction Service Workflow-Research Object Transformation (WF-RO) Service Access & Usage Clients RO-enabled myexperiment (web interface) RO Portal RO Manager Tool Collaboration Spheres Conclusions Copyright lies with the respective authors and their institutions.

6 Page 6 of 34 Wf4Ever STREP FP7-ICT References... 34

7 D1.3v1: Wf4Ever Architecture Phase I Page 7 of 34 List of Figures Figure 1. Wf4Ever Toolkit... 8 Figure 2 Faceted search in RO Portal Figure 3 RO inspection in RO-enabled myexperiment Figure 4 RO inspection in RO Portal Figure 5 Importing data from myexperiment Figure 6 Managing Annotations in RO Portal Figure 7 Checklist result page Figure 8 Research object quality visualization page Figure 9 Recommendations in RO-enabled myexperiment Figure 10 Research Object evolution visualization in RO Portal Figure 11 Display annotations of a local resource with RO Manager Figure 12 Deployment diagram with partial view of interactions between Wf4Ever components, described in more detail in the subsection associated to arrow colour Copyright lies with the respective authors and their institutions.

8 Page 8 of 34 Wf4Ever STREP FP7-ICT Introduction One of the main tangible outcomes of Wf4Ever project is a technological infrastructure for the preservation and efficient retrieval and reuse of scientific workflows in a range of disciplines. In order to produce this outcome, we defined a software architecture for a scientific workflow preservation infrastructure, building upon the Research Object model developed in the project [1], and realized this architecture in a reference implementation called the Wf4Ever Toolkit. The architecture describes interfaces to functionalities, which have been grouped into categories representing areas addressed by a preservation infrastructure (see deliverable 1.3v2 [2]), and the toolkit implements these functionalities through a set of services and clients. For this process, we adopted an agile development approach, with early prototyping, where the toolkit coevolved alongside the architecture definition. The Wf4Ever Toolkit comprises a set of services implementing storage, lifecycle, data management and analysis functionalities, where these functionalities are exposed via RESTful APIs, alongside client applications providing access and usage functionalities. Figure 1 depicts the toolkit components in the context of the functional categorization prescribed by Wf4Ever architecture. Note that although the set of components in the toolkit is final, the components are still evolving, improving or complementing their implementation, and will continue to evolve during the whole lifetime of the project. Figure 1. Wf4Ever Toolkit The toolkit components are deployed in the Wf4Ever Sandbox, which can be used through its live instance ( or it can be downloaded and deployed locally along the toolkit components ( The only exception is RO-enabled myexperiment that uses its own Sandbox ( For more information about Wf4Ever Sandbox, including user guides of the client applications, please refer to D1.2v2 [3] and D1.2v3 [4].

9 D1.3v1: Wf4Ever Architecture Phase I Page 9 of Relations with other deliverables This document is the final reference implementation document, and supersedes the previous version D1.4v1. This document includes a technical overview of the toolkit components, and serves as an index to other M32 deliverables from technical work packages (D2.2v2 [6], D3.2v2 [7] and D4.2v2 [8]), which provide detailed description of each of these components, including implementation aspects and interaction diagrams. A complete documentation of the implemented APIs is provided in the architecture deliverable (D1.3v2 [2]) and user guides for the client applications is provided in the Sandbox deliverables (D1.2v2 [3] and D1.2v3 [4]). The remainder of this document is organized as follows: In section 2, we describe a set of short use case scenarios, aligned with the user documents, which motivate and illustrate the toolkit component and how users can interact with them. Next in Section 3, we introduce the toolkit components, highlighting the changes (since D1.4v1), and then for each of them, this section presents an information card including: a brief description, the API implemented (for services), the interrelations with other components, the location of its source code and live instance, and a link to the corresponding M32 deliverable where this component is described in more detail, as discussed above. Finally, we conclude in Section Copyright lies with the respective authors and their institutions.

10 Page 10 of 34 Wf4Ever STREP FP7-ICT Use case scenarios To illustrate the usage of the Wf4Ever toolkit, and the interactions between their components, we describe the ten most representative short use case scenarios selected in collaboration with our users. Although some of these scenarios may depend on each other, in general they can be performed in any order. Note that user guides for the client applications are presented in D1.2v2 [3] and D1.2v3 [4]. 2.1 Scenario: Discover Research Objects Users start searching for research objects stored in RO Digital Library from RO portal or RO-enabled myexperiment using some keyword(s). Results are displayed with useful information about each research object. For instance, such information may include fields providing a glimpse of what the research object is about (e.g., title, author and tags), its complexity (e.g., number of resources or annotations) and its usefulness or quality (e.g., status, date and number of comments). Results can then be restricted or filtered through a faceted search interface. For instance, in RO portal users can restrict the results based on the their author, status, creation date, number of annotations, number of resources or any combination of these (see Figure 2). Moreover, results can be sorted by any of these properties via a dropdown list (shown at the top of the results list in the figure) with the fields and their order (ascending/descending). Figure 2 Faceted search in RO Portal

11 D1.3v1: Wf4Ever Architecture Phase I Page 11 of Scenario: Inspect Research Objects Once a research object is selected in RO-enabled myexperiment or RO Portal, users can inspect its content (see Figure 3 and Figure 4). The information displayed includes an overview of the research object with its description, status, complexity, quality and a graphical representation (if available) that may help in understanding the research object. Additionally, users are able to navigate the structure of the research objects, inspect its aggregated resources including their annotations, and view the relations among them. In RO portal, users are also able to visualize the evolution of the research object, including events affecting the research object, such as changes in its content and in its quality. Similarly, users are able to download the whole research object or any of its aggregated resources, for instance, for reusing it or running it locally (e.g., a workflow). Figure 3 RO inspection in RO-enabled myexperiment 2.3 Scenario: Research Object creation and management Users are able to create a new research object in different ways. For instance, both in RO-enabled myexperiment and RO Portal, they can start by creating an empty research object, optionally with a predefined folder structure, and adding individual resources one by one (e.g., workflows, scripts, example input data and others). Alternatively, in RO Portal users are able to create a research object from a ZIP file with a set of folders and files, or by importing data from myexperiment. For this task, users select content from myexperiment such as workflows and packs (see Figure 5), which are transformed into research objects. The content itself is not modified and the available metadata, such as title and description, are copied as semantic annotations. For workflows, the Workflow Transformation Service is used, which 2013 Copyright lies with the respective authors and their institutions.

12 Page 12 of 34 Wf4Ever STREP FP7-ICT generates additional annotations related to the workflow structure, sub-resources, and history. In any case, while working with the research object users can establish the relationships between the aggregated resources (e.g., the example input data to the workflow). Figure 4 RO inspection in RO Portal Similarly, at any time users can modify the structure of a research object as well as its content. For instance, they can edit the folder structure of the research object adding folders and files, moving the files among folders, or deleting resources. They can also modify the research object properties (e.g., title, description) or its aggregated resources, e.g., uploading a new version or changing their description.

13 D1.3v1: Wf4Ever Architecture Phase I Page 13 of 34 Figure 5 Importing data from myexperiment 2.4 Scenario: Annotations The user can create, edit and remove annotations on the research object and its aggregated resources from RO Portal or RO-enabled myexperiment. Many of these annotations serve to associate components with resource type definition (hypothesis, workflow, script, input, result, document, conclusion, etc.), and also to describe relationships between them (such as a resource being an input for a workflow), based on predefined vocabularies and models compliant with Wf4Ever RO model. Other annotations are related to authoring and crediting, including keeping the authors and contributors, as well as relating a research object or its resources to other existing resources that they reused or derive from, and when possible automatically crediting the authors of these resources. It is also important to provide the possibility to include free-text descriptions for describing, for instance, the configuration settings needed for the execution environment at the resource level or the order of every step performed in the procedural protocol of the whole execution of the research object. Very often, users undertake these steps taking into account personal decisions that should be registered. Others may be related with scientific information very specific to the experiment. Scientific considerations on the provenance of the initial input data for the RO (e.g., how did I get or prepare them?), mathematical formulae or equations used in the model/process/script to execute, etc. These annotations may also be complemented with bibliography used or produced by the experiment. RO Portal provides a basic view of the resource annotations for predefined annotations and an advanced view for managing any annotation (see Figure 6) Copyright lies with the respective authors and their institutions.

14 Page 14 of 34 Wf4Ever STREP FP7-ICT Figure 6 Managing Annotations in RO Portal 2.5 Scenario: Quality Users can check the quality of a research object by running the Checklist Evaluation Service, providing as input a research object stored in RO Digital Library and a minim model [5] that formally describes the requirements to be satisfied by the research object to be suitable for some purpose (e.g., completeness). This service can be run from RO-enabled myexperiment (Figure 7 shows the result page returned after selecting the Checklist link on the right panel in Figure 3) or RO Portal, which automatically displays a short summary of the results in the research object Overview page (see Figure 4) and more detailed information in the research object Quality page. The Checklist Evaluation Service has been developed using some of the same code base as RO Manager, and the same evaluation can be performed from the command line using the RO Manager (see Sec 2.9 Scenario: Manage local Research Object).

15 D1.3v1: Wf4Ever Architecture Phase I Page 15 of 34 Figure 7 Checklist result page Additionally, users can check the normalized completeness, stability and reliability of the research object using the Quality Evaluation Service. This service executes a series of checklist evaluations (over time) on research objects stored in RO Digital Library and calculates these quality dimensions values, i.e., the extent to which the research object satisfies the checklist requirements, its ability to preserve its properties through time and its ability to converge towards a scenario free of decay (complete and stable through time) - see D4.2v2 [8]). The service can return a web page where the user can explore how the research object quality has evolved over time in terms of these quality scores. This page can be opened from RO-enabled myexperiment (Analytics and Quality link) or RO Portal (Quality page) see Figure Copyright lies with the respective authors and their institutions.

16 Page 16 of 34 Wf4Ever STREP FP7-ICT Figure 8 Research object quality visualization page 2.6 Scenario: Workflow run evidence When a research object aggregates a workflow that has not provided evidence that can be run, the research object quality checklist that can be shown from RO Portal or RO-enabled myexperiment warns the user, for instance, with a cross indicator on the run evidence requirement. The ongoing work includes enabling users to select this indicator from the portal, and give them the possibility to either upload run data or to run the workflow. In the first case, the user would provide a workflow run including, for instance, real (example) input data, real (example) output data, and information about the workflow engine used (e.g.,taverna version) and how long it took to run. In the second case, the user would execute the workflow from the portal using the Workflow Runner Service. The run provenance would be subsequently exposed by the service and then aggregated by the research object. As a result, the research object quality checklist would improve. 2.7 Scenario: Recommendations Users can get recommendations of other users, research objects and individual resources (e.g., workflows, datasets) using the Recommender Service, based on their profile (e.g., research objects they have created,

17 D1.3v1: Wf4Ever Architecture Phase I Page 17 of 34 keywords previously proposed), or on a research object description (e.g., title, description, tags). This service can be executed from RO-enabled myexperiment and RO Portal (Figure 9 shows an exemplary result page of recommendations in RO-enabled myexperiment). The service is also used by the Collaboration Spheres client application, available through RO-enabled myexperiment, which provides an alternative view for browsing recommendations. Figure 9 Recommendations in RO-enabled myexperiment 2.8 Scenario: Evolution & Preservation Users may at any moment create a snapshot of a research object, capturing the state of the research object at the given point in time, and create a textual annotation describing why they decided to do it and what represent the state of the research object. Such snapshots may be useful to release the current version of the research outcome of an experiment, submit it to be peer reviewed or to be published, share it with supervisors or collaborators, or for acknowledgement and citation purposes Copyright lies with the respective authors and their institutions.

18 Page 18 of 34 Wf4Ever STREP FP7-ICT Users can create snapshots from RO Portal, where they also can consult the evolution history of the research object as depicted in Figure 10. The user may decide to recover one snapshot of the research object, with all its contents and annotations as they were at that stage. Figure 10 Research Object evolution visualization in RO Portal 2.9 Scenario: Manage local Research Object Using the RO Manager command line tool a user is able to create and manage a research object on a local drive, aggregate his files as resources and make annotations. Figure 11 shows how to display the annotations of a resource with RO Manager. The research object may also be synchronized and pushed to the RO Digital Library for sharing and publication. Figure 11 Display annotations of a local resource with RO Manager

19 D1.3v1: Wf4Ever Architecture Phase I Page 19 of Scenario: Manage Research Object using different tools Users are able to use different tools for managing a research object. For instance, the user may create a research object from RO-enabled myexperiment, including aggregating resources and annotations, and then use RO Portal for managing the evolution of the research object, e.g., create snapshots and visualize the history. These applications use RO Digital Library for storing the research objects. Moreover, the user may download the research object, use RO Manager for making additional annotations or aggregating additional resources, and then push it back to RO Digital Library Copyright lies with the respective authors and their institutions.

20 Page 20 of 34 Wf4Ever STREP FP7-ICT Toolkit components The toolkit consists of 12 components: 3 storage and lifecycle services (Research Object Digital Library - RODL, myexperiment and Workflow Runner), 5 data management and analysis services (Checklist Evaluation, Quality Evaluation, Recommender, Workflow Abstraction and Workflow Transformation) and 4 access and usage clients (RO-enabled myexperiment, RO Portal, RO Manager Tool and Collaboration Spheres). Most of these components were presented in D1.4v1. However, as noted in the introduction, even though the set of toolkit components is final, they are still evolving, improving or complementing their implementation. Hence, in this section we summarize their current state and introduce the Workflow Abstraction service and the Quality Evaluation service. The latter subsumes (and replaces) the former Stability Evaluation service. Additionally, since RO-enabled myexperiment has been recently extended to implement the Research Object RO API 1 (partially), it has been included as one of the storage services (replacing current myexperiment). We have also removed in this document the ROBox client application as it was deprecated and is no longer maintained. Diagram in Figure 12 depicts the toolkit components along with a partial view of their interactions, described in more detail in the subsection associated to the arrow colour. The toolkit components have been deployed in Wf4Ever Sandbox, available online through its live instance ( and for download ( see D1.3v2 [3] and D1.3v3 [4]. The only exception is RO-enabled myexperiment that uses its own Sandbox ( and Individual deployment instructions may be also available at the source code repository of each component (included in their information card). In the following, we provide an information card for each component including: a brief description, the API implemented (for services), the interactions with other components, the location of source code and live instance, and a link to the corresponding M32 technical WP deliverable where the component implementation and usage is described in more detail. To illustrate the interactions with other components, we include small diagrams showing the related components with arrows depicting the flow of data, in the context of the functional categorization prescribed by Wf4Ever Architecture, providing partial views of Figure 1. A detailed documentation of the APIs is available in the final architecture deliverable D1.3v2 [2], and also in the following links: Research Object RO API: Research Object Evolution RO EVO API: Checklist Evaluation API: Reliability Evaluation API 2 : 1 Research Object RO API was formerly known as Research Object Storage & Retrieval ROSR API 2 The Reliability Evaluation API subsumes (and replaces) the former Stability Evaluation API

21 D1.3v1: Wf4Ever Architecture Phase I Page 21 of 34 Figure 12 Deployment diagram with partial view of interactions between Wf4Ever components, described in more detail in the subsection associated to arrow colour Recommendation API: Workflow Abstraction API: Workflow Runner API: Workflow-RO Transformation API: RO+transformation+service+API User Management API: Copyright lies with the respective authors and their institutions.

22 Page 22 of 34 Wf4Ever STREP FP7-ICT Storage and Lifecycle services Research Object Digital Library RODL Service Name RO Digital Library - RODL Description Software system which collects, manages and preserves aggregations of scientific methods (e.g., workflows) and related artifacts along with their annotations, organized as research objects. The key features are: Create, Edit & Delete ROs Retrieve ROs in different formats Creation of different RO types (Live, Snapshot, Archived) Retrieve RO evolution provenance Add & Remove aggregated resources and annotations Create & Delete users Search & Index ROs Query metadata Retrieve metadata in different formats (e.g., RDF/XML, TTL) RODL has a modular structure that comprises access components (built on top of dlibra digital library services - and Jena TDB store - long-term preservation components (built on top of darceo - and a controller that manages the flow of data. URI Source Code Repository APIs Implemented Research Object - RO Research Object Evolution RO EVO Notification User Management SPARQL Endpoint 3 Additionally, RODL exposes Solr schema REST API 4 Services Used RODL is built on top of dlibra services, Jena TDB store and darceo system. It is a core service of Wf4Ever and therefore it does not uses or rely on other Wf4Ever services. Additional Information D2.2v2 [6]

23 D1.3v1: Wf4Ever Architecture Phase I Page 23 of RO-enabled myexperiment (repository) Service Name Description URI Source Code Repository APIs Implemented RO-enabled myexperiment (repository) Repository of scientific workflows, other digital objects and research objects (called packs). The key features in the context of Wf4Ever include: Create, Edit & Delete packs/ros Add & Remove Workflows and other resources Search & Index Workflows, packs and other resources Versioning of resources, such as workflows Sorting and pagination Permissions management and myexperiment 5 Research Object RO (partially) SPARQL Endpoint Services Used RO-enabled myexperiment sends a copy of the research objects (packs) to RO Digital Library Additional Information D2.2v2 [6] Workflow Runner Service Name Workflow Runner Description Workflow execution service, key for workflow decay analysis. The key features are: Remote execution of workflows Expose workflow runs as ROs following a subset of RO API, aggregating inputs, outputs, console logs, provenance and annotations including mappings to RO vocabularies for workflow description (wfdesc) and execution provenance (wfprov) [1]. Workflow Runner is built on top of the Taverna Server Copyright lies with the respective authors and their institutions.

24 Page 24 of 34 Wf4Ever STREP FP7-ICT URI Source Code Repository APIs Implemented Workflow Runner Services Used Workflow runner does not use or rely on other Wf4Ever service; however, it can execute workflows stored in any remote repository, including RO-enabled myexperiment repository and RO Digital Library. Additional Information Data Management & Analysis Services Checklist Evaluation Service Service Name Checklist Evaluation Service Description URI Source Code Repository APIs Implemented Service for testing completeness, executability, repeatability and other desired features of a research object. In particular, it evaluates a research object against a minimum information model and for a particular purpose (e.g., completeness). Checklist Services Used The checklist service does not use or rely on other Wf4Ever service; however it can

25 D1.3v1: Wf4Ever Architecture Phase I Page 25 of 34 evaluate research objects available on the local file system or in any repository implementing RO API, such as RO Digital Library and RO-enabled myexperiment. Additional Information Notes D4.2v2 [8] The checklist service also comprises a traffic-light service component, which returns HTML or JSON data for facilitating the generation of a simple display of the checklist results. This service implements the traffic-light API 6, which is very similar in structure to the checklist evaluation API. Since the service provides HTML/JSON interfaces, a user guide is provided in D1.2v3 [4] Quality Evaluation Service Service Name Quality Evaluation Service Description This service test the quality of a RO, a measure of how healthy it is based on the ability of the research object to achieve its original purpose after being subject of changes on its resources. In particular, the service monitors the RO over time, capturing concrete values provided by the completeness, stability, and the reliability scores for the RO in different moments of its evolution. These score represent the extent to which the RO satisfies the checklist requirements, its ability to preserve its properties through time and its ability to converge towards a scenario free of decay (complete and stable through time). URI Source Code Repository APIs Implemented Reliability Notification Copyright lies with the respective authors and their institutions.

26 Page 26 of 34 Wf4Ever STREP FP7-ICT Services Used The quality service uses the checklist service to check the status of the research object at different moments in time, and consequently it can analyze research objects available on the local file system or in any repository implementing RO API, such as RO Digital Library and RO-enabled myexperiment. Additionally, this service generates notifications about changes in the research object quality that are collected by RO Digital Library and merged with other notifications about the research object. Additional Information Notes D4.2v2 [8] The quality service includes a decay-monitoring component, a web application where the user can explore the evolution in terms of quality scores of his Research Object over time. The decay-monitoring is available at and it has a parameter id which receives the RO_URI of the research object to be analyzed, e.g., Recommender Service Service Name Recommender Service Description Service that provides recommendations of users, Research Objects and their aggregated resources (e.g., scientific workflows and datasets), using three different algorithms: keyword Content-based, collaborative filtering and social networks. The recommendations are calculated for a particular user, sorted by the strength of the recommendation, and optionally filtered to include only particular type of resources, e.g., users, workflows, files and packs. URI Source Code Repository APIs Implemented Recommendation

27 D1.3v1: Wf4Ever Architecture Phase I Page 27 of 34 Services Used The recommender service uses former myexperiment ( as the source of information as it requires considerable amount of resources available to create recommendations, but it can also use any repository implementing RO API as source, such as RO Digital Library and RO-enabled myexperiment Additional Information D3.2v2 [7] Workflow Abstraction Service Service Name Description URI Source Code Repository APIs Implemented Workflow Abstraction Service This service enables scientist to search workflows by their functionality, properties, or other conceptualization allowing their easy accessibility. For this task, it provides functionalities for indexing, searching and recommending workflow processes. In particular, the two main operations are: Search: returns URIs of workflows that contain the sequence of executed processes that has been introduced as input parameter. Recommend: returns the most frequently used processes after the sequence of executed processes that has been introduced as input parameter. The output returns the id of the process, its probability of usage and its frequency. Search: Recommend: Workflow Abstraction 2013 Copyright lies with the respective authors and their institutions.

28 Page 28 of 34 Wf4Ever STREP FP7-ICT Services Used The workflow abstraction service uses a Provenance Corpus to create the indexation. The corpus is accessed through a SPARQL endpoint that exposes provenance using wfprov vocabulary [1]. At the moment of writing, the service is in the progress of connecting to SPARQL endpoints of Wf4Ever services exposing provenance information, e.g., RO Digital Library and RO-enabled myexperiment. Additional Information D2.2v2 [6] Workflow-Research Object Transformation (WF-RO) Service Service Name Workflow-Research Object Transformation (WF-RO) Service Description URI Source Code Repository APIs Implemented Service for converting workflows into research objects. The key features are: Transform workflows into Research Objects, creating new one or updating an existing one. Generates workflow description using wfdesc vocabulary [1] Generates workflow history using roevo vocabulary [7] Workflow-RO Transformation Services Used

29 D1.3v1: Wf4Ever Architecture Phase I Page 29 of 34 The WF-RO service takes as input a workflow (e.g., in former myexperiment, RO Digital Library or RO-enabled myexperiment) and stores the resulting research object in a repository implementing RO API, such as RO Digital Library or RO-enabled myexperiment. Additional Information Access & Usage Clients RO-enabled myexperiment (web interface) Client Name RO-enabled myexperiment (web interface) Description A collaborative environment where scientists can publish their workflows and in silico experiments (organized as research objects), share them with groups and find those of others. URI Source Code Repository and Interface Web User Interface (WUI), a type of GUI. Services Used RO-enabled myexperiment uses or is in the progress of using all Wf4Ever services, except for the lifecycle services of RO Digital Library. At the moment of writing, Workflow Abstraction and Workflow Runner are not being used yet. RO-enabled myexperiment stores research objects internally and in RO Digital Library. It also communicates with other clients, e.g., RO Portal and Collaboration Spheres. Additional Information D2.2v2 [6] 2013 Copyright lies with the respective authors and their institutions.

30 Page 30 of 34 Wf4Ever STREP FP7-ICT RO Portal Client Name Description URI Source Code Repository Interface RO Portal Web client application that enables access and use of research objects and aggregated resources. The key features are: Explore RODL Manage & visualize RO structure and annotations Search ROs Query RODL Import & Transform resources from myexperiment Visualize recommendations Web User Interface (WUI), a type of GUI. Services Used The RO portal uses most of the Wf4Ever services, except from the workflow-specific ones, i.e., Workflow Abstraction and Workflow Runner. The RO Portal has been developed alongside the RO Digital Library. Additional Information D2.2v2 [6] and D1.2v3 [4] (user guide) RO Manager Tool Client Name Description RO Manager Tool A command line tool that enables access and use of research objects and aggregated resources stored on the local file system, and the synchronization with remote repositories, e.g., RODL. The key features are: Create ROs in the local file system Add aggregated resources references in manifest

31 D1.3v1: Wf4Ever Architecture Phase I Page 31 of 34 URI Add annotations Read & Write ROs to RODL Source Code Repository Interface Command Line Interface (CLI) Services Used RO Manager Tool mainly uses resources on the local file system. However, it can also communicate with a repository implementing RO API (push, pull). The checklist evaluation service shares much its code base with RO Manager, and is packed with RO Manager. Additional Information D2.2v2 [6] and D1.2v2 [3] (user guide) Documentation: Collaboration Spheres Client Name Collaboration Spheres Description A graphical user interface intended to improve the sharing and reuse of ROs based on the exploitation of semantic descriptions, relations and similarities between ROs and users in order to support advanced search mechanisms. The search activity has a very strong social analysis aspect and is based on collaborative filtering and versatile keyword content-based recommendations. It implements a visual metaphor based on spheres, which uses concentric circles, where the similarity is represented using the distance from the center of the sphere to the place where the object is shown. The key features are: Creation of a four-layer representation for social and recommendation data, where circles and squares identify items (ROs and users respectively). The different layers are presented by using different colors. Provision of information related to the selected item from the four-layer representation (e. g. workflow representation) Copyright lies with the respective authors and their institutions.

32 Page 32 of 34 Wf4Ever STREP FP7-ICT Provision of a graph representation for the relations between users and ROs. Provision of ROs and users ordered by they importance for the current active item. URI Source Code Repository Interface Graphical User Interface (GUI) Services Used The collaboration spheres client uses the recommender service and consequently it uses former myexperiment as a source of information, but it can also use any repository implementing RO API, such as RO Digital Library and RO-enabled. Additional Information Notes D3.2v2 [7] and D1.2v2 [3] (user guide) Documentation: It uses myexperiment user id as a parameter, e.g.,

33 D1.3v1: Wf4Ever Architecture Phase I Page 33 of 34 4 Conclusions In this document we have presented the Wf4Ever Toolkit, starting with a motivation of the developed components. In particular, we have introduced the 12 components comprising the toolkit: 8 services and 4 client applications, including a brief description of their main features, the set of API implemented (for services), the interactions with other components, the location of source code and live instance, and the reference to the corresponding M32 deliverable where detailed information about their internal structure, interaction flow is described. Note that in some cases, there is a one to one mapping between APIs and services (e.g., checklist service); there are other cases where a service implements several APIs (e.g., Research Object Digital Library); and there could be cases where an API is implemented by several services (e.g., RO API). That shows the importance of defining open APIs that can be implemented by any service within or outside Wf4Ever. The Wf4Ever Toolkit was developed alongside the architecture using a co-evolutionary approach, working closely with the use case partners and drawing on the models defined by the technical Work Packages. We decided to follow an agile development approach with early prototyping, which allowed us to test our models and other outcomes as early as possible during the project lifetime, receive timely feedback from our users and identify and resolve any technological issues. For instance, users were able to interact and assess early implementations of Research Objects since our first prototype in month 6 of the project lifetime (see D1.2v1). The development of the toolkit during the second phase was concentrated in enhancing the support for the achievement of workflow preservation and conservation (e.g., Research Object Digital Library was extended with a preservation component as described D2.2v2), the realization of RO-enabled myexperiment (also described in D2.2v2), testing and improving components and providing missing functionalities (e.g., notifications). Note that although the set of components comprising the toolkit is final, they are still under development and may change during the remaining lifetime of the project in order to improve their implementation or add some additional features. For instance, the realization of RO-enabled myexperiment is still under intensive development, the RO portal is being redesigned and the RO digital library access control features are still under development. Hence, this document shows their state at month 32 of the project. This document is intended to serve as a guide of the toolkit with links to additional documentation rather than providing an exhaustive description of each component. In particular, it serves as an index to other M32 deliverables from technical workpackages (D2.2v2, D3.2v2 and D4.2v2), where each of the toolkit is described in detail Copyright lies with the respective authors and their institutions.

34 Page 34 of 34 Wf4Ever STREP FP7-ICT References [1] Wf4Ever Research Object Model. [2] D1.3v2 Wf4Ever Architecture Phase II. OXF. March 2013 [3] D1.2v2 Wf4Ever Sandbox Phase II. PSNC. July 2012 [4] D1.2v3 Wf4Ever Sandbox Phase III. PSNC. July 2013 [5] Minimum Information Model Vocabulary Specification. [6] D2.2v2 Design, implementation and deployment of workflow lifecycle management components Phase II. UNIMAN. July [7] D3.2v2 Design, implementation and deployment of Workflow Evolution, Sharing and Collaboration components Phase II. UPM. July [8] D4.2v2 Design, implementation and deployment of Workflow Integrity and Authenticity Maintenance components Phase II. ISOCO. July 2013.