Evaluation of Data Management and Transfer Tools for the biomedical Community

Transcription

1 Evaluation of Data Management and Transfer Tools for the biomedical Community Romannus Gruetz Department of Medical Informatics Maximilian Brodhun Department of Medical Informatics Benjamin Loehnhardt Department of Information Technology Frank Dickmann Department of Medical Informatics Abstract The biomedical community produces and uses a continuously growing amount of data while it lacks an interinstitutional research data management system. The LABIMI/F project founded by the German Research Foundation (DFG) implements an infrastructure to close this gap with an exemplary prototype for the use cases of medical image and genome research data. To determine suitable application(s) for this intention, several criteria for data management and data transfer, concerning their up-to-dateness, usability, metadata and payload management etc., are developed. These criteria are applied to three data management tools (DSpace, Fedora Commons and ISA-Tab tools), and four data transfer tools (PowerFolder, irods, CryptShare and Globus-Online), in an use-value analysis (UVA). The UVA reveals that no application meets all criteria; therefore other tools, e.g. escidoc, should be evaluated before making a final decision. Fedora Commons scores highly in the category metadata and payload management but has no sufficient user frontend. Therefore, further research is necessary in order to find an appropriate frontend for Fedora Commons. The data transfer tool with the highest total application score is PowerFolder, which provides easy synchronization of files and folders between the users and a central repository. Keywords-Archive; Data Management; Data Transfer; Grid; LABIMI/F; WissGrid I. INTRODUCTION In the biomedical community, the amount of data produced by the numerous analysis devices and procedures is growing continuously [1], [2]. To be able to handle that amount of data, a data management system becomes more and more necessary. The structured handling of data and their corresponding metadata can enable an effective retrieval and provides the basis for long-term digital data preservation. Due to data-intensive methods in distributed biomedical research projects, (meta-)data sometimes need to be acquired at multiple sites and then pooled in order to minimize the effort. To support the collaboration via internet-based infrastructures for the biomedical community, widely agreed standards as well as secure communication and data transfer are essential. Grid computing provides an infrastructure concept to overcome interaction-barriers between distributed heterogeneous computer systems [3]. The LABIMI/F project, founded by the German Research Foundation (DFG), aims to determine the capabilities of the reuse of research data and set up a digital preservation infrastructure prototype for two biomedical research data use cases: genome data and biomedical image data [4]. This infrastructure prototype will employ Grid computing technology and will cover content data (payload) as well as metadata management, in order to implement research data management (RDM). Furthermore, due to large genome (e.g. 700 GB per Binary Alignment/Map [BAM]) and image data (e.g. 3.2 GB per magnetic resonance imaging scan [DICOM]) the reliability of the data transfer is equally important. In addition, research data management will probably change the research workflows and could increase the workload for researchers. Therefore, it is of high importance to achieve a high level of acceptance by a) providing solutions, which are as intuitive as possible; b) a seamless integration into respective research workflows. Valid solutions should compensate for the additional workload. In this paper, three data management tools and four data transfer tools are evaluated to determine suitable solutions for the LABIMI/F project. The second paragraph describes the conducted use-value analysis (UVA) and its criteria. The third paragraph presents and the fourth discusses the results, followed by an outlook in paragraph five. II. MATERIALS AND METHODS The strategy to determine appropriate solutions, which meet the requirements of the LABIMI/F project, is to carry out a

2 structured market analysis: in conjunction with a use-value analysis (UVA) [5] advice of several domain experts was used. The consulted RDM domain experts are from the Alfred Wegener Institute for Polar and Marine Research (AWI) [6], the German competence network for digital preservation (nestor) [7], the Lower Saxony State and University Library (SUB) [8], and the WisNetGrid project [9]. The UVA is a method to compare two or more options in a multidimensional analysis. UVA is based on [5] and supports decision making in cases of criteria without monetary basis, but by scoring. The first step of UVA is to formulate relevant criteria categories for the given decision problem; each category includes sub-criteria [10]. All sub-criteria are mapped to a consistent grading scale. In general, higher levels of criterion match will result in a higher grade. For the evaluations of the data management tools as well as the data transfer tools, the applied mapping is shown in table II. To differentiate the relevance of the criteria, each criterion category is weighted in relation to all other criteria. From a sum of 100 weight points for all categories, the single category weights are derived. Within a category, the weights of all subcriteria add up to the category weight. The total score of an investigated application is the sum of the multiplied grades of sub-categories with their weight factors as shown in formula (1). For this paper, sub-criteria are described for each category and the scores given on a category level. grade weight total score (1) criteria The applied weight factors (see table III and IV) are developed in an internal iterative process by the project group. The involved members are RDM domain as well as grid computing experts. For the further understanding it has to be distinguished between payload, which is non-descriptive or primary data and descriptive data. Generally, descriptive data is known as metadata. That is, metadata is data that is used for describing other data. [11] Therefore the decision whether specific information has to be modeled as payload or metadata often depends on the applied use case(s). For the discussion of the UVA results the German university grading scale [12] is applied (see table I). TABLE I. GERMAN GRADING SCALE, DERIVED FROM [12] Percentage Scale TABLE II. MAPPING OF CRITERION MATCH AND CORR. GRADE. Grade Criterion Match 0 Not or insufficient implementation 1 Satisfactory implementation 2 Good implementation 3 Very good implementation A. Criteria for the data management tools The criteria of the evaluation of the data management tools are grouped in the following four top categories: up-todateness; data management; frontend; backend. 1) Up-to-dateness In the technical field rapid and dynamic changes take place. Furthermore, the up-to-dateness is an indicator of the development and user community activities. Thus, up-todateness is important and receives 5% of relevance = 5 weight points. 2) Metadata Management This category contains technical as well as organizational metadata. Technical metadata include information about the file / containers, like last modification date, location, type, owner, size. Organizational metadata may be e.g. of an administrative or a legal nature, and contain the context in which the data has been generated. Besides the metadata structure, the input of metadata is an important part of data management solutions. Automation via the extraction of metadata from existing data can accelerate the research data ingest and reduce the probability of human error or media break. To support the user in manual input, an application should be able to mark required metadata fields and to provide rule-based input checks. To search and find relevant data, an application should allow to establish and follow relations between different data sets. The relations could be 1:1, 1:n, or n:m. Grouping and building data groups of more than one item (usually a file) is important as well since content data sets often encompass more than one item. Another criterion in this category is the possibility to adapt the application to given requirements. Each area in biomedicine has its own data types and different sets of metadata. Thus, the application has to be customizable for diverse metadata sets. The optimal implementation should generate these metadata recommendations from a given ontology, this could improve ingest automation. This comprehensive category receives 35% of relevance = 35 weight points. 3) Frontend The frontend is the user interface (UI) and has to be accessible from the user workplace. Thus, the UI has to have low hard-/software requirements. If the required software components are not already installed, each of them often involves local installation at each individual workplace. For platform-independent access and central administration a webbased approach is favored. To measure and score the usability, the criteria of the Jakob Nielsen heuristic were applied: visibility of the system status; match between system and the real world; user control and freedom; consistency and standards; error prevention; recognition rather than recall; flexibility and efficiency of use; aesthetic and minimalist design; help users recognize, diagnose, and recover from errors; help and documentation [13].

3 The manual ingest is prone to errors; thus, the application should supply visual feedback for users based on validation checks (see paragraph II.A.2). Furthermore, the frontend should provide users with the possibility to change access rights of his/her own files/objects, adapt the layout individually, and set the level of user notification. This category receives 35% of relevance = 35 weight points. 4) Backend Like the frontend, the backend should have low software requirements. The installation of the backend has to be carried out by an IT expert only once. Notwithstanding, the requirements have a long term effect for underlying IT infrastructure. In the scientific environment and workflow, data represent the (static) fundament of research but are also working material, and are subject to dynamic changes, e.g. annotations. Therefore, the application should contain an automatic versioning and logging system, which enables the users to manage several versions of a file/object and to generate a history log for any file/object, so that the provenance can be determined at each point in time. To guarantee and enforce user policies and to generate correct history logs, the application has to conduct an access control. In the optimal case, the access control is role-based and can utilize existing authentication/authorization infrastructures (AAI) e.g. based on X.509 certificates. Finegrained access control is assumed to be important for any biomedical research data application. To assure synchronization with other payload metadata repositories at different locations and systems, an application has to provide import and export functions from/to at least one non-proprietary format. This category receives 25% of relevance = 25 weigh points. 5) Summary of the weight factors The main goal of the LABIMI/F project is the determination of relevant payload and corresponding metadata in the applied use cases. Afterwards these payload / data have to be stored in a structured manner and researchers should be able to up- and download them. For this reason the most important categories are the Metadata and payload management, which covers the capability to create and be adaptable to own content models, and the frontend. Each of them receives 35% of relevance. The category backend receives 10% less than the frontend because the installation and maintenance can be done by an administrator of the data center at a single location. The usability at the backend is therefore less critical than at the frontend, where researchers / non IT experts apply the tools. The category up-to-dateness indicates an active development of the tool, but is the least important category because it could be more efficient to customize an older proven tool than to start from scratch. B. Criteria for the data transfer tools The data transfer tools evaluation is divided into six categories: up-to-dateness; data transfer; provenance; technical requirements; usability; user management. The category up-to-dateness (5% of relevance) has the same criteria as the corr. category of the data management tools (see paragraph II.A). The category technical requirements (10% of relevance) corresponds mainly to backend and usability (20% of relevance) to frontend. 1) Data transfer The application has to provide the user with the ability to transfer data from one storage resource to another. In detail, it enables the user to establish a transfer between a local working computer and one, or even two, remote storage resource(s). To minimize the effort and waiting time for users, the transfer of multiple files/objects should be parallelized. Multiselection of files/objects and directories increases process efficiency as well. TABLE III. CATEGORY AND TOTAL APPLICATION SCORES (ALREADY WEIGHTED) OF THE INVESTIGATED DATA MANAGEMENT TOOLS. Tool Weights DSpace Fedora Commons ISA-Tab Tools Category Up-to-dateness 5% 15 of of of 15 Metadata and payload management 35% 50 of of of 105 Frontend 35% 63 of of of 105 Backend 25% 30 of of of 75 Total application score 100% 158 of of of 300 TABLE IV. CATEGORY AND TOTAL APPLICATION SCORES (ALREADY WEIGHTED) OF THE INVESTIGATED DATA TRANSFER TOOLS. Tool Weights Globus-Online PowerFolder irods CryptShare Category Up-to-dateness 5% 15 of of of of 15 Technical requirements 10% 30 of of of of 30 Data transfer 50% 94 of of of of 150 Usability 20% 46 of of of of 60 User management 10% 0 of 30 2 of of 30 0 of 30 Provenance 5% 0 of 15 0 of of 15 0 of 15 Total application score 100% 185 of of of of 300

4 Medical image data as well as genome data are to be anonymized or pseudonymized, but still should be encrypted for the transfer since researchers require confidentiality for their data until they are published. The applied encryption algorithm should be an implemented standard and may not be proprietary, e.g. the Advanced Encryption Standard (AES). This is the most comprehensive category and is set to 50% of relevance. 2) User management The application should provide mechanisms to use existing AAIs (see data management tools in paragraph II.A.4). The used IT infrastructure should enable users to collaborate interinstitutional. Thus, the application has to support different AAIs depending on the systems the different users work with. This category receives 10% of relevance. 3) Provenance To trace the whole history of an object, it is necessary to log all relevant data of a transfer. This includes initiator, receiver, source, destination, object, protocol, time, size, and status, and receives 5% of relevance [14]. 4) Summary of the weight factors Due to the main aim of the data transfer tools, to provide a secure and reliable possibility to move data, the category data transfer receives the highest weight factor of 50% of relevance (see table IV). The data transfer should be used to move data from the workstations of the researchers to the data management software and vice versa. Therefore, the researchers use this tool and thus the usability, which has a huge impact to the acceptance, is very important (see paragraph II) and receives 20% of relevance. The technical requirements are less important because the users have to install it only once and the most tools have an intuitive installer. The user management is as important as the technical requirements. It has to consider and enforce the privacy policies. Both categories receive 10% of relevance. The up-to-dateness (see paragraph III.A) and the provenance are the least important categories and receive only 5% of relevance. At this point it is important to distinguish between the provenance of the managed research data (see paragraph III.A.4), which is essential to proof the correctness and reconstruct the data, and the provenance of the transfer process, which contains information about the transfer, e.g. source(s). III. RESULTS The pre-selection via domain experts limits the set of investigated applications to DSpace [15], Fedora Commons [16], and Investigation / Study / Assay (ISA)-Tab Tools [17] for data management, and Globus Online [18], PowerFolder [19], irods [20], and CryptShare [21] for data transfer. The application scores of the data management tools are listed in table III, the scores of the data transfer tools in table IV. A. Data Management Tools DSpace is an open source digital repository used to manage data of any format as well as the corresponding metadata. The repository hierarchical structure consists of a community which can be divided into sub-communities. These can encompass collections which include items like files and metadata. DSpace provides mechanisms to customize the application layout and metadata schemes for different collections. The Fedora (Flexible Extensible Digital Object Repository Architecture) Commons Repository is an open source metadata repository to manage, preserve, and link data of any format with corresponding metadata. The data are organized in Fedora Digital Objects (FDO). An FDO consists of the following basic components: persistent identifier (PID), object properties, and 1:n datastream(s). The PID is a persistent and unique identifier for the FDO and its contents. The object properties are systemdefined descriptive properties to manage and track the object within the repository, and the datastream(s) represent metadata and payload item(s). Metadata include the Dublin Core standard, internal audit trail data, and references to digital objects. The ISA-Tab Tools are a collection of open source tools of the biomedical community used to assist local management of experimental metadata [17]. The data is managed in the (ISA) tab-delimited (TAB) format that each tool of this collection works with. The hierarchical structure of ISA-TAB consists of different investigations which include studies. Studies are divided into assay(s), which can point to other external files. B. Data Transfer Tools Globus-Online (GO) is a web-based data transfer system using GridFTP and X.509 certificates. GO enables direct data transfer between two storage resources, called endpoints. Both transfer endpoints need to have either a GridFTP server or client. The Globus Online Connect Tool can be used without a GridFTP server. This local tool can be installed easily and without administrative rights. Globus Connect enables the local computer to communicate via GridFTP, but the other endpoint requires a GridFTP server with a global IP address. Thus, a direct data transfer between two Globus Connect endpoints is not possible. Other than the data flow, the control flow is routed via a GO server, located in the USA. PowerFolder is a tool to synchronize files and folders. It provides a dedicated user management. Besides direct synchronization between two or more storage resources via a peer-to-peer mechanism, it offers an online backup/storage in a central repository. This allows synchronization even if the corresponding storage resource is offline. irods is an integrated rule-oriented data management system [22] with a user management adaptable to several AAIs, like the Grid security infrastructure, Kerberos, Shibboleth etc. As the data management options of irods are not relevant in our context, they are not further considered. irods is based on a client-server-architecture and allows the transfer of data either between a local and a remote resource, or between two remote resources. Furthermore, irods facilitates the creation and modification of event-driven rules. CryptShare is a web-based data transfer application used to send data up to 2 GB to everybody owning an address. The whole data transfer is directed over a CryptShare server

5 whose exact location could not be determined. The sender has to transfer the data to the central server; the intended user receives an with an URL and the password with which to access the file. The data is encrypted during transfer and temporarily stored. In addition, CryptShare offers logging for the administrator and user notifications, e.g. forwarding the sender an if the receiver has downloaded the file. IV. DISCUSSION Using the German university grading scale [12], only DSpace meets a sufficient number of criteria as a data management tool. According to the scale the other data management solutions failed the test. However, Fedora Commons scores highest in the backend category with metadata and payload management. Therefore, with a frontend, Fedora Commons would probably achieve the best score. The data transfer tools have to provide less complexity than the data management tools. Consequently, two of four tools achieve a satisfactory grade. All of the three investigated data management tools fulfill the category up-to-dateness very well but the remaining categories only by a limited degree. A. ISA-Tab The ISA-Tab tools are developed with a focus on the management of library data, biomedical experiment results, and their publishing. ISA-Tab includes tools to construct new metadata structures which are grouped by their functionalities. The focus on management functionalities of metadata and many criteria for the management of payload are not met. Thus, an additional tool to manage payload is necessary. The total application score of ISA-Tab is 112 out of 300 (37%), which corr. to 5.0; therefore, it will not be considered further. B. Fedora-Commons Fedora Commons provides a comprehensive metadata repository system. It is the only analyzed data management solution which enables relations between the managed objects. This feature allows for flexible modeling of multiple data relations beyond the classical hierarchical structure. Due to its modular architecture, Fedora Commons also is highly adaptable and customizable. Several tools are integrated by default, but may be replaced, if necessary. One example is the Java Messaging Service (JMS) [23], which by default is implemented via Apache ActiveMQ [24] and enables reacting on each system interaction. This reaction might be an administrator or user notification or logging to implement provenance. Indexing and (full text) search service are other features of the modular architecture of Fedora Commons and are implemented with Lucene [25], Solr [26]. Due to the used FDO data structure the exchange of the metadata needs to be realized via import / export functions, which convert the data from a given format into the FDO format or vice versa. Besides im- and export Fedora Commons provides an OAI-PMH interface to enable metadata harvesting [27]. Fedora Commons obtains the highest score in the category metadata and payload management with 91 out of 105 (87%), which corr. to a university grade of 1.7. The reason why Fedora Commons failed in total (5.0) is that it has no user management / user frontend and only a rudimentary administrator frontend. However, due to its modular architecture Fedora Commons provides several interfaces to be extended. With a frontend Fedora Commons can easily achieve the highest score. There are various applications, based on Fedora Commons, which add user management and graphical UI e.g. escidoc [28]. C. DSpace DSpace offers a web-based frontend with a score of 63 out of 105 (60%), which corresponds to a university grade of 3.7. The web frontend is the most remarkable feature of DSpace. It is highly customizable, enables the administrator to create a hierarchical data structure and to model metadata templates for multiple different projects. The user management is satisfactory and provides user groups. Group-based access control is implemented for data objects; users can only have no or full access to a collection (set of items). A more fine-grained access control would be necessary for the LABIMI/F project. DSpace provides the possibility to structure data in collections and communities, but not to model relationships. The total application score of DSpace is 158 out of 300 (53%), which corr. to 4.0. It is deemed sufficient by only 4% but obtains the highest score of all investigated management tools. Thus, DSpace will be object of further investigations. D. CryptShare CryptShare received the lowest score of the investigated data transfer tools. The data are stored and transferred encrypted but the transfer is limited to 2 GB and the server location is unknown. The server might thus be located in a foreign country, which means that its use would be illegal, as German data protection and data security laws state that this kind of data must be stored and processed in Germany [29], [30]. The limitation to 2 GB is another disadvantage because medical image data as well as genome data are often very much bigger. Both facts and the total application score of 102 out of 300 (34%), which corr. to 5.0, lead to the conclusion that CryptShare is not suitable for the LABIMI/F project. E. irods irods is a data management tool for Grid computing environments. It contains functions to replicate and manage distributed data as well as corresponding metadata. This functionality can complement data management tools, but due to the command line base usage these functionalities are not analyzed further. irods provides an interface to the grid security infrastructure, which enables this application to be part of a grid infrastructure. Furthermore, it contains an own user management, and sufficient access control. The total application score of 148 out of 300 (48%) corr. to 5.0. The negative score results from the score outcome in the category data transfer: unencrypted transfer, no features for the selection of multiple files, no parallelized transfer, and no possibility to pause and resume transfers.

6 F. Globus-Online Globus-Online performs well in the categories technical requirements, data transfer, and usability. It provides an easyto-use web-based frontend which enables multiple file selection and parallel transfer, including remote-to-remote transfers / connections. However, user management and access control depend on the implementation of storage resources. Globus Connect enables communication with a GridFTP server, even when the client is behind network address translation (NAT). However, two problems occur: 1) it is not possible to transfer data between two Globus Connect clients (see paragraph III.B). 2) The used ports 2223 (server) and (client) [31] are usually blocked by hospital firewalls. Globus-Online receives a total score of 185 out of 300 (62%). This corresponds to a university grade of 3.3. However, the control flow via a server in the USA can be subject to data protection law. In the USA the patriot act [32] allows the US government to access all data within their territory. Thus, Globus-Online is less suitable for the LABIMI/F project. G. PowerFolder PowerFolder has only a satisfactory user management and by default no logging mechanisms for provenance information. It provides users the ability to invite other users to access their files / folders via but has no sophisticated access control. Nevertheless, PowerFolder is a solid data transfer tool using peer-to-peer technology. It enables the users to synchronize files and folders and it is easy to use and receives a total score of 190 out of 300 (63%). According to the highest score and because it has been selected as local solution in Goettingen, PowerFolder will be included in further evaluations. V. OUTLOOK Due to the limited performance of the investigated tools, further research and analysis especially for the data management tools is necessary. The evaluation indicates an advantage of the modular architecture and potential of Fedora Commons as a backend solution. Thus, additional data management tools based on Fedora Commons will be evaluated, e.g. escidoc [28] or Islandora [33]. Furthermore, the irods as well as DSpace can be applied as possible additions to Fedora Commons. The German project WissGrid [34] develops an extension to Fedora Commons for irods, where irods can provide integrated payload data transfer capabilities, while Fedora Commons acts as metadata repository. In addition, the DSpace community has announced to use Fedora Commons as metadata repository instead of their own. Both Fedora Commons extensions are of high relevance for the LABIMI/F project. ACKNOWLEDGMENTS This publication was supported by the joint project WissGrid, funded by the German Federal Ministry of Education and Research (BMBF), FKZ 01IG09005A-L, and the project LABIMI/F, funded by the German Research Foundation (DFG), FKZ RI 1000/2-1. REFERENCES [1] K.G. Helmer, et al.: Enabling collaborative research using the Biomedical Informatics Research Network (BIRN). Journal of the American Medical Informatics Association 18, , [2] S.D Kahn: On the Future of Genomic Data. Science 331, , [3] F. Dickmann, et. al.: Solutions for biomedical grid computing--case studies from the D-Grid project Services@MediGRID. Journal of Computational Science 6394, [4] F. Dickmann, R. Grütz: LABIMI/F - Digital Preservation of Biomedical Research Data, Poster_LABIMI-F_v6.pdf, Date accessed: [5] C. Zangemeister: Nutzwertanalyse in der Systemtechnik, Hamburg, [6] AWI, [7] nestor, [8] SUB, [9] WisNetGrid, [10] R. L. Keeney, R. F. Meyer, H. Raiffa: Decisions with multiple objectives. Preferences and value tradeoffs, Cambridge Univ. Press, Cambridge, [11] ISO/IEC , Information Technology -- Metadata registries (MDR) Part 1: Framework. [12] T. Karran: Pan-European Grading Scales: Lessons from National Systems and the ECTS. Higher Education in Europe, 5-22, [13] J. Nielsen: Usability Engineering Morgan Kaufmann, [14] Deutsche Forschungsgemeinschaft (ed.): Vorschläge zur Sicherung guter wissenschaftlicher Praxis. Empfehlungen der Kommission "Selbstkontrolle in der Wissenschaft". Denkschrift Wiley-VCH, Weinheim, [15] DSpace, [16] Fedora Commons - Home, [17] ISA-Tab Tools, [18] Globus-Online, [19] PowerFolder, [20] irods, [21] Cryptshare - Home, [22] M. Hedges, T. Blanke, A. Hasan: Rule-based curation and preservation of data: A data grid approach using irods. Future Generation Computer Systems 25, , [23] R. Monson-Haefel, D. A. Chappell: Java message service, O'Reilly, Cambridge, [24] R. Henjes, et al.: Throughput Performance of the ActiveMQ JMS Server Kommunikation in Verteilten Systemen KiVS, , 2007 [25] Apache Lucene, [26] Welcome to Solr, [27] OAI-PMH, [28] escidoc, [29] Directive 95/46/EC of the European Parliament and of the Council of 24 October 1995 on the protection of individuals with regard to the processing of personal data and on the free movement of such data, [30] R. Gellman: Privacy in the Clouds - Risks to Privacy and Confidentiality from Cloud Computing, Cloud_Privacy_Report.pdf, Date accessed: [31] Globus Connect, Connect, Date accessed: [32] M. T. McCarthy: USA Patriot Act, Harv. J. on Legis. 39, 435, [33] Islandora, [34] WissGrid,