Deliverable A comprehensive and standardised e-infrastructure for analysing medical metabolic phenotype data. 1st September 2015

Transcription

1 Deliverable Project ID Project Title A comprehensive and standardised e-infrastructure for analysing medical metabolic phenotype data. Project Acronym Start Date of the Project PhenoMeNal 1st September 2015 Duration of the Project 36 Months Work Number Package 9 Work Title Package WP9 Tools, Workflows, Audit and Data Management Deliverable Title D9.5.1 Updated Preprocess Virtual Machine Image Delivery Date M32 Work leader Package IPB Contributing Partners IPB, ICL, EMBL-EBI, SIB Authors Evangelos Chandakas, Tim Ebbels, Pablo Moreno, Steffen Neumann, Kristian Peters, Rico Rueedi, Daniel Schober Abstract This deliverable reports on the development and use of container images to enable data producers to (pre-)process raw data into standard community-supported formats, locally

2 or in the cloud. This deliverable is an update to D9.2.1 PhenoMeNal-Preprocess VM. Table of Contents 1. Executive Summary Contribution towards the project objectives Detailed report on the deliverable Background MS Data Preprocessing NMR Data Preprocessing nmrmlconv Container User documentation and training BATMAN Delivery and Schedule Conclusion

3 1.Executive Summary The PhenoMeNal project supports several of the most common workflows in metabolomics covering, amongst others, NMR and Mass Spectrometry data processing. The diversity of instrument vendor-specific data formats requires preprocessing tools to convert data files from proprietary formats into standardized open formats for instance, converting raw MS and NMR data files into the mzml or nmrml formats, respectively. This deliverable reports on the development and use of container images to enable data producers to (pre-)process raw data into standard community-supported formats, locally or in the cloud. This deliverable is an update to D9.2.1 PhenoMeNal-Preprocess VM. 2. Contribution towards the project objectives The deliverable has contributed towards the following project objectives for WP9: Specify and integrate software pipelines and tools utilised in the PhenoMeNal e- Infrastructure into VMIs, adhering to data standards developed in WP8 and supporting the interoperability and federation middleware developed in WP5. We will develop new applications only to complete missing links in pipelines. We will use public repositories and continuous integration to always provide development snapshots of the infrastructure VMIs. Develop methods to scale-up software pipelines for high-throughput analysis, supporting distributed execution on e.g. local clusters, private clouds, federated clouds, or GRIDs. 3. Detailed report on the deliverable 3.1 Background The PhenoMeNal project supports several of the most common workflows in metabolomics, covering NMR, Mass Spectrometry and downstream statistical analysis. The diversity of instrument vendor-specific data formats creates incompatibilities between processing tools and strategies that can be avoided by using community-accepted open standard data formats starting from the instrument level. Hence, PhenoMeNal supports preprocessing tools to convert data files from proprietary formats into standardized open formats for instance, converting raw MS and NMR data files into the mzml or nmrml formats, respectively. 3

4 This deliverable is an update to D9.2.1 PhenoMeNal-Preprocess VM: Virtual Machine Images to enable data producers to locally process raw data into standard formats supported in PhenoMeNal MS Data Preprocessing For the conversion vendor formats to mzml we are using the open source msconvert, developed by the ProteoWizard team ( 2, which is one of the reference implementations for mzml. It can convert to mzml from Sciex, Bruker, Thermo, Agilent, Shimadzu, Waters and also the earlier file formats like mzdata or mzxml and is consequently widely used. MSconvert and the integration into the continuous integration Jenkins and as Galaxy tool have already been described in D Since then, we have especially worked on the container image. The container was rebased from the seemingly unmaintained container suchja/wine:dev with pre-installed WINE (v1.8.0) windows layer, to the i386/debian:stretch-backports image that is part of the docker-library/official-images collection with regular updates. This also allowed to move to the WINE version 3.6.0, supporting a wider range of windows applications and better stability. The build file was also cleaned up, where several workarounds related to the earlier WINE version could be removed. The Proteowizard version was updated from version to The Dockerfile was also updated to conform to the latest PhenoMeNal guidelines, i.e. has updated LABELs that are used in the Build process, on the Jenkins continuous integration 3 and in the app library 4. A new data testing strategy was added, where we have started to collect a range of different MS files in vendor formats, which are then converted. We then check that the converted mzml matched a known-good output as Jenkins job Chambers, Matthew C et al. "A cross-platform toolkit for mass spectrometry and proteomics." Nature biotechnology (2012):

5 Figure 1: Screenshot of MSconvertGUI converting Thermo RAW data in a docker container on Ubuntu/Linux. It is now possible to also convert Thermo RAW files in this container using the GUI under Linux/X11, which is a long-standing wishlist item in the metabolomics community (see screenshot in Figure 1). However, it is not yet possible to perform this conversion also with the command line msconvert.exe tool. We are in contact with the Proteowizard team and identified the underlying reason, which lies in the complex interaction between VisualStudio 2012 C++ and the windows CLR runtime used in the msconvert.exe tool and vendor libraries. At the HUPO PSI 2018 meeting ( ) in Heidelberg (Germany), we were in contact with Jim Shofstahl from Thermo Scientific. We have received a version of the Thermo RawFileReader that is based on.net, and avoids the C++ runtime issues under WINE/Linux. We have created a local container- RawFileReader successfully running a command line program that extracts required information from the Thermo RAW file under linux, and informed the Proteowizard team. We can hence expect that an upcoming version of proteowizard will be able to successfully convert also Thermo data on the command line, and henceforth also in Galaxy.. We also worked on the license compliance. Due to the nature of the container registries and the kubernetes cloud setup it is not possible to request acknowledgement of the license terms during a download step, as is the case for the Proteowizard binaries 6. Instead, we are using a mechanism that is also used in the pwiz build process, where adding --i-agree-to-the-vendor-licenses while building will acknowledge the vendor licenses. With this analogy, the resulting container is called phnmnl/pwiz--i-agree-to-thevendor-licenses. All licensing information was added to the container repository and the container itself

6 Several tools in the PhenoMeNal workflows can process metabolomics data in mzml, including XCMS, CAMERA and MetFrag (via MSnbase), and the OpenMS tools. 3.3 NMR Data Preprocessing The nmrml standard is an open XML-based exchange and storage format for NMR spectral data. The nmrml format is intended to be fully compatible with existing NMR data for chemical, biochemical, and metabolomics experiments. nmrml can capture raw NMR data, spectral data acquisition parameters, and where available spectral metadata, such as chemical structures associated with spectral assignments, see Figure 2 for an overview of the role of nmrml. The nmrml format is compatible with pure-compound NMR data for reference spectral libraries as well as NMR data from complex biomixtures, i.e., metabolomics experiments. The manuscript nmrml: A Community Supported Open Data Standard for the Description, Storage, and Exchange of NMR Data has been published recently 7. Figure 2: workflow of nmrml data flow. The main route to nmrml-formatted data is using our open source converter nmrmlconv, which is part of the nmrml package. The converter is also packaged as a container image, together with the required Java runtime environment, and is available as a Galaxy tool

7 Since the Deliverable D9.2.1, we have updated the container image to the latest version of nmrmlconv and we have created a Galaxy module and a Galaxy workflow that uses nmrmlconv to process NMR RAW data, see Figure 3. The module can be used with both singular and many NMR RAW acquisitions that are bundled in a dataset collection. When processing the nmrmlconv module, the Galaxy workflow engine starts the nmrmlconvcontainer 8, which is launched within the PhenoMeNal e-infrastructure and is further described below. In the process, we have raised several github issues 9 to further improve the nmrml converter. Figure 3: Screenshot of the NMR workflow running in Galaxy. The module nmrmlconv has been emphasized and is now integral part of the NMR RAW processing workflow nmrmlconv Container The nmrmlconv container itself integrates the official nmrml converter sources 10 and packages them into a docker container. This container is launched within the PhenoMeNal e-infrastructure. In order to ensure that NMR RAW data is processed reproducibly, our Continuous Integration Framework automatically tests the container for

8 launching correctly 11 and with actual data 12. The nmrmlconv currently can process NMR RAW data from any of Bruker, Varian and Jeol vendor formats. The vendor to nmrml converter itself was annotated via the EDAM ontology and published under bio.tools repository (see Bio.tools now enables to find this tool via appropriate controlled vocabularies describing a tools function and I/O formats. We are also currently investigating how we can add/use nmrml as vendor independent NMR raw data standard within the newly emerging NMReData Record.zip NMR assignment data standard (see This standard could gain momentum at the late stages of NMR processing workflows, as it is supported by many molecule to spectrum feature assignment tools (e.g. mnova, TopSpin, ) and small molecule databases (e.g. NMRShiftDB, C6H6.org,...). The NMReData standard currently stores the vendor native formats in the Record.zip file, which impairs cross-vendor data access, e.g. as desired in molecule analytics databases. Repository providers (e.g. the drug data repository) have indicated interest to switch to nmrml due to that drawback and would therefore welcome nmrml inclusion User documentation and training The nmrml conversion is part of the NMR Workflow MTBLS1 Tutorial 13, was featured in a YouTube tutorial 14, and covered in the workshops CloudMET 15 and at the MetaboMeeting

9 Figure 4: User tutorial for the NMR workflow including the nmrml conversion BATMAN Bayesian AuTtomated Metabolite ANalyzer (BATMAN) for NMR is a protocol for automated metabolite deconvolution and quantification from complex NMR spectra. BATMAN deconvolves resonances from 1-dimensional NMR spectra and assigns them to specific metabolites from a target list and obtains concentration estimates. It applies a Markov Chain Monte Carlo (MCMC) algorithm to sample from a joint posterior distribution of the model parameters and obtains concentration estimates with reduced error compared with conventional numerical integration and comparable to manual deconvolution by experienced spectroscopists. BATMAN is available for PhenoMeNal galaxy users. The BATMAN workflow performs a 1D NMR spectra analysis using NMR raw data, coming from e.g. the MetaboLights database. The user can connect to this database via PhenoMeNal galaxy and can import data using the tool Metabolights downloader. Then, the raw NMR data is converted to multiple nmrml files using the tool nmrmlconv. As a next step, using the tool ZIP nmrml Collection a zip archive of a nmrml collection is created, which is imported via the nmrml2batman Converter, followed by the 9

10 remaining BATMAN data processing workflow, which will be covered in the upcoming D9.5.2 Updated Data Processing. Figure 5: NMR workflow that includes the nmrml2batman conversion and the BATMAN tool. 4. Delivery and Schedule The delivery is delayed: No 5. Conclusion In PhenoMeNal, we have worked to improve the preprocessing of metabolomics data, which starts with the conversion of raw vendor file formats to open formats supported in PhenoMeNal. We are now supporting the conversion to the mass spectrometry data format mzml and to nmrml on non-linux systems. Our efforts have been acknowledged by other developer in the mass spectrometry and workflow community. 10