Tools for Data Management. Research Data Management : Session 3 9 th June 2015

Transcription

1 Tools for Data Management Research Data Management : Session 3 9 th June 2015

2 What do we mean by tools for data? A system that automates in some way the process of creating, transforming, analysing, visualising, enriching or arranging data.

3 What kind of tools are available? Traditional Modern Free/ Open source Commercial off the shelf Innovative Experimental Commercial tailored Self-build

4 What kind of tools are available? Factors to bear in mind: Project Constraints Traditional Cost Time Procurement Number of users Innovative Types of users Modern Experimental User competence Free/ Open source Local support Commercial tailored Commercial off the shelf Replication of existing systems Philosophy/thinking system Self-build

5 Treat data as a flow Consider: variety velocity volume

6 Metadata File Data Processing Sharing Long 2. Storage Data Management Term and Collection & and Dissemination & Preservation Documentation Enrichment Collaboration Tools Planning

7 1. Data Management Planning

8 Data Management Plans (DMPs) Often required by funders Submitted at grant application stage EPSRC a notable exception nothing at application stage but a DMP is expected for each project Useful for checking you ve considered all aspects of your data management Covers each aspect of the lifecycle of your data Never to late to do one for your project!

9 Seven key sections: 1. Data collection 2. Documentation and metadata 3. Ethics & Legal Compliance 4. Storage & Backup 5. Selection & Preservation 6. Data Sharing 7. Responsibilities and resources

10 DMP Online

11 But remember

12 2. Data Collection Tools

13 Bring your own data: Born digital, digitised and non digital Born digital Digitised Non-digital Capture Complete data Varying degrees of capture No capture Example Application Data format Simulation, animation, HPC, webarchive.org Numerical, textual, graphical representation Microscopy Historical manuscript transcription Video, audio, photo, numerical (sensors) Paper survey Physical object

14 Experiment producing data Experimental data storage Analytical environment data storage Data validation Convert from proprietary format Data analysis (destructive) Archive raw experimental data Preserve research data outputs

15 Trends in data collection Exponential resolution of imaging Citizen Science/Crowdsourcing Zooniverse/Scripto New sources of data Social graph. Big data to elicit the rare and tiny New uses for existing data

16 Use of Existing Data What data exists that you could use? Do you have to create all of your data from scratch? Key expectation for data management planning Tools to helps find datasets in your discipline: Funder specific data centres; e.g. UK Data Service (ESRC); NERC data centres DataBib - Registry of Research Data Repositories - Figshare - Dryad Zenodo -

17 All Datasets are equal? Data sharing requires excellent RDM Most repositories will have minimum standards for: Organisation of data Documentation Rare also to fit datasets that exactly fit your needs Balance between time take to repurpose vs time take to reproduce Justify decision to funder

18 Data Sharing Example

19 3. Metadata & Documentation

20 Leave yourself a Rosetta Stone What might seem obvious to you now might not be so obvious in a year or two!

21 Documentation Who created it, when and why Description of the item Methodology and methods Units of measurement Definitions of jargon, acronyms and code References to related data

22 Metadata May be automatically provided by the software used to generate or analyse the data Standards for metadata will vary widely between disciplines DataCite is a general standard that covers the basics of most data: Controlled vocabularies Useful if operating in a group, provides consistency Jisc Media has a list of subject specific controlled vocabularies

23 Basic metadata for a dataset Title Unique resource identifier Description Subject Creator(s) Funder Resource Language Publication date Publisher Contact address Title for the dataset Working data = project ID or departmental identifier. Published data = DOI or persistent URL Abstract for the dataset Subject or classification code describing the resource Main researchers involved in producing the data Source of financial support for the dataset Default will be set to 'eng' (English) Date when the data is publicly available Publisher of dataset, University of Warwick in many cases Person to direct correspondence related to the data set to

24 Tools Taverna (open source) Document your workflow myexperiment.org Share your workflow: find, use and share scientific workflows and other Research Objects, and to build communities. LabTrove (open source)*, labguru (commercial), textgrid (free), LIMS Electronic Lab book: enables researchers to share their experimental plans, thoughts, observations and achievements with the wider online community in a secure, semantically rich and extensible manner. Onenote (free)/evernote (freemium) /kustomnote (freemium) Christian Proske

25 4. File Storage and Collaboration

26 Information Security Moodle.warwick.ac.uk Information security essentials

27 Collaborative Storage Can I... OneDrive for Business Files.Warwick Shared Folders + MyFiles/ ExpanDrive Send files directly to people? Yes Yes Yes No Share files with colleagues? No Yes Yes Limited Access this from the web? Yes Yes Yes Limited Mechanism for inbound external files? Yes No Yes No Work with large files (50Mb+)? No Yes Yes Yes Easilly manage lots of files? No Yes Limited Yes Work collaboratively? Limited Yes No Yes Share files externally? Limited Limited Yes No Directly edit files? No Yes Yes Yes

28 5. Processing & Enrichment

29 What does this mean? Enrichment Annotation Clean-up (ETL) Editing Analysis Structural Visualisation Spatial Network Content Interpretation Contextualisation Modelling Theorising

30 Workflow examples

31 6. Sharing & Dissemination

32

33 7. Long Term Preservation

34 Tools for Preservation Wide range of tools for preservation BUT most work best on a large scale Choose a repository/archive based on: Discipline / Funder instruction Tools / services offered Best fit for project Look to use the resources on finding datasets to get information about repositories and data centres Key role for projects and researchers is deciding what to preserve, where and when rather than how

35 What data will you save? What must you save? Legally, contractually etc. How long do you need to preserve the data? Short term (e.g. 3-5 years) Long term (e.g. 10 years) Indefinite Keep all versions? Just final version? First and last? What are the re-processing costs? Keep only software and protocol/methodology information? Are there tools/software needed to create, process or visualise the data? Archive these with your data if possible!

36 File formats for preservation Unencrypted Uncompressed If possible, sometime compression is vital Non-proprietary/patent-encumbered E.g. avoid Word, Excel, Quicktime Risk management key Open, documented standard Standard representation (ASCII, Unicode) Examples: PNG, CSV, RTF, HTML

37 Useful resources Software Wish List Innovations in Scholarly Communication - the Changing Research Workflow DIRT Digital Research Tools directory

38 Thanks for listening! Any Questions? Steve Ranford, Senior Academic Yvonne Budden, Scholarly Communications Manager Research Data Management Support