Data Workflow Workshop

Transcription

1 Data Workflow Workshop Elizabeth Wickes, Data Curation Specialist Heidi Imker, Director of Research Data Service Upcoming workshops Data Rescue (Data Management Workshop 1) October 4, 12-1pm, Grainger 335 Registration link: Have a mess of data floating around your computer or lab? Data management is an essential task for students and faculty but hard to get started. Go no further! In this workshop you will identify, group, and plan on how to reorganize your current and future data. We will discuss strategies for organization, folder structure, and create an organizational plan. This workshop is the first in a three- part series about data management. Creating Data Documentation (Data Management Workshop 2) October 11, 12-1pm, Grainger 335 Registration link: Writing project, code, and data documentation doesn t need to be the worst part of your day. This hands on workshop will give you experience using various types of documentation, discuss strategies for writing documentation, and get you started writing a template for your projects. Bring a dataset you d like to work with but examples will be provided. Preparing for Data Sharing (Data Management Workshop 3) October 25, 12-1pm, Grainger 335 Registration link: Making research data public is becoming a reality for many disciplines, but for many researchers and disciplines there is a complicated set of issues to consider before publication or sharing data. This workshop will cover the basics steps of research data publication, from considerations to depositing. Participants will work through guidance to help them make decisions about when and how to publish or share data. Smart and Simple Data Management October 7, 1-2:30pm, Funk/ACES 509 Registration link: October 12, 1-2:30pm, Grainger 335 Registration link: This session aims to provide you with data management best practices and tools to increase your research efficiency and impact. We ll present a basic introduction to data management using a data management plan framework, hands on activities, and discuss how to find and vet resources for making data publicly accessible. Workflow Mapping Handout 1

2 Workflow Mapping Handout 2

3 Workflow Mapping Activity: Cheat sheet Data Products: These are the things that you make in the course of the research. Examples: data files, visualizations, documentation, etc. that are produced in the course of the workflow. Provide the names and locations when possible. Activities: Think very high level and use active voice. These should all be tasks or stages within your project. Scripts, Software, & Tools: These should be the things that you use in the course of the project. Examples: scripts, software tools, microscopes or other scientific instruments, documentation, etc. Provide names and locations when possible. Workflow activities Large yellow post- it. Name the activities or stages within your project. Start at the very high level and only add detail when needed. These create columns that data products and tools belong to. Data Input/Output (from others) Small pink post- it. Name the data products received from or created for other people on your team. Data (NOT for others) Small light blue post- it. Name the data products or other files created during your workflow but not used by other people. Tools Used Small dark blue post- it. Name the scripts, software, and tools used at each stage of the project. Notes or Annotations Small yellow post- it. Use these to make notes or other annotations within your workflow. Workflow Mapping Handout 3

4 Workflow Mapping Handout 4

5 Example Completed Workflow Diagram, for reference test$dataset$ training$ dataset$ model$ sqlite$ database$ visualiza4ons$ for$paper$ Data$Input$ (from$others)$ report$for$ developer$ Data$Input$ (from$others)$ Ac4vity$1 $ Ac4vity$2 $ Ac4vity$3 $ Ac4vity$4 $ Loop$here$ un4l$model$ fits$ script1.py$ some$other$ socware$ analysis.r$ model$ fi@ng$script$ for$dev$ Workflow Mapping Handout 5

6 Homework! Write up a summary of the data inputs that you use and data (or other research materials) that you create for other people. Share this with your group/team leader. Think about how you might transcribe your workflow to more formal documentation. Other important considerations: Strike a balance between consistency and uniqueness in your file/folder names. Include important metadata, such as author, date, location, version number, etc. in the file name when possible. Use YYYY- MM- DD format for your dates in all places, and enforce this in your labs, project groups, etc. Discuss and document standardized or controlled vocabularies for acronyms, project names, and especially any other qualitative coding project. Create project documentation, even if you only have time to make it brief. Develop templates for yourself or members of your lab to fill out for every project. Follow domain- specific metadata schemas when appropriate. Keep your raw data protected and unchanged. At a minimum, keep an extra copy of the original raw data somewhere in case of accidental changes or deletions. Keep separate copies of every data transformation made during your processing stages. Backup your data and projects! Follow the rule for backups, meaning that you should have three copies: using two media types and one remote copy. Utilize an external hard drive and your Box account (or other applicable cloud data storage) to make regular backups of your projects. Develop backup access plans for any proprietary or legacy software required for your research. Use version control for all code. GitHub Desktop is a lightweight application that can track changes to files within a folder. Projects can also be committed to GitHub repositories (private options are available) for additional backup. Several analytics platforms, including R Studio, have git and GitHub integration built in. Need more information about any of the above? The Research Data Service ( team is happy to provide more information about all data management topics and offers personal data consultations. They can be reached at researchdata@library.illinois.edu or on This workshop and many others can be run for your research group or lab by request. All Research Data Service consultations are free and confidential to all members of the Urbana- Champaign campus community. Workflow Mapping Handout 6