BIG DATA BIG IMPACT: IMPROVING THE REACH OF YOUR RESEARCH DATA

Transcription

1 BIG DATA BIG IMPACT: IMPROVING THE REACH OF YOUR RESEARCH DATA TATIANA ZARAISKAYA E-SCIENCE QUEEN S FRANCINE BERISH GEOSPATIAL DATA QUEEN S PLAN Big Data Basics (definition, Hadoop, semantics, tableau ) Big Data Challengers (examples) Data Services at Queen s DMP Assistant - Data Management Planning tool tz10@queensu.ca francine.berish@queensu.ca 11/21/ BIG DATA VOLUME Bon Echo pictograms until 1985 print only end 90s digital publications digital-born materails LinkedIn facebook youtube 2006 twitter instagram + VALUE = 5 V s 11/21/ /21/

2 WHAT IS THE PREDICTION? Typical pdf article 2-3 MB, movie 3-5 GB 11/21/ /21/ HADOOP ARCHITECTURE HADOOP facts Open-source Apache project out of Yahoo, written in Java Distributing data storage and parallel processing Provides linear scalability on commodity hardware Adopted by Amazon, Google, Facebook, Twitter, ebay 11/21/ /21/

3 WHY HADOOP? Scalable Flexible Inexpensive storage for Peta-byte data sets Yahoo! 100,000 CPUs in 40,000 servers running Hadoop, biggest 4500 nodes. Over 455 Petabytes of data in Hadoop. ~ 4times larger than Facebook batch queries and analytics process hierarchical data - structured and unstructured WHEN DOES DATA BECOME INFORMATION? 11/21/ /21/ METADATA FOR BIG DATA metadata is information that describes data; essentially, data about data. It is the descriptive, administrative and structural data that defines assets. City TYPES OF METADATA Descriptive Structural Kingston Ontario 11/21/ /21/

4 11/21/ /21/ KNOWLEDGE RELATIONSHIP WITH DATA KNOWLEDGE DISCOVERY SEARCH TOOLS LINKED DATA DATA IN DATABASES, PUBLICATIONS 11/21/ /21/

5 BIG DATA SEMANTICS -VISION Ontology data model that represents knowledge as a set of concepts within a domain and the relationships between concepts. Form of knowledge management URI Uniform Resource Locator /physics/people/ mary.xml Ontologies are represented by CLASSES and RELATIONSHIPS CLASSES RELATIONSHIPS Resource Description Framework = RDF ibrary/people/dat aservice/tanya.xml THIS IS A TRIPLE 11/21/ /21/ TABLEAU big data visualization tool 11/21/ /21/

6 BIG DATA, BIG CHALLENGES DATA PRIVACY DATA QUALITY DATA OWNERSHIP DATA SECURITY Data Privacy: Safety in Numbers 11/21/ By Alexandra Cooper, with revisions Data and Web Support Assistant, Data Services Why is data privacy important? What is the difference between de-identified and anonymous data? De-identified data Severing of a data set from the identity of the respondent, but preserving identifying information which could be re-linked. name, , IP address, etc. are removed from dataset and stored separately from the data. These are replaced by a unique code that creates a new identity for the respondent. Researchers are able to re-establish the link between the respondent and the data. Anonymous data Irreversibly severing a data set from the identity of a respondent preventing any future re-identification even by the study s creators. identifying variables have been removed (deleted) and not even the researcher can connect the data to the individual who provided it

7 What is the danger with microdata? Ah ha!! How do we make a dataset anonymous and minimize risk of disclosure? Suppress Variables remove variable completely from the dataset leave variable in the original dataset but not in the public version documentation may note that the variable exists in the original dataset, but no data is included in the public version Regroup Variables collapse categories of the original variable Create Derived Variables using two or more variables, create a new variable Variables with Text Fields open-ended questions allowing free-text responses can cause problems edit or suppress these variables, depending on content Example - Regroup Variable Example - Derived Variable Original variable Study Site Regrouped variable (new variable) Study Region Original variables Type of program attended PROGRAT01 - Educational/Vocational PROGRAT02 - Therapy PROGRAT03 - Anger Management PROGRAT04 - VPP or other Violence Programs PROGRAT05 - NSAP or other substance abuse programs PROGRAT06 - Cognitive skills PROGRAT07 - Religious PROGRAT08 - Restorative Justice/Victims Impact PROGRAT09 - Preparation for Release PROGRAT10 Other PROGRAT11 - Sex Offender Programs Inmate attended program 0 No 1 Yes From: Practice and Research in Nursing (PRN) Wound Studies From: Inmate Health in Canadian Federal Prisons 7

8 Example - Derived Variable, continued Example - Variable with Text Fields New derived variables prograt_ed Attended education, vocational, cognitive skills, prepare for release programs (based on PROGRAT01, PROGRAT06, PROGRAT09) prograt_ther Attended therapy, counselling programs (including anger management, violence programs, substance abuse, sex offender programs) (based on PROGRAT02, PROGRAT03, PROGRAT04, PROGRAT05, PROGRAT11) prograt_reli Attended Religion, Restorative Justice, victim impact programs (based on PROGRAT07, PROGRAT08) prograt_other Attended other programs (based on PROGRAT10) Inmate attended at least one program listed 0 No 1 Yes Text edited and left in dataset Good system of care via [Nursing Agency], Rehab, and [Senior s residence] From: Practice and Research in Nursing (PRN) Wound Studies Text suppressed [--]. Very open, very pleasant. Came back for [--] and [--]. He is [--], the [--]. He tries to make sure ensure that [--] strong [--] for [--]. He has [--] because of [--]. He [--] on the outside, and has [--], but [--]. He finds it [--] he feels that he is [--]. He talked about [--] and [--], including the [--]. From: Inmate Health in Canadian Federal Prisons Changing conceptions of recognized data Questions? NOVEMBER

9 Queen s Humanities & Social Sciences Data Geospatial Data Local Dataverse installation HISTORICAL MATERIALS LIKE DIARIES CAN DIGITIZED WITH OCR AND ANALYZED AS A BIG DATASET Diary by Barnaby, on Flickr 11/21/ VISUALIZATIONS IN THE NEWS Syrian IDP s and Refugees in the neighbouring countries (as of ). European Commission. 11/21/ Using maps and data vis to understand tennis. National Geographic. September 4,

10 The Scholars Geoportal INTRODUCTION TO: Data Models in GIS 11/21/ GEOSim by Ph.D. Pau Fonseca i Casas retrieved April 3, Conceptualization of GIS framework by Georgetown University retrieved April 2, 2016 Image courtesy of City University London, Health Geomatics GEOGRAPHY MATTERS: MAPS AND GOVERNMENT DATA 10

11 PR CMA CT QUEEN S DATAVERSE 11/21/ /21/

12 11/21/ /21/ /21/ /21/

13 BIG DATA EXAMPLES AT QUEEN S Geology Classics Kinesiology Epidemiology Medicine English Digital Scholarship 11/21/ As datasets increase in size, so does the need for data visualization 11/21/ /21/

14 WHY MANAGE YOUR RESEARCH DATA? 11/21/ BENEFITS OF RESEARCH DATA MANAGEMENT: ensure long-term preservation of data for future researchers meet grant requirements and/or produce a more competitive grant application increase the impact and visibility of your research encourage discovery and use of existing data to explore new research questions make sure your data are accurate, complete, authentic, and reliable, and reusable ensure compliance with ethics and privacy policies 11/21/ Funding Agency Policies NSERC, SSHRC, CIHR RESEARCH DATA MANAGEMENT PLANNING Tri Agency Statement of principles on Digital Data Management July /21/ /21/

15 THE RESEARCH PROCESS FROM A RESEARCHER S PROSPECTIVE: FROM A DATA MANAGEMENT PROSPECTIVE: Idea Ethics Funding Research Results [data] Process often seen as a means to an end Idea Ethics Funding Research Results [data] More emphasis on the importance of data And documenting the entire research process /21/ /21/ WE CAN HELP YOU WITH: DATA QUEEN S Planning for data management data management plans & grant applications Preparing your data coding, formatting, anonymizing, and more Documenting your data describing your data so others can understand and use it Archiving your data finding an appropriate repository or archive Making your data discoverable promoting open access, indexing, assigning DOI s 11/21/ /21/

16 Beyond Queen s Discipline-specific Data Repositories 11/21/ Full list at NIH Full list at NIH Full list at CIHR 16

17 QUL COLLABORATION WITH CAC o/dataverse/que ens 11/21/ /21/ DMP ASSISTANT National bilingual data management planning tool for researchers aligned with funding agency requirements Template with guidance addressing: Data collection Documentation and metadata Backup and storage Preservation Data sharing and reuse Responsibilities and resources Ethics and legal compliance 11/21/

18 11/21/ /21/ /21/ /21/

19 11/21/ /21/ WRAPPING UP Research Data Management at Queen s University: academic.services@queensu.ca New Tri-Agency Statement of Principles on Digital Data Management Michele Chittenden Acting Head, Academic Services chittend@queensu.ca (613) Jeff Moon on leave for 2016 Data Librarian Academic Director, Queen s Research Data Centre Ext Try out DMP Assistant tool: Francine Berish Geospatial Data Librarian francine.berish@queensu.ca Ext Alexandra Cooper Data Technician coopera@queensu.ca Ext Tatiana Zaraiskaya E Science librarian tz10@queensu.ca Ext /21/ /21/

20 11/21/