Steering Committee Meeting

Transcription

1 Steering Committee Meeting To hear the meeting, you must call in Toll-free phone number: Access Code: For international call in numbers, please visit:

2 Agenda v v v v Pilot project presentation Progress report v v v v Brief review development roadmap Deliverables Overview Working Groups and Core Dev Updates Pilot project updates Announcement v Repository Workshop v biocaddie All-Hands 2016 Discussion

3 Development Roadmap Personalized search Share/save search results User account Link dataset to external resources PubMed Grants Search algorithm Boolean/advanced search Data repository search function Core Development Roadmap Usability study User study Track user s action Ranking algorithm Refine search results based on user s selection Report from WG 8 on Ranking Data duplication problem Metadata management February 2016 Version 0.5 We are here June 2016 Version 1.0

4 Deliverables Overview What s on track? What s Delayed? Coming up next Prototype 0.5 was successfully released Ingestion of datasets is proceeding on schedule Currently have 21 ingested and planning 10+ more by the end of June Supplement Projects (CountEverything, Force 11 and Omics DI) are underway Pilot projects 4.3 only began recently due to contract issues WG 9 (End User Evaluation Criteria) Just started inviting members Working Groups will begin later this year Draft of point paper is complete, paper will be published RFA for new pilot (innovated metadata generation and capture) will be released this summer

5 WG2: Data ID Recommendation Completed Final report being published to Zenodo

6 WG3 Metadata Phase 2 (Sansone) Outcome: Metadata specification v1.2: DATS model with several serializations and plan for use of schema.org Acting on feedback on metadata spec v1 received from: biocaddie Core Dev Team WG3 members and community, including Harold Solbrig and Guoqian Jiang leaders of the biocaddie Pilot Project 4.2 Hyeoneui Kim on how to map with the Observational Medical Outcomes Partnership (OMOP) model Yasset Perez-Riverol on mapping to the Heart2BD2K OmicsDI model done/ ongoing Reviewing and editing metadata spec v1: add metadata elements, vocabularies for licenses and terms of access defined by WG7 Accessibility Metadata for Datasets address suggestions received, mostly minor edits and a way to identify different dataset sources, e.g. primary, aggregators Tracking links to and interest received from other relevant efforts: W3C Healthcare Schema Vocabulary (ScheMed: health.schema.org extension) s chair Marc Twagirumukiza now member of WG3 done/ ongoing NIH and FNIH Accelerating Medicines Partnership program (AMP) Data Integration Working Group s lead Jon Green ELIXIR EXCELERATE Interoperability WP5 dataset profiles activities and discussion to use schema.org (also supported by the NIH Commons) Susanna presented biocaddie WG3 at ELIXIR AHM and will coordinate with Morris Swertz (lead of this WP5 task) now member of WG3. Release metadata spec v1.1 final open review, especially by WG3 and WG6-10, implementation by Core Dev Team and Harvester Pilot 1-15 Mar (1 April) Address any final comment and complete the metadata model release to Core Dev Team and Harvester Pilot only 30 Mar (delayed to end of April) Package documents, create schema, format and examples release of metadata spec v1.2, DAtaset Tag Suite model and JSON serialization 30 April Define steps and timelines to create a schema.org extension, under the bioschemas.org umbrella (also involving ELIXIR) tbd

7

8 WG4 Use Cases and Testing Benchmarks Chair: Hua Xu Details 13 members Completed Functioned from August December 2015 Deliverables: v User needs analysis report v Usability tests v Benchmarking dataset collection v Testing queries 8

9 WG4 Use Cases and Testing Benchmarks Benchmark Dataset Generation Competency Questions grouped & prioritized Queries Instantiated (30-50) Run on 3-4 search engines Results pooled, Manually annotated 9

10 WG4 Use Cases and Testing Benchmarks Usability Analysis Recommendations PHASE I Collect feedback from online users that use the DDI interface PHASE II Gather information from beta (test) users about researchers dataset seeking needs, strategies, and challenges PHASE III Determine if integrating several data sources in one place improves discoverability by comparing biocaddie to researchers standard approach to finding data 10

11 WG5 Dataset Citation Metrics (Grethe & Sansone) Progress Identifying groups outside of biocaddie working on Citation Metrics (CASRAI/RDA, PLOS/CDL/DataOne, ) Initial conference call held with CASRAI/RDA group and PLOS/CDL/NISO group Action Items: Additional participants identified (THOR) Conference call to review PLOS/CDL/DataONE, NISO and THOR groups Agenda: Get lesson learned and/or recommendation they can share; also get DDI qualitative and quantitative metrics in their use cases Follow-up meeting on hold until external projects progress. Begin finding date for next meeting Date Completed Completed Completed 11

12 WG6 Criteria for being included in the DDI (Ohno-Machado & Fore) Seventh Meeting occurred 4/22/16 Report being formulated by group members on Google Drive Final Meeting 5/06/16 Repository Workshop 6/23/16 Executive Summary: Repositories considered for inclusion in the BioCADDIE Prototype (DataMed) will be evaluated on five dimensions. Key aspects of those dimensions are: Prominence: Priority should be given to types of data that testers of the prototype will expect to find. Less prominent repositories should be considered if they introduce diversity. Sustainability: Repositories should actively maintain their holdings and have sustainable funding. Scope: Additional repository included in biocaddie should introduce diversity and new data. Quality: Data sets should be accompanied by sufficient metadata to make them findable, usable, and interoperable. Data formats should follow community standards Access: Data should be accessible to the scientific community. Metadata must be in a format usable by the biocaddie team for ingest to the DataMed index. Objects must be uniquely identified and web-resolvable by persistent identifiers. 12

13 WG7: Accessibility Metadata for Datasets (Alter) Deliverables v Metadata model and controlled vocabularies for data access that can be incorporated in the DDI core metadata. v Recommendations for implementing the machine readable portion of the license Status v Final draft agreed December 2015 v Recommended items will be added to the biocaddie metadata schema by WG3 Completed

14 Accessibility Metadata ENTITY PROPERTY DEFINITION TYPE CARDINALITY Access Description of the access conditions for the object accesstypes A controlled vocabulary of access types. Multiple values are allowed. string REQUIRE MENT LEVEL SHOULD authorization authentication license api A controlled vocabulary of of authorization types. A controlled vocabulary of authentication types. string string Text of a license or data use URI agreement. URI of an application programming URI interface for automated access to the object. SHOULD SHOULD MAY MAY granularity Granularity of Remote Services (remoteservice or api) string Only when accesstype includes remoteservice. MAY 14

15 Accessibility Metadata: a recent publication

16 Accessibility Metadata: a recent publication

17 Task/Description/Topic WG8 Ranking Search Results (Xu/Cohen) GOAL Bring together information retrieval experts and biomedical domain experts with expertise in searching and using large datasets to identify priorities for the system users to facilitate aggregation and sorting of the results. ACTIVITIES/RESPONSIBILITIES Define the characteristics of a set of ranking metrics for relevance. Examples might include number of citations to the dataset and/or initial publication or a PageRank-like metric, number of downloads (if available), completeness of metadata, and an estimate of the relative importance of indexed metadata fields to users. DELIVERABLES Report that identifies the categories that are useful features (based on user needs) for ranking of search results for implementation in the prototype. ACTION 15 members in the WG First meeting: Discussed goals and deliverables; Current ranking algorithm reviewed briefly Second meeting: earlier WG outputs discussed (WG4), relevance ranking algorithms discussed (language models etc), review families of ranking algorithms for next meeting Third meeting: Various default ES plugin ranking algorithms discussed, others to be reviewed for discussion in next meeting Fourth meeting: ES plugin as well as state of the art ranking algorithms were reviewed. First draft report of the WG is being generated and will be circulated. Date 01/14 02/11 03/10 04/07 Supported by the NIH grant 1U24 AI to the University of California, San Diego 17

18 WG9 End User Evaluation Criteria (Xu/Johnson) Task/Description/Topic GOAL Engage with various stakeholders (End Users) to enumerate primary characteristics that need to be used to evaluate DataMed. The primary end-users as defined by the biocaddie White Paper are the users looking for the datasets (biomedical researchers, clinicians, funders, data scientist, external search engines, and general public) and the data producers (Individual researchers, data repositories, centers producing large amounts of data, publishers, health care providers and curators) ACTIVITIES/RESPONSIBILITIES The working group will review common evaluation criteria for a DDI with specific relevance to the biomedical domain and the current usability testing plans by DataMed development team to determine appropriate evaluation metrics for DataMed. DELIVERABLES Report defining the user-centric criteria for evaluation of DataMed that will be used to evaluate the prototype and refine the usability testing plans for DataMed. Date MEMBERS Being recruited ( s sent to potential members) Require recommendation of potential members from the following communities: Biomedical researchers Clinicians Funders Data scientist General public Data Repositories Centers Publishers Curators Supported by the NIH grant 1U24 AI to the University of California, San Diego 18

19 Core Dev. Progress Task Public ElasticSearch Endpoint - Increasing VM size and update to latest ES version Public ElasticSearch EndPoint Documentation Development ElasticSearch Endpoints Data Ingestion Pipeline Development of index enhancers: synonyms & semantic relationships, metadata summarizer, Citation count External (web) services for remote pipeline control Reconciliation of indices with WG3 metadata model Providing feedback for updated metadata model Data Set Ingestion Dataset ingestion process tested at three sites: UT-Health; UCSD-DBMI; and UCSD-CRBS Curators using GitHub for sharing transformation files Web service has been deployed for checking transformations Review current data sources Continue working on data sources; Reconcile curator differences LINCS meeting Enhance LINCS metadata with entity API Metadata semantic mapping to map gene, protein, cell lines etc. using LINCS API Paper as proof of concept soon Due Upgrade Operational Operational 19

20 Core Dev. Progress Task UI Development Preliminary UI (v0.1) demo UI (v0.2) deployed (new design) DataMed (v0.5) deployed NLP-based term identification and indexing Tutorials Improvements/additional functions based on feedback (for v1.0): 1. Spelling correction 2. Breadcrumb for website navigation 3. Visualization 4. Export to citation managers; copy citation to clipboard 5. Search/sort on any column by clicking on column header 6. Display all records from one repository 7. Select number of results per page Documentation : Source code, Help menu, Tutorials Data Duplication issue Meeting with CEDAR 1. Metadata Submission Webform simple form on our website with link to CEDAR system for entry of detailed information 2. Generate biocaddie elements/template in CEDAR sysytem Collaborative research support Due 09/01 10/09 03/13 Evaluation Handling to be discussed 20

21 Issues Aggregated in GitHub from those reported via GitHub, Contact Us form and Questionnaire Total Issues 84 Number Open 48 Number Closed 36 Associated with v0.5 Number of Bugs Number of Enhancements Number of Questions Number Open 37 Number Closed 23 Number Open 2 Number Closed 3 Number Open 19 Number Closed 10 Number Open 5 Number Closed 3 21

22 Core Dev. Progress Task Ontology web services: 1) Imported terminologies (6) and validate them 2) Created UI-Browser for TS 3) Created auto complete feature 4) Integration to Scigraph API 5) Integrate terminology server (UI) 6) Integrate terminology server (Indexing) Benchmark dataset generation/system Evaluation: 23 repositories frozen for use 1) Four search engines: INDRI, Terrier, Lucene, Semantic Vectors 2) Specific queries from competency questions 3) Execute queries 4) Generate list of returned datasets 5) Annotate results Due 08/31 09/30 10/20 11/20 1/19 Annotation Usability & User testing: 1) First set of 5 interviews completed (for Phase II), Data Analysis 2) Recruitment of users for 5 additional interviews (for Phase II): 3 interviews completed, data analysis ongoing 3) User recruitment/feedback for Phase I 02/22, 03/22 22

23 Pilot Project Updates v v v Feasibility study of Indexing Clinical Research Data Using HL7 FHIR (Dr. Guoqian, Mayo Clinic) w Completed Metadata Discovery and Integration to Support Hetergeneous Data using the OpenFurther Platform (Drs. Gouripeddi & Facelli, U of UT) w Expected completion: July 2016 Distributed data Discovery using GYM: Github, YAML, and Markdown (Dr. Mungall, Lawrence Berkeley National Laboratory) w Expected completion: October 2016

24 BioCADDIE Workshop: Data Indexing for Data Providers v The biocaddie Project invites data repositories and databases to a workshop about adding items to the DataMed data discovery index. Presenters will explain the biocaddie metadata model, and participants will be given hands-on experience mapping their metadata into the DatA Tag Suite (DATS) for indexing. v June 23, 2016 v Estancia Hotel, La Jolla v Sign up to attend: w BioCADDIE homepage à Announcement à Repository workshop à Sign-up form (

25 BioCADDIE Workshop: Data Indexing for Data Providers DRAFT AGENDA Time% Agenda' Presenter' 9:00%)%9:20% Introduction%and%overview% Lucila% 9:20% %9:40% biocaddie s%use%cases%and%the% George% perspective%of%a%repository% 9:20%)%10:30% Presentation%of%the%metadata%model,% ingestion%process% [Susanna s%designees],%% Jeff,%Hua,%Hyeon)eui% 10:30% %10:45% Break% % 10:45% %11:15% Demonstration%of%the%DataMed% Hua,%Jeff%&%CDT% prototype,%next%steps% 11:15%)%12:15% Data%Tag%Suite%(DATS)%examples,% walkthrough%the%indexing%process%for%one% CDT,% George% repository% 12:15% %1:30% Lunch% % 1:30% %3:00% Hands)on%DATS%mapping%with%advice%from% all% the%biocaddie%team% 3:00% %4:00% Debrief%and%summary% all%!

26 BioCADDIE Workshop: Data Indexing for Data Providers 145 (potential) invitees, draft invitation message

27 biocaddie All-Hands Meeting v 2016 biocaddie all-hands meeting is scheduled for w September 18, 2016 w Atkinson Hall on UCSD campus.

28 Other Discussions Comments/Questions Next Steering Committee Meeting: June 2