Steering Committee Meeting

Transcription

1 Steering Committee Meeting To hear the meeting, you must call in Toll-free phone number: Access Code: For international call in numbers, please isit: Supported by the NIH grant 1U24 AI to the Uniersity of California, San Diego

2 Agenda Progress and Updates Brief reiew deelopment roadmap Delierables Oeriew Working Groups and Core De Updates DataMed release Announcements biocaddie All-Hands September 10 Summary of workshop Discussion 6 online + 28 in-person ealuations Supported by the NIH grant 1U24 AI to the Uniersity of California, San Diego

3 Deelopment Roadmap Personalized search Share/sae search results User account Link dataset to external resources PubMed Grants Search algorithm Boolean/adanced search Data repository search function Core Deelopment Roadmap Usability study User study Track user s action Ranking algorithm Refine search results based on user s selection Report from WG 8 on Ranking Data duplication problem Metadata management February 2016 Version 0.5 We are here June 2016 Version 1.0

4 Delierables Oeriew What s on track? What s Delayed? Coming up next Prototype 0.5 was successfully released Ingestion of datasets is proceeding on schedule Currently hae 21 ingested and planning 10+ more by the end of June Supplement Projects (CountEerything, Force 11 and Omics DI) are underway Pilot projects 4.3 only began recently due to contract issues WG 9 (End User Ealuation Criteria) Just started initing members Working Groups will begin later this year Draft of point paper is complete, paper will be published RFA for new pilot (innoated metadata generation and capture) will be released this summer Supported by the NIH grant 1U24 AI to the Uniersity of California, San Diego

5 WG2: Data ID Recommendation Completed Task/Description/Topic Identifiers Working Group 2 completed its actiities. Follow-on discussions occurring ia a number of separate groups Date Feb 2016 Data Citation Implementation Pilot (DCIP) Proide basic coordination between publishers, repositories and identifier / metadata serices for early adopters of data citation according to the JDDCP Identifiers Expert Group of technical experts on web-resolble biomedical data identification which is supporting harmonization of identifier resolution serices (e.g. CDL N2T, identifiers.org) Minid: A BD2K Minimal Identifier Pilot Minimal Viable Identifier (minid) infrastructure for uniquely and unambiguously identifying research data products throughout their life from data generation -> data release. Identifier implementation within the biocaddie prototype Initial work focusing on GEO/ArrayExpress and ICPSR/Dataerse on how to better represent primary repository as part of continuing work with 2 of the metadata model biocaddie Supplement BD2K Collaboration biocaddie Supported by the NIH grant 1U24 AI to the Uniersity of California, San Diego

6 WG3 Metadata Phase 2 (Sansone) Task/Description/Topic schema.org annotated DATS DATS 2 released DATS 2 open webinar Discussion with EG4 DCIP: repositories early adopters supplement (Philippe attended the) DCIP F2F meeting (Philippe attended the) Repository Workshop Tackle list of actions post-workshops Ø Follow-up on collaboration/alignment with DataCite Discussion with ELIXIR EXCELERATE WP5 Interoperability Dataset profiles task also on schema.org and bioschemas.org Alignment with OmicsDI Contribution to NIH Commons Data-Object Registry, FAIR-ness Metrics and subgroups Date completed completed completed completed completed ongoing ongoing ongoing ongoing ongoing Supported by the NIH grant 1U24 AI to the Uniersity of California, San Diego

7 WG4 Use Cases and Testing Benchmarks Chair: Hua Xu Details 13 members Completed Functioned from August December 2015 Delierables: User needs analysis report Usability tests Benchmarking dataset collection Testing queries Report: Supported by the NIH grant 1U24 AI to the Uniersity of California, San Diego 7

8 WG4 Use Cases and Testing Benchmarks Benchmark Dataset Generation Competency Questions grouped & prioritized Queries Instantiated (30-50) Run on 3-4 search engines Results pooled, Manually annotated Supported by the NIH grant 1U24 AI to the Uniersity of California, San Diego 8

9 WG4 Use Cases and Testing Benchmarks Usability Analysis Recommendations PHASE I Collect feedback from online users that use the DDI interface PHASE II Gather information from beta (test) users about researchers dataset seeking needs, strategies, and challenges PHASE III Determine if integrating seeral data sources in one place improes discoerability by comparing biocaddie to researchers standard approach to finding data Supported by the NIH grant 1U24 AI to the Uniersity of California, San Diego 9

10 WG5 Dataset Citation Metrics (Grethe & Sansone) Progress Date Identifying groups outside of biocaddie working on Citation Metrics (CASRAI/RDA, PLOS/CDL/DataOne, ) Initial conference call held with CASRAI/RDA group and PLOS/CDL/NISO group Action Items: Additional participants identified (THOR) Conference call to reiew PLOS/CDL/DataONE, NISO and THOR groups Agenda: Get lesson learned and/or recommendation they can share; also get DDI qualitatie and quantitatie metrics in their use cases Completed Completed Completed Continue to get feedback from external projects Ongoing CDT is working on incorporating citation metrics into Datamed prototype - a component of a number of metadata enhancers (e.g. NLP) that are slated to be incorporated Supported by the NIH grant 1U24 AI to the Uniersity of California, San Diego 10

11 WG6 Criteria for being included in the DDI (Ohno-Machado & Fore) Seenth Meeting occurred 4/22/16 Report being formulated by group members on Google Drie Final Meeting 5/06/16 Repository Workshop 6/23/16 Executie Summary: Repositories considered for inclusion in the BioCADDIE Prototype (DataMed) will be ealuated on fie dimensions. Key aspects of those dimensions are: Prominence: Priority should be gien to types of data that testers of the prototype will expect to find. Less prominent repositories should be considered if they introduce diersity. Sustainability: Repositories should actiely maintain their holdings and hae sustainable funding. Scope: Additional repository included in biocaddie should introduce diersity and new data. Quality: Data sets should be accompanied by sufficient metadata to make them findable, usable, and interoperable. Data formats should follow community standards Access: Data should be accessible to the scientific community. Metadata must be in a format usable by the biocaddie team for ingest to the DataMed index. Objects must be uniquely identified and web-resolable by persistent identifiers. Supported by the NIH grant 1U24 AI to the Uniersity of California, San Diego 11

12 WG7: Accessibility Metadata for Datasets (Alter) Delierables Metadata model and controlled ocabularies for data access that can be incorporated in the DDI core metadata. Recommendations for implementing the machine readable portion of the license Status Final draft agreed December 2015 Recommended items will be added to the biocaddie metadata schema by WG3 Completed

13 Accessibility Metadata ENTITY PROPERTY DEFINITION TYPE CARDINALITY Access Description of the access conditions for the object accesstypes A controlled ocabulary of access types. Multiple alues are allowed. string REQUIRE MENT LEVEL SHOULD authorization authentication license api A controlled ocabulary of of authorization types. A controlled ocabulary of authentication types. string string Text of a license or data use URI agreement. URI of an application programming URI interface for automated access to the object. SHOULD SHOULD MAY MAY granularity Granularity of Remote Serices (remoteserice or api) string Only when accesstype includes remoteserice. MAY 13

14 Accessibility Metadata: a recent publication Supported by the NIH grant 1U24 AI to the Uniersity of California, San Diego

15 Accessibility Metadata: a recent publication Supported by the NIH grant 1U24 AI to the Uniersity of California, San Diego

16 WG8 Ranking Search Results (Xu/Cohen) Task/Description/Topic GOAL Bring together information retrieal experts and biomedical domain experts with expertise in searching and using large datasets to identify priorities for the system users to facilitate aggregation and sorting of the results. Date ACTIVITIES/RESPONSIBILITIES Define the characteristics of a set of ranking metrics for releance. Examples might include number of citations to the dataset and/or initial publication or a PageRank-like metric, number of downloads (if aailable), completeness of metadata, and an estimate of the relatie importance of indexed metadata fields to users. DELIVERABLES Report that identifies the categories that are useful features (based on user needs) for ranking of search results for implementation in the prototype. ACTION 15 members in the WG First meeting: Discussed goals and delierables; Current ranking algorithm reiewed briefly 01/14 Second meeting: earlier WG outputs discussed (WG4), releance ranking algorithms discussed (language models etc), reiew families of ranking algorithms for next meeting 02/11 Third meeting: Various default ES plugin ranking algorithms discussed, others to be reiewed for discussion in next meeting Fourth meeting: ES plugin as well as state of the art ranking algorithms were reiewed. First draft report of the WG is ready. Fifth meeting: To collect feedback on draft report. Attending members needed another week s time 05/19 03/10 04/07 16

17 WG9 End User Ealuation Criteria (Xu/Johnson) Task/Description/Topic GOAL Engage with arious stakeholders (End Users) to enumerate primary characteristics that need to be used to ealuate DataMed. The primary end-users as defined by the biocaddie White Paper are the users looking for the datasets (biomedical researchers, clinicians, funders, data scientist, external search engines, and general public) and the data producers (Indiidual researchers, data repositories, centers producing large amounts of data, publishers, health care proiders and curators) ACTIVITIES/RESPONSIBILITIES The working group will reiew common ealuation criteria for a DDI with specific releance to the biomedical domain and the current usability testing plans by DataMed deelopment team to determine appropriate ealuation metrics for DataMed. DELIVERABLES Report defining the user-centric criteria for ealuation of DataMed that will be used to ealuate the prototype and refine the usability testing plans for DataMed. MEMBERS Being recruited ( s sent to potential members) Require recommendation of potential members from the following communities: Biomedical researchers Clinicians Funders Data scientist General public Data Repositories Centers Publishers Curators 6 MEMBERS AGREED TO PARTICIPATE, 2 DECLINED, WAITING ON RESPONSE FROM 3 MORE INVITEES SCHEDULING OF FIRST MEETING IS ONGOING Supported by the NIH grant 1U24 AI to the Uniersity of California, San Diego 17

18 Core De. Team - Progress Task Public ElasticSearch and Datamed Endpoint Public ElasticSearch EndPoint Documentation Deelopment ElasticSearch Endpoints Data Ingestion Pipeline Deelopment of index enhancers: synonyms & semantic relationships, metadata summarizer, Citation count External (web) serices and user interface for remote pipeline control Reconciliation of indices with WG3 metadata model 2 Working with Oxford group in reiewing metadata model 2 implemenation Data Set Ingestion Dataset ingestion process tested at three sites: UT-Health; UCSD-DBMI; and UCSD-CRBS Curators using GitHub for sharing transformation files Web serice has been deployed for checking transformations Reiew current data sources O8kjEikFpewpn9GWQ6K_4erJ9Bz8w/edit#gid= Working on updates to bring metadata in line with recently released metadata standard update Deelop curator guidelines Supported ( AI to the Uniersity of by the NIH grant 1U24 Remapping of preiously ingested California, repositories San Diegoto DATS additional repositories being ingested CEDAR Interactions: Finalizing biocaddie template Due Implemented Operational Operational Ongoing Ongoing Ongoing Ongoing 18

19 Core De. Team - Progress Task UI Deelopment Preliminary UI (0.1) demo UI (0.2) deployed (new design) DataMed (0.5) deployed NLP-based term identification and indexing Tutorials DataMed (1.0) deployed Improements/additional functions based on feedback (for 1.0): 1. Spelling correction 2. Breadcrumb for website naigation 3. Export to citation managers; copy citation to clipboard 4. Display all records from one repository 5. Select number of results per page 6. Data repository inclusion form 7. Collection features 8. Implement similar dataset function to more repositories Documentation : Source code, Help menu, Tutorials Data Duplication issue: 1. Strategies at the backend: additional field in index or Supported by the NIH grant 1U24 AI to the Uniersity of California, San Diego separate table for multiple relations? 2. Strategies for display in UI: Display duplicates together and remoe repeating instances Due 09/01 10/09 03/13 Ealuation Ongoing Ongoing 06/30/2016 Ongoing Discussed in Meeting 19

20 DataMed 1.0 release Released on June 30 th, 2016 Website : Please proide feedback Release announcement sent out to the NIH/BD2K community ia newsletter (thanks to Ron Margolis/Lisa Dunnenbacke) Release announcement also sent to workshop attendees and other members of the biocaddie community Please help to garner community feedback by sharing the information Supported by the NIH grant 1U24 AI to the Uniersity of California, San Diego 20

21 Core De. Team - Progress Task Due Ontology web serices: 1) Imported terminologies (6) and alidate them 2) Created UI-Browser for TS 3) Created auto complete feature 4) Integration to Scigraph API 5) Integrate terminology serer (UI) 6) Integrate terminology serer (Indexing) Benchmark dataset generation/system Ealuation: 23 repositories frozen for use 1) Four search engines: INDRI, Terrier, Lucene, Semantic Vectors 2) Specific queries from competency questions 3) Execute queries 4) Generate list of returned datasets 5) Annotate results: queries for training annotators, annotation guideline deelopment, IAA agreement Usability & User testing: 1) First set of 5 interiews Supported completed by the (for NIH grant Phase 1U24 II), AI Data Analysis to the Uniersity 2) Recruitment of users for 5 additional of interiews California, San (for Diego Phase II): 3 interiews completed, data analysis completed: a) We receied ery few negatie comments, most were positie. 65 positie, 58 challenge, 5 negatie b) We categorized the problem and challenge areas - most were in the search results page and had to do with display of data. c) Metadata issues oerall there were not any major issues with metadata. 3) User recruitment/feedback for Phase I 4) Planning of next Phase II/Phase III 08/31 09/30 10/20 11/20 1/19 Ongoing Annotation Ongoing Ongoing 02/22, 03/22 5/02 Ongoing Ongoing 21

22 Issues Aggregated in GitHub from those reported ia GitHub, Contact Us form and Questionnaire We hae started pushing issues generated ia the Contact Us form into GitHub using the GitHub API Total Issues 108 Number Open 59 Number Closed 72 Associated with 0.5 Number of Bugs Number of Enhancements Number Open 38 Number Closed 46 Number Open 5 Number Closed 10 Number Open 21 Number Closed 20 Number of Questions Number Open 7 Number Closed 9 Supported by the NIH grant 1U24 AI to the Uniersity of California, San Diego 22

23 Pilot Project Updates Feasibility study of Indexing Clinical Research Data Using HL7 FHIR (Dr. Guoqian, Mayo Clinic) w Completed Metadata Discoery and Integration to Support Hetergeneous Data using the OpenFurther Platform (Drs. Gouripeddi & Facelli, U of UT) w Expected completion: July 2016 Distributed data Discoery using GYM: Github, YAML, and Markdown (Dr. Mungall, Lawrence Berkeley National Laboratory) w Expected completion: October 2016 Supported by the NIH grant 1U24 AI to the Uniersity of California, San Diego

24 biocaddie All-Hands Meeting 2016 biocaddie all-hands meeting date and place has changed Scheduled for: w September 10, 2016 w in Dener Supported by the NIH grant 1U24 AI to the Uniersity of California, San Diego

25 Other Discussions Comments/Questions Next Steering Committee Meeting: August 4 Supported by the NIH grant 1U24 AI to the Uniersity of California, San Diego