Executive Committee Meeting

Transcription

1 Executive Committee Meeting To hear the meeting, you must call in Toll-free phone number: Access Code: For international call in numbers, please visit: Supported by the NIH grant 1U24 AI to the University of California, San Diego

2 Agenda v biocaddie all hands agenda v Discussion Points v Core Dev Updates v Working Groups Updates v Upcoming Deliverables Supported by the NIH grant 1U24 AI to the University of California, San Diego

3 All- Hands Meeting Supported by the NIH grant 1U24 AI to the University of California, San Diego

4 Core Dev. Team Presentation for the biocaddie AHM 90 minutes total Presentation Presentation Ti5me Person Introduction to DataMed: 1) Overall architecture 2) Demonstration of DataMed Prototype 3) PP integration 4) Feedback mechanism 15 min Hua Ingestion & Indexing pipeline: 1) Ingestion pipeline 2) Indexing pipeline User Interface: 1) Technical details 2) Features in v1.0 System Evaluation: 1) Evaluation Plan 2) Benchmark Dataset Usability & User testing: 1) Phase I survey 2) Phase II completed results + plans for next one 3) Phase III Plan outline 15 min Supported by Jeff the NIH grant 1U24 15 AI min Hua to the University of 15 California, min Trevor San Diego 15 min Todd 2016 biocaddie DataSet Retrieval Challenge: 1) CFP 2) Queries/Benchmark Datasets from WG3, WG4 & CDT 15 min Hua 4

5 Year 3 v 1:45 2:45pm is biocaddie deliverables Year 3 w Who should lead this discussion? Or will it be more of a slideshow presentation of our planned deliverables? w Are there specific items we need to cover? Supported by the NIH grant 1U24 AI to the University of California, San Diego

6 Moderated discussion at lunch v Ron would like to have a minute moderated discussion during the lunch hour but wants to solicit ideas from the EC as to the topic Supported by the NIH grant 1U24 AI to the University of California, San Diego

7 NLP Challenge 2016 biocaddie DataSet Retrieval Challenge Timeline Registration: begins September 9, 2016 DataSets and Sample Queries Release: September 16, 2016 Test Queries Release: October 21, 2016 System Outputs Due: October 28, 2016 Workshop: TBD Supported by the NIH grant 1U24 AI to the University of California, San Diego

8 Pilot Project Updates v v v Feasibility study of Indexing Clinical Research Data Using HL7FHIR - Dr. Guoqian Mayo w Expected completion: Completed! Metadata Discovery and Integration to Support Hetergeneous Data using the OpenFurther Platform Drs. Gouripeddi & Facelli Utah w Expected completion: Completed except for literature writeups. Distributed data discovery using GYM: GitHub, YAML and Markdown Dr. Mungall LBNL w Expected completion: October 2016 Supported by the NIH grant 1U24 AI to the University of California, San Diego

9 Core Dev. Team - Progress Task Public ElasticSearch and Datamed Endpoint Public ElasticSearch EndPoint Documentation Development ElasticSearch Endpoints Data Ingestion Pipeline Development of index enhancers: synonyms & semantic relationships, metadata summarizer, Citation count External (web) services and user interface for remote pipeline control Reconciliation of indices with WG3 metadata model v2 Working with Oxford group in reviewing metadata model v2 implementation Generating review documents utilizing full pipeline Data Set Ingestion Curators using GitHub for sharing transformation Supporte files Web service has been deployed for checking transformations d by the Review current data sources NIH grant 1U24 O8kjEikFpewpn9GWQ6K_4erJ9Bz8w/edit#gid= AI Working on updates to bring metadata in line with -01 recently to released metadata standard update the Develop curator guidelines ( Universit y of Remapping of previously ingested repositories to DATS 2.0 Californi Documentation of ingestion pipeline a, San Diego CEDAR Interactions: Finalizing biocaddie template using CEDAR tools Due Implemented Operational Operational 9

10 Core Dev. Team - Progress Task UI Development Preliminary UI (v0.1) demo UI (v0.2) deployed (new design) DataMed (v0.5) deployed Tutorials DataMed (v1.0) deployed Improvements/additional functions based on feedback (for v1.5 or 2.0): 1. NLP-based term identification and indexing 2. Improved response time 3. Bulk download of indices 4. Most accessed datasets 5. User activity tracking system 6. Sorting on publication date and authors 7. Visualization 8. Primary repository vs aggregators 9. Linkout 10.Duplicate dataset display 11.More as we get more feedback. Supported by the NIH grant 1U24 AI to the University of California, San Diego Documentation : Source code, overviews of major modules, Help menu, Tutorials Google Analytics for both datamed.orghttps://analytics.google.com/analytics/web/#home/a w p / 2016 biocaddie DataSet Retrieval Challenge CFP : A5RpkJaxYuqcleb0YG3ymwvsNsWpg7D9NA3cpB7ts/edit Due 09/01 10/09 03/13 06/30/

11 Core Dev. Team - Progress Task Ontology web services: 1) Integrate terminology server (UI) 2) Integrate terminology server (Indexing) Benchmark dataset generation/system Evaluation: 23 repositories frozen for use 1) Four search engines: INDRI, Terrier, Lucene, Semantic Vectors 2) Specific queries from competency questions 3) Execute queries 4) Generate list of returned datasets 5) Annotate results: queries for training annotators, annotation guideline development, IAA agreement Supported by Usability & User testing: the NIH grant 1) First set of 5 interviews completed (for Phase II), Data Analysis 1U24 AI ) Recruitment of users for 5 additional interviews (for Phase II): 3 interviews completed, data analysis to the completed: University of a) We received very few negative comments, most California, were positive. 65 positive, 58 challenge, 5 negative San Diego b) We categorized the problem and challenge areas - most were in the search results page and had to do with display of data. c) Metadata issues overall there were not any major issues with metadata. 3) User recruitment/feedback for Phase I: 4) Planning of next Phase II/Phase III Pilot project integration: 1) Guoqian Jiang : Feasibility Study of Indexing Clinical Research Data Using HL7 FHIR 2) Ram Gouripeddi: Metadata Discovery and Integration to Support Repurposing of Heterogeneous Data using the OpenFurther Platform Due 1/19 Annotation 02/22, 03/22 5/02 08/09 09/27 11

12 Issues Total Issues 142 Number Open 63 Number Closed 79 Associated with v1.0 Usability Issues Associated with v0.5 Number of Bugs Number of Enhancements Number of Questions Number Open 11 Number Closed 1 Number Open 28 Number Closed 4 Number Open 38 Number Closed 46 Number Open 6 Number Closed 11 Number Open 25 Number Closed 22 Number Open 8 Number Closed 10 Supported by the NIH grant 1U24 AI to the University of California, San Diego 12

13 Working Groups Updates v Working Group 2 Jeff v Working Group 3 Susanna v Working Group 4 Hua v Working Group 5 - Jeff v Working Group 6+10 Lucila/Ian/George v Working Group 7 George v Working Group 8 Hua v Working Group 9 Hua v Working Group 11- George Supported by the NIH grant 1U24 AI to the University of California, San Diego

14 WG2 Identifiers (Grethe) Completed Task/Description/Topic Identifiers Working Group 2 completed its activities. Follow-on discussions occurring via a number of separate groups Date Feb 2016 Data Citation Implementation Pilot (DCIP) Provide basic coordination between publishers, repositories and identifier / metadata services for early adopters of data citation according to the JDDCP Identifiers Expert Group of technical experts on web-resolvble biomedical data identification which is supporting harmonization of identifier resolution services (e.g. CDL N2T, identifiers.org) Minid: A BD2K Minimal Identifier Pilot Minimal Viable Identifier (minid) infrastructure for uniquely and unambiguously identifying research data products throughout their life from data generation -> data release. Identifier implementation within the biocaddie prototype Initial work focusing on GEO/ArrayExpress and ICPSR/Dataverse on how to better represent primary repository as part of continuing work with v2 of the metadata model biocaddie Supplement BD2K Collaboration biocaddie 14

15 WG3 Metadata Phase 2 (Sansone) Main community phases completed; work carry out internally, interacting with selected external groups as needed Task/Description/Topic Date schema.org annotated DATS DATS v2 released DATS v2 open webinar Discussion with DCIP, data citation admin supplement (Force11) Attending DCIP F2F and Repository workshops Tackle list of actions to review and refine DATS (post-workshops) Follow-up with DataCite (Martin Fenner), working to export in DATS format (new task) Create list/taxonomy of data type terms: need to classify datasets in DataMed completed completed completed completed TBC Discussion with ELIXIR EXCELERATE WP5 Interoperability also on schema.org and bioschemas.org Mapping DATS v2 to with OmicsDI to ensure they can also work to export to DataMed Contribution to NIH Commons Data-Object Registry, FAIR-ness Metrics and subgroups Supported by the NIH grant 1U24 AI to the University of California, San Diego

16 WG4 Use Cases and Testing Benchmarks (Xu) - Details 13 members Completed Functioned from August December 2015 Deliverables: v User needs analysis report v Usability tests v Benchmarking dataset collection v Testing queries Report: Final auwapavw/edit Supported by the NIH grant 1U24 AI to the University of California, San Diego 16

17 Progress WG5 Dataset Citation Metrics (Grethe & Sansone) Identifying groups outside of biocaddie working on Citation Metrics (CASRAI/RDA, PLOS/CDL/DataOne, ) Initial conference call held with CASRAI/RDA group and PLOS/CDL/NISO group Action Items: Additional participants identified (THOR) Conference call to review PLOS/CDL/DataONE, NISO and THOR groups Agenda: Get lesson learned and/or recommendation they can share; also get DDI qualitative and quantitative metrics in their use cases Continue to get feedback from external projects Completed Date Completed Completed Completed CDT is working on incorporating citation metrics into Datamed prototype - Citation count from citation DB implemented (e.g. PDB citations from Pilot Project) and being tested as part of full Supported by the NIH grant 1U24 AI to the University of California, San Diego pipeline run for DATS reconciliation finalization 17

18 WG6 Criteria for being included in the DDI (Ohno-Machado & Fore) Completed Last meeting occurred 5/06/16 Report being formulated by group members on Google Drive Repository Workshop 6/23/16 Executive Summary: Repositories considered for inclusion in the BioCADDIE Prototype (DataMed) will be evaluated on five dimensions. Key aspects of those dimensions are: Prominence: Priority should be given to types of data that testers of the prototype will expect to find. Less prominent repositories should be considered if they introduce diversity. Sustainability: Repositories should actively maintain their holdings and have sustainable funding. Scope: Additional repository included in biocaddie should introduce diversity and new data. Quality: Data sets should be accompanied by sufficient metadata to make them findable, usable, and interoperable. Data formats should follow community standards Access: Data should be accessible to the scientific community. Metadata must be in a format usable by the biocaddie team for ingest to the DataMed index. Objects must be uniquely identified and web-resolvable by persistent identifiers. Supported by the NIH grant 1U24 AI to the University of California, San Diego 18

19 WG7: Accessibility Metadata for Datasets (Alter) Deliverables v Metadata model and controlled vocabularies for data access that can be incorporated in the DDI core metadata. v Recommendations for implementing the machine readable portion of the license Status v Final draft agreed December 2015 v Recommended items will be added to the biocaddie metadata schema by WG3 Completed

20 WG8 Ranking Search Results (Xu/Cohen) Completed Task/Description/Topic GOAL Bring together information retrieval experts and biomedical domain experts with expertise in searching and using large datasets to identify priorities for the system users to facilitate aggregation and sorting of the results. Date ACTIVITIES/RESPONSIBILITIES Define the characteristics of a set of ranking metrics for relevance. Examples might include number of citations to the dataset and/or initial publication or a PageRank-like metric, number of downloads (if available), completeness of metadata, and an estimate of the relative importance of indexed metadata fields to users. DELIVERABLES Report that identifies the categories that are useful features (based on user needs) for ranking of search results for implementation in the prototype. ACTION 15 members in the WG First meeting: Discussed goals and deliverables; Current ranking algorithm reviewed briefly 01/14 Second meeting: earlier WG outputs discussed (WG4), relevance ranking algorithms discussed (language models etc), review families of ranking algorithms for next meeting Third meeting: Various default ES plugin ranking algorithms discussed, others to be reviewed for discussion in next meeting Fourth meeting: ES plugin as well as state of the art ranking algorithms were reviewed. First draft report of the WG is ready. Fifth meeting: To collect feedback on draft report. Attending members needed another week s time 05/19 Final report generated and published on the WG homepage 02/11 03/10 04/07 20

21 WG9 End User Evaluation Criteria (Xu/Johnson) Task/Description/Topic GOAL Engage with various stakeholders (End Users) to enumerate primary characteristics that need to be used to evaluate DataMed. The primary end-users as defined by the biocaddie White Paper are the users looking for the datasets (biomedical researchers, clinicians, funders, data scientist, external search engines, and general public) and the data producers (Individual researchers, data repositories, centers producing large amounts of data, publishers, health care providers and curators) ACTIVITIES/RESPONSIBILITIES The working group will review common evaluation criteria for a DDI with specific relevance to the biomedical domain and the current usability testing plans by DataMed development team to determine appropriate evaluation metrics for DataMed. DELIVERABLES Report defining the user-centric criteria for evaluation of DataMed that will be used to evaluate the prototype and refine the usability testing plans for DataMed. DATE FIRST MEETING: Introduction to the WG Discussion of the CDT user testing plans SECOND MEETING: Survey: Summative testing draft outline : Goal: Compare Usability of biocaddie to a standard control Proposed control: Hyperlinks of all repositories indexed by biocaddie (being discussed) Tasks: 4 tasks from benchmarking effort (being discussed) THIRD MEETING: Summative testing draft outline discussion Users for the testing JULY 22 AUGUST 24 SEPTEMBER 7 Supported by the NIH grant 1U24 AI to the University of California, San Diego 21

22 WG11: Repository Contact Workflow v Workflow process is being implemented w Responsibility for repositories assigned in Github v Exec Committee advice needed on process for evaluating repositories w Is the Repository Evaluation form sufficient? w What levels of assistance will be offered? w How will decisions be made? 22

23 WG12: Standard-Driven Curation Best Practices v Scheduled to begin Sept v White paper says: w A working group will be devoted to the issue of data curation, determining the minimal amount of curation that is necessary and understanding the best mechanisms for ensuring that curation takes place. w A working group in Year 2 will discuss criteria for adding repositories to the prototype to be sure that we are covering all of our use cases. We expect to prioritize NIH funded repositories, and the working group will consider also community repositories and generic repositories (like Dryad and Figshare), and potentially those associated to data journals (e.g., GigaScience and Scientific Data). We define community repositories as those databases that accept data contributions from a specific research area, e.g., PDB and ICPSR. A broader discussion of data curation requirements will occur in a working group scheduled for Year 3. 23

24 Deliverables Time Deliverables Core team WG/PP Y2-Q4 June 2016 to August Data submission tool for individual researchers to submit metadata to the DDI - Prototype v.1 User experience evaluation - RFA for Pilot/Challenge on innovative methods of metadata generation and capture - Outreach meeting: Repository operators (WG11) Y3 Q1 Sept 2016 to Nov Pilot Project evaluation - Evaluation by users - Demonstration of prototype in professional meetings - Standard- driven curation best practices (WG12) - Pilot/Challenge on innovative methods of metadata generation and capture (PP 6) Supported by the NIH grant 1U24 AI to the University of California, San Diego 24