Taming the SHREW. SDTM Heuristic Research and Evaluation Workshop

Transcription

1 Taming the SHREW SDTM Heuristic Research and Evaluation Workshop September 13, 2013 Carlo Radovsky

2 2 Overview Introductions The Backstory CDISC IntraChange History of a Rule The Challenge Discuss Amongst Ourselves

3 3 Introductions Carlo Radovsky Member of FDA/PhUSE DVQA (Data Validation and Quality Assessment) Working Group Member of CDISC SDS Team Anne Russotto PhUSE Steering Committee Liaison to DVQA Working Group Max Kanevsky Industry Co-Lead for FDA/PhUSE DVQA Working Group Solution Architect for FDA DataFit OpenCDISC

4 4 Backstory: CDISC IntraChange CDISC IntraChange 2013: July 30 August 1 CDISC teams meet internally as well as interact with other to address common challenges and goals Goal is to make progress on key 2013 deliverables, and to raise awareness of all teams on major issues and topics affecting us all Work together, learn from each other, solve problems, make progress, and identify new volunteers - Source: IntraChange 2013 Opening Plenary, Wayne Kubick

5 5 Backstory: CDISC IntraChange Cross-Team Session 5 (XT5): Validation Conformance Requirements Agenda Item 1: What are the obligations of CDISC teams when publishing new standards? Agenda Item 2: How do SDOs, sponsors, industry groups and vendors interact to work toward a better standards process for validation requirements? Agenda Item 3: Standards Governance Agenda Item 4: Standards validation Forum, a place where checks can be discussed currently takes place via PhUSE Data Quality Working Group - Source: XT5 - Validation Conformance Requirements Session Summary, Frank Newby

6 6 Backstory: CDISC IntraChange XT5 Item 1: What are the obligations of CDISC teams when publishing new standards? CDISC teams own/drive conformance requirements Conformance requirements should be called out explicitly within specification and IG documents Highlighted, linked, etc. A separate conformance document needs to be prepared (e.g., the ADaM, Define, and SEND teams material) - Source: XT5 - Validation Conformance Requirements Session Summary, Frank Newby

7 7 Backstory: CDISC IntraChange XT5 Item 2: How do SDOs, sponsors, industry groups and vendors interact to work toward a better standards process for validation requirements? FDA/PhUSE DVQA Working Group a common place for finding all of the Rules for a standard and will support the governance process Liaisons to be identified to participate and exchange information between working group and CDISC teams - Source: XT5 - Validation Conformance Requirements Session Summary, Frank Newby

8 8 Backstory: CDISC IntraChange XT5 Item 3: Standards Governance Work with PhUSE group to determine governance process and SMEs involved if deemed necessary For newly defined standards, identify where in the CDISC development process conformance requirements are established Determine relationship to data quality/submission rules and who owns those Any governance should be simple, straightforward - Source: XT5 - Validation Conformance Requirements Session Summary, Frank Newby

9 9 Backstory: CDISC IntraChange XT5 Item 4: Standards validation Forum Engage with PhUSE to determine how CDISC can best leverage the Data Quality Working Group wiki as a discussion forum - Source: XT5 - Validation Conformance Requirements Session Summary, Frank Newby

10 10 Backstory: CDISC IntraChange Obvious Gap: SDTM conformance requirements How to address?

11 11 Backstory: History of a Rule Original SDTM rule set developed for FDA JANUS Vendors implemented these rules, extending based on operational needs and additional understanding of model OpenCDISC started there, but more recently has been working with PhUSE and FDA to develop additional checks

12 12 Backstory: History of a Rule Rule Then & Now Attribute JANUS OpenCDISC Rule ID IR4000 SD0001 SDTM Version 3.1/ /3.1.2/3.1.3 (per ruleset) Effective Date Category Description Identifies domain table that has zero rows and therefore contains no data Presence Severity Low Warning Domain table should have at least one record Error Message No rows in domain table No records in data source Impact Study loaded and error noted in error report

13 13 Backstory: History of a Rule Rule Then & Now JANUS, ADaM identify failure criteria Attribute Useful, how about JANUS rule OpenCDISC version? OpenCDISC has success Rule ID IR4000 SD0001 SDTM Version 3.1/ /3.1.2/3.1.3 (per ruleset) Effective Date Category Description Identifies domain table that has zero rows and therefore contains no data Presence Severity Low Warning Domain table should have at least one record Error Message No rows in domain table No records in data source Impact Study loaded and error noted in error report

14 14 Backstory: History of a Rule Rule Then & Now Attribute JANUS OpenCDISC Rule ID IR4000 SD0001 JANUS terms not necessarily relevant OpenCDISC terms self evident SDTM Version 3.1/3.1.1 (Error, Warning, 3.1.1/3.1.2/3.1.3 Informational) High: Prevents JANUS loading (per ruleset) Medium: Effective Impacts Date reviewability Generally assess impact for end-user Low: Category May impact reviewability Both Warning and Error require resolution Presence or explanation to FDA Description Identifies domain table that has Distinct Domain but, how table operationally should have zero rows and therefore different? at least one record contains no data Severity Low Warning Error Message No rows in domain table No records in data source Impact Study ADaM loaded Team and did error not define noted severity in error Define report Team?

15 15 Backstory: History of a Rule CDISC References SDTM IG 3.1.2/3.1.3, Section 3.2, p. 17: In the event that no records are present in a dataset (e.g., a small PK study where no subjects took concomitant medications), the empty dataset should not be submitted and should not be described in the define.xml document. The annotated CRF will show the data that would have been submitted had data been received; it need not be re-annotated to indicate that no records exist. SDTM Metadata Submission Guide, Section 3.2, p. 8: In the event that no records are present in a dataset (e.g., a small study where no subjects took concomitant medications), the empty dataset should not be submitted or described in the define.xml. Please refer to Section for information regarding the annotations of such a dataset. SDTM Metadata Submission Guide, Section 4.1.1, p. 17: In the event that a dataset is not submitted because it is empty, the annotated CRF will show the annotations for the data that would have been submitted had data been received; it is not necessary to return to the CRF post data collection and change the annotation to indicate that no records were collected.

16 16 Backstory: History of a Rule CDISC References Rule 2 Rule 1 Seems Warning from end-user perspective Error from Conformance perspective? SDTM IG 3.1.2/3.1.3, Section 3.2, p. 17: In the event that no records are present in a dataset (e.g., a small PK study where no subjects took concomitant medications), the empty dataset should not be submitted and should not be described in the define.xml document. The annotated CRF will show the data that would have been submitted had data been received; it need not be re-annotated to indicate that no records exist. SDTM Metadata Submission Guide, Section 3.2, p. 8: Rule 3 (not machine-testable) In the event that no records are present in a dataset (e.g., a small study where no subjects took concomitant medications), the empty dataset should not be submitted or described in the define.xml. Please refer to Section for information regarding the annotations of such a dataset. SDTM Metadata Submission Guide, Section 4.1.1, p. 17: In the event that a dataset is not submitted because it is empty, the annotated CRF will show the annotations for the data that would have been submitted had data been received; it is not necessary to return to the CRF post data collection and change the annotation to indicate that no records were collected.

17 17 The Challenge: SDTM Conformance Rules Lessons Learned from ADaM and Define Teams Tremendous amount of work Lots of discussion/debate

18 18 The Challenge: SDTM Conformance Rules What is different (maybe?) Existing references JANUS CAB Validation Project OpenCDISC Others? Upcoming FDA Technical Conformance document? Broader/Deeper SME community

19 Discuss Amongst Ourselves 19