Detecting Aberrant Data as an Indicator of Data Integrity Issues

Transcription

1 Detecting Aberrant Data as an Indicator of Data Integrity Issues Mark E Newton, Consultant QA Global Laboratory Informatics Eli Lilly and Company Indianapolis, IN newton_mark_e@lilly.com Overview Expectation of Oversight Options for Integrity Limits of Detection Ideas for Detection Two Levels: Governance and Operations Governance Metrics The Rise of Data Summary 1

2 Expectation of Oversight Regulators expect firms to routinely look for improper activities as part of review There was a lack of basic oversight by operations, quality unit, and site managers, as rewriting and destruction of original CGMP records was allowed to persist over a significant period without implementation of systems and controls to prevent data manipulation. Sun Pharmaceuticals 2014 May Failure of your quality unit to properly review production records and detect instances where testing was not performed to support your company s certifications on your COAs. Canton Labs 2014 Feb Your firm s failure to prevent, detect, and rectify the falsification of your GMP documentation is concerning. Posh Chemicals 2013 Aug Options for Integrity Reliable Prevent Technical Human Controls configured into system, to prevent data integrity issues Procedures, to address gaps in technical controls, or creation of manual data Inconsistent Detect Technical Human Queries, to highlight suspicious/undesired patterns or situations in electronic data Data review, to identify issues not detectable by queries, or manual data 2

3 Limits of Detection Most data integrity issues (slips, lapses, mistakes, violations) cannot be prevented due to type of data entry and limitations of technology Limitations of Detection: Some issues cannot be detected only observed (e.g. re do before save) Every hit on a report must be investigated A suite of reports is needed each one for a behavior pattern Reports must be automated manual reports require too many resources Manual processes have only data review for issue detection Bottom line: detection has limits, but is the best we have User Interface/design improvements will be next logical leap forward Ideas for Detection Enforcement Actions Look at Enforcements from regulators (e.g. Notice of Concern, 483, WL) Answer the Question: how would I detect that violation? Lifecycle Status Review Look at normal process path what is the status of everything? (materials, intermediates, specifications, batch, environment, equipment, CAPAs, etc ) Answer the Question: What situations create out of alignment statuses? Deviations (CAPA) Review events: could the right data have prevented this event? 3

4 Two Levels of Detection Governance Assess trends, or issues at enterprise level (all sites, operations) Measure degree to which detection metrics are used How often is each report used? How many records per report? Search criteria? Assess effectiveness of controls and detection How many issues were discovered? (what percent of all query hits?) What issues were missed? (discovered by inspection or luck) Modify detection queries (based on inputs) Finding no integrity issues has two reasons: (1) your data is good; (2) your detection is not adequate. Which is likely the correct response? Two Levels of Detection Operations Execute query, based on needs/data in operation Review query hits, to decide if integrity issue exists Keep metrics on queries (investigation time, reports used, etc) Capture other events with integrity issues (e.g inspections) Document/escalate issues to prevent recurrence Integrate integrity learnings into overall Quality system Track time consumed in investigating query hits, to measure efficiency of queries in finding issues. 4

5 Governance Metrics For Each Query (use of metrics) Site, # executions, # Hits (avg), search range (weeks) For Each Site (work volume) Material, # batches released For Each Issue Site, category, issue, method of discovery, action to prevent, repeat issue? Short Audit Trails (Short text entries) Any audit trail where user can enter a value Find records where Reason is fewer than ZZ characters User ID, Date/time, reason Why? Short Reason entries are less likely to offer adequate justification 5

6 Multiple Roles Any system Find records where a person has multiple security roles User ID Date/Time Role #1 Date/Time Role #2 Why? Multiple roles can lead to conflict of interest (admin + user) Batch Alignment Any manufacturing process Find records where a batch has supporting records NOT in expected status (e.g. a terminal status, like Approved) Site of Mfg, Material, Batch #, Status, Release Date System, current value, expected value Why? Supporting data not consistent with batch release status 6

7 Orphan Requests Any system Find records that should have been withdrawn or released Site, System, Record Name, Status, Age of Record Why? Indicates records to be investigated and (usually) withdrawn. Housekeeping is needed Aborted Runs Any data collection system that supports runs (chrom, dissolution) Find runs with Aborted status (# results and samples don t match) Run #, User ID, Date/Time, Position at abort, Reason Why? Aborts are suspicious can indicated undesirable results 7

8 Short Run Any data collection system that supports runs (chrom, dissolution) Find runs with a small number of injections (e.g. 1 2) Run #, Method ID, Sample ID(s), # Injections, Analyst ID Why? Users engaging in testing into compliance behaviors will inject solutions multiple times, often in short runs. Demo/Test/Trial Any system that matches results to sample description Look for files named demo%, test%, or trial% and/or samples with these name(s) Run #, Method ID, Sample ID(s), Analyst ID Why? Users might be testing into compliance by re labeling injections 8

9 Duplicate File Any system that retains data in flat files (not a database) List of files in a library (or set of libraries) with identical checksum, or with identical File Creation Dates. File name, Creation Date, Modified Date, Modified By NOTE: Checksum test might fail if Sample ID or other metadata is embedded in file Why? Indicates copying of data files, to potentially re use data on different samples The Rise of Data Data should be seen as equal to validation GAMP Records & Data Integrity Guide is a sister guide to GAMP 5 We verify that systems are fit for purpose When will we do this for data? (is it fit for purpose? ) 9

10 Summary Analytics can assist in quality oversight Analytics highlight conditions that merit review Each suspect record requires confirmation Some real issues will not appear Look at enforcement findings, deviations, and your own process for query ideas How would we detect that? What happened in the past? Data quality is equally as important as systems quality 10