An Approach Finding the Right Tolerance Level for Clinical Data Acceptance

Transcription

1 Paper P024 An Approach Finding the Right Tolerance Level for Clinical Data Acceptance Karen Walker, Walker Consulting LLC, Chandler Arizona ABSTRACT Highly broadcasted zero tolerance initiatives for database error involving clinical trials are discussed ad infinitum among pharmaceutical research executives. Their cause is construed as 100% audit of efficacy variables and serious adverse events to prove the quality of the data used in the clinical trial is good. For some clinical trials studies databases are too large to perform an audit of all the efficacy and safety variables. Therefore, an audit proving a large database acceptable without a structured approach may be dubious at best. For instances such as this, it is reasonable to employ a systematic approach for finding the right level of tolerance for clinical data acceptance. This approach must test the following markers: tolerance limit, error rate, edit checks, manual review, sample sizes, and rank of sampling schemes; given the tolerance limit is the amount of meaningful data communicated properly. In this paper we will explore the use of a statistical inference to determine how good the error rate is with regard to specific tolerance limits. We will set an acceptable level of quality or tolerance limit that certifies the elements converted and stored in the database match all sources of information. INTRODUCTION The purpose of this presentation is to add rhyme or reason to the task of validating a clinical study database. For the scope of this discussion we will examine a typical collection of case report forms converted to the database by doing manual review. From a random sample of records; we will discover how many records match with the original source data, and how many records are flawed. With further evaluation of the data elements using a computerized edit check, we will learn more information about the data. Thus errors can be corrected without compromising data integrity. A test of hypothesis will be employed as a systematic way to tell us weather or not to accept the database. Finally I will demonstrate how to arrive at the same conclusion using as SAS procedure. TOLERANCE LIMIT The tolerance level is a value prespecified by clinical research management. They usually impose a zero tolerance level being under constraints of executive management. Clinical data management is lucky if the tolerance level is within the bounds of the rate of errors in the database. The error rate is calculated from the amount of meaningful data communicated properly. This covers all obvious mismatches with source data as well as data incongruencies. Consider a database having 100,000 records, after checking 1000, if only 10 errors are found the estimated error rate is 10 for every 1000 records checked. This error rate is called P hat (ˆp =0.01). The tolerance level is compared to the error rate (ˆp ) because to satisfy our objective the errors must be less. If the tolerance level is zero, it can be difficult to prove any error rate is lower. Therefore, it is reasonable to consider the tolerance level as the limit as the true error rate (p) approaches zero. Thus, tolerance level is better understood as the Tolerance Limit and will be subsequently denoted by P knot (p 0 0). My undergraduate calculus professor would express the tolerance limit in these terms: p 0 = LIM p P 0 ERROR RATE The number of data items converted to the database is checked against the original source information. The error rate is the ratio of mismatched observations over the total observations checked. In our paper the true error rate is denoted as (p). As a subject undergoes treatment during a clinical study, case report questionnaires are filled out. The number of discrepancies from what was written on the form and that converted to the database adds up as mismatches and increases the true error rate. For example, case report form INFORMED CONSENT contains information about a subject who has agreed to participate in a study. We should see the subject s initials, a unique identifier, and date the consent was given. If not all this data is converted to the database correctly it counts as an error.

2 ERROR RATE Also note there are other types of errors that arise from situations conducting the clinical trial. For instance, consider a randomized double blind clinical trial for a drug that generates stem cells. This study measures growth of certain cells based on dose group. We have a subject on this study that has experienced logarithmic growth of cells for that group. When an investigator mistakenly reported an adverse event a dose modification is done. Now on a different time line, and with a different dose the logarithmic growth is stopped, and dates on the CRF pages are inconsistent for that group. This is another type error, which will be addressed in my next paper. Case Report Book 01 INFORMED CONSENT 02 INCLUSION /EXCLUSION CRITERIA 03 MEDICAL HISTORY 04 PE / VITAL SIGNS 05 DOSE ADMINISTRATION 06 LABS 07 COLONY STIMULATING FACTOR 08 ADVERSE EVENTS 09 CONCOMITANT MEDICATIONS 10 STUDY COMPLETION/TERMINATION Clinical Database MANUAL REVIEW To insure the correctness of the data that has been converted to the database from the source, the data is audited. Every item from the source is compared to every item in the database manually. EDIT CHECKS Queries are small programs created to perform comparisons of source data to that which is in the database. This kind of edit checking serves well as it counts the number of errors no matter the size of the database. It will systematically alert clinical data management of errors. A simple edit check query routine may resemble the following: Proc SQL; Create table informedyes as Select subjects, 1 as errorflg From INFCNST Where INFCNDT ^=.; Quit; This query will execute SQL running on a UNIX server, under WINDOWS, or even MVS to name just a few. The routine asks the system to create a table called INFORMEDYES and write to the table all SUBJECTS from the INFCNST database table having no informed consent date. You will recall in a previous section of this paper, having no informed consent date represents a discrepancy and counts as an error. In order to trigger an alert we tag the records by setting the errorflg variable to 1. This table can be used to generate Alerts for clinical data management. A list of alerts is used to ascertain the information necessary to correct the error. This will reduce the true error rate p.

3 EDIT CHECKS A routine is created here using SAS to count the number of errors or alerts in the database from a table ALERTS. The SAS routine computes the sum of the errors, the number of records, and the standard error for a normal distribution. These variables are stored in a SAS table QUALITY for later use. proc means data=db.alerts noprint; var errorflg; output out=quality n=totalrec stderr=se sum=error min=minerror max=maxerror; SAMPLE SIZES If we checked each and every record in our clinical database we would know the absolute number of differences from the original source data in the converted database. The error rate (p) calculated is undisputable. Bear in mind that key clinical data for efficacy or safety should be checked in this manner, however for large databases it is not always feasible to do this. It s fair to make statistical inference based upon a carefully selected portion of observations, in other words from a sampling of the database. The error rate becomes the number of incorrect items over not the total possibilities but over the number checked (ˆp). APPROACH Our objective is to prove to management that the clinical database is of the highest quality possible. We will show this by presenting a test of hypothesis that supports acceptance of the database. Consider a database audited 100% of data elements with no other issues the test is proven if the following is true: H 0: ^p < p 0 -- Error rate is less than the Tolerance Limit ^p p 0 -- Error rate is greater than the Tolerance Limit (no good!) RANK OF SAMPLING SCHEMES Nothing so far has been determined about the distribution of errors for our clinical database. We do know that it is very likely that for a large database there will be errors. If not in exact data elements there will be errors in CRF casebooks. Since our errors are base upon the number of incorrect items over the number checked, close attention must be paid to the number checked. We know that to make a fair and unbiased statistical inference about the database as a whole we must give careful consideration to how we select samples from the database. Therefore, we have choices to make about the sampling distribution which will get us where we want to be. From the following sampling schemes: random samples, casebook samples There little influence upon the outcome of the hypothesis test. So let s move on to the statistical inference test.

4 MODIFIED APPROACH The strength of our approach depends upon how sure we are about the errors. For a complete 100% audit, the hypothesis test gives the strongest assurance the statistical inference is good. However, we need a systematic approach where it isn t required to do 100% audit to prove the statistical inference is good. We must now consider how likely it is for the inference to breakdown when using an estimated error rate. Factors affecting the precision of the hypothesis test other than the error rate and prespecified tolerance limit are: alpha (α) type I errors, or beta (β) type II errors. The alpha error is the chance that we could falsely reject the null hypothesis H 0. The beta error is the chance we could falsely accept the null hypothesis H 0. We know we can control type I and type II errors because the number of errors in a sample from a finite population is hypogenometirc, especially for situations where the sample size is large. Furthermore, if we choose a large enough sample size, and let the tolerance limit approach zero but not be exactly zero we can use a standard normal distribution which yields the following standard error : SE= p(1-p) N-n N N-1 Using a standard normal distribution we have full control of type I & type II errors: Type I = α = 0.5 Type II = β = 0.5 Using a zscore of (1- α)100% we have With a standard normal distribution we gain better control over the issues affecting our statistical inference. We have definitions for the standard error(se), alpha (α), beta (β), true error rate (p), calculated error rate (ˆp) and the tolerance limit (p 0 ). Therefore the null hypothesis that the true error rate is greater than or equal to the tolerance limit can be tested against the alternative hypothesis that the true error rate is less than the tolerance limit. Where p 0 has been redefined as: H 0 : p p 0 vs. H 1 : p< p 0 p 0 = p 0 ± SE*zscore

5 SAS PROCEDURE *************************************************************************** * Purpose : DATABASE ACCEPTANCE * * Description: This program pulls in database discrepancy counts and * * compares them to Acceptable Quality Standards per SOP * * inputs: db.quality.sas7bdat * * error = error flag * * totalrec = number of observations in DB * * phat = error rate * * se = Standard Error * * l_se = log of the standard error * * pknot = tolerance limit * options ls=132 ps=60 nodate nonumber libname db ~ ; *** input quality SAS dataset; %let aplha = 0.05 %let beta = 0.05 data pknot; input pknot ; cards; *** P knot approaches zero; data test; merge db.quality pknot; l_se = log(se); phat = error/totalrec; zscore= probit(1-&alpha); lower = pknot-se*zscore; loglw = exp(log(pknot)-1_se*zscore); if (phat < lower) or (phat < loglw) then h0 = Accept ; else if phat => pknot then h0 = Reject ; else h0 = Please take next observation ; *** Print out Error Rate and Decision ***; proc print data=test noobs label; var phat lower loglw ho; label phat = Error Rate for the Database lower = Lower Bound 1oglw = Log of Lower Bound ho = Result of Hypothesis Test ;

6 CONCLUSION We ve learned that validating a large database can be proven by using a statistical test of hypothesis if we consider that the chance of making an error or falsely reporting a clinical study database acceptable depends upon the true error rate, the tolerance limit, and the selection of the sample size. Make fully use of manual review, and edit checks to ascertain the true error rate p. However extreme caution must be taken and research must be done to understand the exact sampling size and for a large database the distribution must be normal. REFERENCES 1. Rondel RD, Varley SA, Webb C F. Clinical Data Management. 2 nd edition New York, NY: John Wiley and Sons; Kirk Paul Lafler, PROC: SQL Beyond the Basics Using SAS SAS Books by Users, Cary NC: Zhang, Drug Information Journal, Volume 38 Number 4, 2004 ISSN pages ACKNOWLEDGMENTS I wish to thank God above who keeps me safe. Thanks for giving us Christ CONTACT INFORMATION Karen Walker Walker Consulting LLC 6481 West Gary Drive Chandler Arizona kwalker77@sprintpcs.com SAS and all other SAS Institute Inc. product or services names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. indicates USA registration. Other brand and product names are trademarks of their respective companies.