Assessing superiority/futility in a clinical trial: from multiplicity to simplicity with SAS

Transcription

1 PharmaSUG2010 Paper SP10 Assessing superiority/futility in a clinical trial: from multiplicity to simplicity with SAS Phil d Almada, Duke Clinical Research Institute (DCRI), Durham, NC Laura Aberle, Duke Clinical Research Institute (DCRI), Durham, NC Abstract SAS macroprocessing was implemented to deal with a simple question that was demonstrated to be more extensive when pursued to completeness. That is, given the sample size, what proportion of successful events would need to be arrived at to consider a study drug as superior or the study as futile at some given significance level. Using the procedure that was developed, the consultant was able to provide the client with, not just a variety of scenarios by which to formulate clinical trial judgements but an exhaustive and complete review based upon the initial request. With this procedure, furthermore, the client was able to gain a sense of the scope of the enormity of a proper assessment of the request without the of SAS, and within a reasonable amount of time. In addition, upon increasing one single parameter in the client request, the exponentially escalating scale of affordable information seemed to provide the client with a sense of satisfaction about the particular pursuit since no further requests were submitted. Introduction During the initial stages of a clinical trial at the DCRI, the trial Sponsor became concerned with assessing the superiority/futility of the study drug under various scenarios. A hypothetical number of subjects per treatment arm was proposed and a number of proportions were tested with attained significance levels to be inspected. Initially, the manner of approach was manual and a sampling of proportions based on the given sample size were tested and submitted to the Sponsor. Since the effort was not exhaustive, and could not be within a reasonable amount of time, the Sponsor was challenged to be satisfied with the parsimonious nature of the initial response. This was evidenced by further inquiry by the Sponsor to assess even more scenarios and initial efforts were made by the DCRI to programmatically test a broader range of proportions. In concert with escalating requests, the programming became more challenging since the need was to address these requests within a favourable response time, and in a manner that would return a complete analysis, showing statistical test results, based on the initial request from the Sponsor. The purpose of this paper is to demonstrate the power of SAS macroprocessing to provide such a response, and yet even an answer to the question, Is this drug still worth the time and resources to test? Method The simple case The initial scenario requested by the Sponsor was to assess a sample size of 30 subjects in each treatment group, specifically, one group being administered the study drug and the other, a placebo. Consequently, the initial programming approach was to draw some realistic proportions of successes under each treatment arm and test for association. Thus, data sets were generated where there were simulated the number (wt) of remissions (p = 1) under the presence of placebo (t = 0) and under the administration of study drug (t = 1). One SAS data step scenario would be represented as: data test1; p = 0; t = 0; wt = 25; output; p = 0; t = 1; wt = 10; output; p = 1; t = 0; wt = 5; output; p = 1; t = 1; wt = 20; output; The SAS code for the subsequent test of association would look like the following: proc freq data=test1; weight wt; table p * t / chisq;

2 where the CHISQ option in the TABLE statement provides for the chi square test of association. A series of SAS data steps then would be constructed to simulate various proportions followed by tests of association in corresponding SAS procedure steps until some informative test outcome(s) was/were achieved. Clearly, this could be a labour-intensive task since there were, at minimum, 30 x 30 possible combinations of weights, that is, x 2 tables and 900 tests of association to be simulated in 900 data steps and 900 procedure steps. Thus, given that the maximum number of subjects per treatment arm was identified, the most logical tool to accomplish all objectives and to be the least labour-intensive was SAS macroprocessing. Code upgrade for labour downgrade stage 1, data generation First, the correct maximum number of simulations was verified as 31 x 31 to accommodate for the scenarios of no remissions under either treatment arm and total remissions under at least one treatment arm. Thus, there would actually be 961 simulations. Further, the simulated weights could be conceived as a 31 x 31 grand matrix composed of elements or components that are each 2 x 2 matrices where any such 2 x 2 component matrix could be indexed by the row-column coordinates of the grand matrix, i,j where i = 1,, 31, and j = 1,, 31. Next, an algorithm was determined that would simulate the series representing changes in the four cell counts or weights and to begin with the scenario of no remissions in both treatment arms and ending with the scenario of total remissions in both treatment arms, and including the two scenarios where there are total remissions in one treatment arm and none in the other. The following code represents the modified data step: a macro program using two macro variables, row and col, that define the row-column position of any 2 x 2 component matrix in the 31 x 31 grand matrix. Any weight (wt) is, therefore, necessarily a function of the position coordinates of the corresponding 2 x 2 component matrix. %do row = 1 %to 31; %do col = 1 %to 31; data ds&row._&col; p = 0; t = 0; wt = &row - 1; output; p = 0; t = 1; wt = 30 - (&row - 1); output; p = 1; t = 0; wt = &col - 1; output; p = 1; t = 1; wt = 30 - (&col - 1); output; To identify each simulated data set that corresponds to a 2 x 2 component matrix, the macro variables, row and col that drive the cell weight computations were d in the data set nomenclature illustrated above. The corresponding procedure steps for the tests of association would be necessarily modified only in the data set name, as proc freq data= ds&row._&col; weight wt; table p * t / chisq; Clearly, 961 statistical tests present a formidable task for review of p values. However, with the ability now achieved to programmatically simulate all possible scenarios based upon 30 subjects in each treatment arm, the ability to assess the outcome of statistical tests must keep pace otherwise the labour saved to construct outcome scenarios would be lost to labour for review of results. Thus, a suitable means of presenting the simulated 2 x 2 tables with corresponding tests would complete the pursuit of decreasing the labour effort. Furthermore, given the completeness of the simulation, the expected cell size of some cells amongst the 961 sets of 2 x 2 tables would highly likely be less than 5. Thus, Fisher s exact test would be the preferred test in these cases and these cases would need to be identified. These features needed to be added and automated, if possible. Code upgrade for labour downgrade stage 2, data presentation To capture the results of Fisher s exact test, the Output Delivery System or ODS was implemented. Two output data sets were generated using two ODS output options, chisq and fishersexact, in each implementation of the FREQ procedure step: 961 sets of two output data sets. Since macroprocessing was already in place, each of these 961 pairs of output data sets was correspondingly indexed as follows: ods output chisq = CS&row._&col ; ods output fishersexact = fx&row._&col ;

3 Further, the FREQ procedure was modified in the TABLE statement with the out= option to generate a procedure output data set containing one record per 2 x 2 cell, with the outexpect option to include the expected cell counts in the procedure output data set, as follows: proc freq data= ds&row._&col; weight wt; table p * t / chisq out=sds&row._&col outexpect; Each procedure output data set containing the cell counts and the expected values was manipulated by the addition to two index variables. One of these was d to count the number of expected values less than 5 in the variable clt5. Such expected values were indicated as the values of the default variable expected. The other index variable was d to count the number of nonzero values in the variable cnzc. When there were counted, in any one output data set, one expected value less than 5 and three nonzero values, or two expected values less than 5 and four nonzero values, then a macro variable, FEndx, was defined that was to index the of Fisher s exact test. Thus, if not lastrec then do; call symput ('FEndx',0); if lastrec then do; if clt5 = 1 & cnzc = 3 clt5 = 2 & cnzc = 4 then do; call symput ('FEndx',1); It should be mentioned that empty cells from the FREQ procedure would default to missing in the procedure output data set so that the counting of nonmissing values in the procedure output data set is equivalent to the counting of nonzero cells. There were, therefore, 961 evaluations for expected values and nonzero cells in 961 data sets. The data set variable, FE1, representing the index to Fisher s exact test was derived from the assigned macro variable, FEndx, described in the preceding paragraph. Thus, %if &FEndx = 1 %then FE1 = "1";; Next, the final presentation of results needed to be simplified and to include the i th j th index that indicates when Fisher s exact test is to be implemented. The i th j th modified procedure output data set, the i th j th ODS output data set defined by the ODS output option chisq, and the i th j th ODS output data set defined by the ODS output option fishersexact, were each complemented with a merge variable defined as the i th j th cell position corresponding to that i th j th 2 x 2 component matrix in the 31 x 31 grand matrix. Such a merge variable, cell, could be defined as length cell $ 5; cell = "&row.:&col"; Thus, for each 2 x 2 simulated data set, the two corresponding ODS output data sets were modified and merged with the corresponding procedure output data set to yield a data set of one record. The two data sets derived from the two extreme scenarios of no remissions and total remissions were excluded leaving 959 pairs of data set modifications and 959 merges of data sets resulting in 959 one-record data sets. The cells of the 31 x 31 grand matrix representing the two excluded data sets were i,j = 1,1 and i,j = 31,31. Therefore, the processing for these 959 sets of modifications and merges was controlled by the following SAS code: %if (&row = 1 and &col > 1) or (&row > 1 and &row < 31) or (&row = 31 and &col < 31) %then %do; Finally, the 959 single-record data sets cells were concatenated with two dummy records representing the grand matrix positions 1,1 and 31,31. It should be noted that there were original table cells from the procedure output data set that contained a cell count of 0. These would default to missing in this final concatenated data set but these were corrected to 0 by direct coding. The final step was to identify whether superiority or futility had been arrived at in any of the 959 scenarios. The significance level had been previously decided upon as part of the study design and was d to compare the attained significance levels in two-tailed tests to detect nondirectional superiority or futility. A simple code

4 construction using the IF THEN cla was implemented to examine all chi-square p values, prob, including where appropriate, those p values from Fisher s exact test, tailt. Thus, encoding superiority and futility in the variable rule with futility as rule = F and superiority as rule = S, the following code is sufficient: The complex case if fe1 = "1" then do; if tailt > then rule = "F"; else if. < tailt < then rule = "S"; else if fe1 = " " then do; if prob > then rule = "F"; else if. < prob < then rule = "S"; Now that x 2 tables, chi-square tests, and Fisher s exact tests could be represented in 959 rows in one SAS data set and indexed as to the of Fisher s exact test, the Sponsor considered and submitted another simple request: a change in the sample size per treatment arm. Increasing the treatment-arm sample size to 60, the grand matrix then becomes 61 x 61 with 3,721 possible rows in a final data set. A sample size of 90 per treatment arm would produce a final data set of 8,281 rows. Anticipating a series of such changes and considering the logistics of implementing such changes in the program, the superiority of SAS macroprocessing was again implemented in the perceived futility of defining the ultimate answer. The data generation code was modified with the macro variable n representing the sample size per treatment arm as, %do row = 1 %to %eval(&n+1); %do col = 1 %to %eval(&n+1); data ds&row._&col; p = 0; t = 0; wt = &row - 1; output; p = 0; t = 1; wt = &n - (&row - 1); output; p = 1; t = 0; wt = &col - 1; output; p = 1; t = 1; wt = &n - (&col - 1); output; The code for processing of all data sets excluding i,j = 1,1 and i,j = n+1,n+1 was also modified with the macro variable n as follows: %if (&row = 1 and &col > 1) or (&row > 1 and &row < %eval(&n+1)) or (&row = %eval(&n+1) and &col < %eval(&n+1)) %then %do; Results The following is a sample of nine consecutive rows of SAS output printed from the final data set of results that were generated from the simulation of 30 subjects per treatment arm. The column headings are the final SAS variable names where cell identifies the position in the grand matrix to which the particular row of results applies. In particular, the rows in the sample output represent the last three columns of row one of the grand matrix, and the first six columns of row two of the grand matrix. The variables count0 and count1 represent the number of remissions under each treatment arm. The chi-square statistic and associated degrees of freedom are provided in CS_stat and CS_df with the attained p value in CS_pval. The calculated index to Fisher s exact test is given by FE1=1, which appears in six rows, along with three p values associated with this test, FE_Lpval, FE_Rpval, FE_Tpval for one-tailed tests to the left and to the right and the two-tailed test, respectively. These nine records or observations illustrate, by the variable rule, three scenarios where superiority was detected, three scenarios where futility was detected, and three scenarios where neither superiority nor futility was detected. 29 1: < S 30 1: < S 31 1: < S

5 32 2: F 33 2: F 34 2: F 35 2: : : Thus, by selecting a significance level, one can identify outcomes where superiority/futility is conclusive. Having a significance level decided upon in advance then programmatically applied to the final data set, a decision rule was assessed to indicate scenarios where superiority was detected or scenarios where attained futility in the trial was detected. The decision rule takes into account the expected cell sizes and, therefore, implementation of either the chi-square test or Fisher s exact test. By the method illustrated herein, the r can now more easily inspect the outcome of all 961 scenarios, or restrict inspection simply to those scenarios where the decision rule indicates superiority or futility. Conclusion When faced with the challenge, during a clinical trial, of simulating a number of potential outcomes, performing appropriate statistical tests, and determining a course of action that would be based on these tests, the enormity of the scale for judicious reasoning becomes unnaturally and disconcertingly stretched. Simplification of any extravagant production process to provide the decision maker with a reasonable summary of critical points could be crucial to the clinical trial or could provide a deterrent to futile pursuits. The superiority of SAS macroprocessing was brought into focus in this exercise in the face of futility of continual investigating. That which appeared at first to be a simple task from the position of the author of the request became escalated conceivably beyond reason and evidently to the deterrence of further posing of regretful requests. The DCRI developed a software program using the SAS System, in particular, SAS macroprocessing, to address not only this current situation but any future query that any investigator might have. This could be done with a highly favourable response time that returns not just an answer to the specific request but an exhaustive and complete analysis based on the initial request. Furthermore, complexity of multiplicity in the problem was reduced to simplicity by SAS macroprocessing, yet the reader should note that the samples of SAS code herein are not collectively complete but are representative of the key steps in the process. Appendix A contains the first 110 records and the final 81 records from the same SAS data set of results. This Appendix appears at the end of this paper. Acknowledgements The authors are appreciative of the DCRI and this forum of SAS rs for the opportunity to promote this knowledge as far as it would contribute ultimately toward the improvement of patient care. Contact Information Phil d Almada DCRI, Duke Medical Center 300 W. Morgan St., Suite 800 Durham, NC Office phone number: Facsimile phone number: address: phil.dalmada@duke.edu SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. indicates USA registration. Other brand and product names are trademarks of their respective companies.

6 Appendix A Partial test results from the simulation of 30 subjects per treatment arm. Futility/superiority simulation 5 Chi-square and Fisher's Exact tests, 961 scenarios. 1 1: : F 3 1: : : : : : : : : : S 13 1: S 14 1: < S 15 1: < S 16 1: < S 17 1: < S 18 1: < S 19 1: < S 20 1: < S 21 1: < S 22 1: < S 23 1: < S 24 1: < S 25 1: < S 26 1: < S 27 1: < S 28 1: < S 29 1: < S 30 1: < S 31 1: < S 32 2: F 33 2: F 34 2: F 35 2: : : : : : : : : : : S 46 2: S 47 2: < S 48 2: < S 49 2: < S 50 2: < S 51 2: < S 52 2: < S 53 2: < S 54 2: < S 55 2: < S

7 Futility/superiority simulation 6 Chi-square and Fisher's Exact tests, 961 scenarios. 56 2: < S 57 2: < S 58 2: < S 59 2: < S 60 2: < S 61 2: < S 62 2: < S 63 3: : F 65 3: F 66 3: F 67 3: : : : : : : : : : : : S 79 3: < S 80 3: < S 81 3: < S 82 3: < S 83 3: < S 84 3: < S 85 3: < S 86 3: < S 87 3: < S 88 3: < S 89 3: < S 90 3: < S 91 3: < S 92 3: < S 93 3: < S 94 4: : : F 97 4: F 98 4: F 99 4: : : : : : : : : : : : S

8 Futility/superiority simulation 21 Chi-square and Fisher's Exact tests, 961 scenarios : < S : < S : < S : S : : : : : : : : : : : : F : F : F : : < S : < S : < S : < S : < S : < S : < S : < S : < S : < S : < S : < S : < S : < S : < S : < S : S : S : : : : : : : : : : : F : F : F : < S : < S : < S : < S : < S

9 Futility/superiority simulation 22 Chi-square and Fisher's Exact tests, 961 scenarios : < S : < S : < S : < S : < S : < S : < S : < S : < S : < S : < S : < S : < S : S : S : : : : : : : : : : F :