dbets - diffusion Breakpoint Estimation Software

Transcription

1 dbets - diffusion Breakpoint Estimation Software This document describes the use of the online application dbets to estimate the diffusion (DIA) test breakpoints from a collection of paired minimum inhibitory concentration (MIC) tests and DIA tests. dbets provides methods for both data visualization and computation using model-based approaches and the error-rate bounded method (ERB). It was created using the Shiny application from RStudio. Please note that dbets and this documentation are works in progress. They are based on the works of Glen DePalma and Bruce A. Craig (Department of Statistics, Purdue University). If you encounter any problems or find the documentation unclear, please contact Glen DePalma at gdepalma@purdue.edu. The example data used throughout this documentation can be found here DIABrkpt/data1.csv. Note this is the default data set inputed on the Load Data page. Contents 1 General Information 2 2 Data Preparation and Data Upload Uploading Data and Setting Parameters ERB Tab 1: Error Rate Bounded Method Error Rate Bounded Method Results Tab 2: Assess Uncertainty ERB Tab 3: Selected Breakpoints Model-Based Approaches Tab 1: Logistic Model Logistic Model Results Tab 2: Spline Model Spline Model Results Tab 3: Compare Model Fits Tab 4: DIA Classification Probability

2 1 General Information On startup, dbets loads a home screen providing general information about the software. dbets is divided into three main sections: load data, ERB procedures, and model-based procedures. You can navigate sections using the Navigate box located at the top-left of each page. When in the ERB or model-based sections, there are multiple tabs across the top (colored in light blue) denoting various procedures. Optional inputs associated with each procedure are located on the left side-bar of each page. A run button is located at the bottom of the side-bar. Set the inputs accordingly and then click the run button to initiate the procedure. You can alter the input settings and re-run the procedure as necessary. After the procedures are completed, plot options and a plot graph button will appear below the run button. You can set graph options and click the plot graph button to produce a graph. Some graphs are downloadable. In those cases a download graph button will also appear. Please note that you should use your latest browser version. This application works best with preferably Firefox, Chrome, or Safari. There may be problems if a different browser, such as Internet Explorer, is used. 2

3 2 Data Preparation and Data Upload Uploading data and setting MIC breakpoints is done on the Load Data page. On startup, a default.csv data set available for download into the application by clicking the load data button. If you have your own data set, you will need to specify the location prior to hitting the load data button. This process is described in detail in Section 2.1. Prior to loading any data set, you need to make sure it is properly formatted. The data should be entered in a.csv (comma delimited) file with two columns labeled MIC and DIA. The first row should include these header names with the MIC data entered in the first column. Censoring information is important. If a value is censored put a < or > symbol before the value, with no space between. No other symbols are permitted. A portion of an example data spreadsheet is shown in Figure 1. Notice the distinction between >256 and 256. The default data set can be downloaded here. Figure 1: Formatted data set with censoring Note the MIC data can be stored on the original dilution scale or log 2 dilution scale. DIA is assumed to be measured in mm. 3

4 2.1 Uploading Data and Setting Parameters Now that the data set is properly formatted, we ll describe in detail the loading procedure. The first option on the side-bar specifies if the.csv file is available online or on your local machine. If the data is online, enter the appropriate address in the File URL text box. If the data is on your local machine, select Use a file stored on my local machine, a pop-up window will appear that will allow you to browse and locate the file. The next option tells the application whether the MIC data are on the log 2 scale (..., 4, 3,...3, 4,...). The last option prior to loading the data is to state the Lower and Upper MIC breakpoints on the log 2 scale. Once these are set click the load data button to upload the data. Data set characteristics and graphing options are then displayed. Please note the MIC breakpoints are used throughout the application. If you wish to change them at any time, you must return to the Load Data page and click the load data button again. Figure 2: Load Data Screen 4

5 3 ERB The ERB section utilizes the error-rate bounded (ERB) method for DIA breakpoint determination. There are three tabs: 1. Error Rate Bounded Method - find the optimal DIA breakpoints 2. Assess Uncertainty ERB - assess the uncertainty in the DIA breakpoints, in other words, given the data, how confident can we be in a specific set of DIA breakpoints selected by the ERB method? 3. Selected Breakpoints - evaluate the performance of selected DIA breakpoints Figure 3: ERB page loaded on startup. Note the three tabs across the top of the screen. 5

6 3.1 Tab 1: Error Rate Bounded Method This page is used to find the ERB breakpoints based on user-specified discrepancy percentages. Two sets of percentages are required, one set for isolates within one of the intermediate range (VM1, M1, m1), and a second set for the isolates outside one of the intermediate range (VM2, M2, m2). The application are converts these percentages into weights, which are in turn used in an overall discrepancy index (described in detail at the bottom of this section). The function searches through all sets of possible DIA breakpoints and selects the ones that minimize this index. Note the default values for these percentages are based on the M23 document. In order to find the optimal ERB breakpoints fo the given sets of percentages, we create weights that reflect the seriousness of such a classification discrepancy. To do this, the algorithm converts the given set of percentages, %, to weights, w, using: w = max(%) %. For the default sets of percentages, the weights are: VM1=4, M1=4, m1=1, VM2=20, M2=20 and, m2=8. Thus a very major error that occurs outside one dilution of the intermediate range is considered to be 5 times more serious than one that occurs within one dilution of the intermediate range. The index is simply an accumulation of these weights and can be expressed as: index = w V M1 #V M1 + w M1 #M1 + w m1 #m1 + w V M2 #V M2 + w M2 #M2 + w m2 #m2 where w represents the weight associated with a discrepancy and #* represents the counts of a particular discrepancy. All possible DIA breakpoints are searched (within the minwidth and maxwidth range) to find the ones that minimize the index. 6

7 3.1.1 Error Rate Bounded Method Results The results for the ERB method for the defualt data set are shown below. Figure 4: Output from ERB Method The optimal DIA breakpoints are 34, 39. Information is given describing the amount of discrepancy errors in the output and graph. Note the index score is.33. 7

8 3.2 Tab 2: Assess Uncertainty ERB A bootstrap procedure is performed 12,000 times. This method involves resampling the data (with replacement) and computing the optimal DIA breakpoints for each sample. Information and a graph are displayed indicating the dispersion of selected DIA breakpoints. The more spread out the selected DIA breakpoints the less confidence one can have for using a particular set of DIA breakpoints. Example output is shown in Figures 5 and 6. Figure 5: DIA breakpoint distribution across 12,000 samples Figure 6: DIA breakpoint distribution across 12,000 samples 8

9 3.3 Tab 3: Selected Breakpoints This page displays ERB information regarding selected DIA breakpoints. One may want to compare the index score from the optimal DIA breakpoints found in Tab 1 to the index score resulting from the selected DIA breakpoints. For the default data set the ERB DIA breakpoints are 34, 39 with an index score of.33. Instead of 34, 39 we input DIA breakpoints of 35, 40; below are the results. Notice the index score is.38 compared to the optimal DIA breakpoints index score of.33. Figure 7: Selected DIA Breakpoints for the ERB Method 9

10 4 Model-Based Approaches The model-based approaches section uses a model-based approach to estimate DIA breakpoints. This approach is recommended due to concerns with the ERB approach. These concerns include: the ERB method is biased, not precise, and does not take into account the test characteristics (rounding, measurement error,...). The model-based methods attempt to get at the true relationship between the DIA and MIC test while the ERB method is focused on observed test results from one sample. The model-based approaches has four tabs: 1. Use a four parameter logistic function to model the relationship between the DIA and MIC test, 2. Use a non-parametric spline model to model the relationship between the DIA and MIC test, 3. Compare the fits of the two models, 4. Compare DIA classification probabilities given one of the model fits. For the model-based procedures we distinguish between true and observed MIC breakpoints. Due to the rounding up of the MIC test we assume the true MIC breakpoints are the test MIC breakpoints -.5. For tabs 1 and 2, one must determine the appropriate loss function for estimating the DIA breakpoints. The different loss functions available are described below. 1. DIA at Least as Powerful - This is the recommended target. It will attempt to make the DIA test at least as powerful as the MIC test in terms of correct classification probability. 2. Minimize Discrepancies - Compute DIA breakpoints that minimize the discrepancies between the two tests. 3. Equate Tests - Compute DIA breakpoints that make the DIA test as similar as possible to the MIC test regarding classification probabilities. 10

11 4.1 Tab 1: Logistic Model The Logistic Model page fits a four-parameter logistic model to the data. The options on the left side-bar allows you to specify the minimum and maximum width of the DIA breakpoints. In addition one can select the target for the estimation of DIA breakpoints. The formula for the four parameter logistic model is as follows: mb = 2 β 3 β 4 β 3 + β 4 1 fx = 1 + exp{ mb (β 2 x)} y = β 1 (fx exp{β 3 (β 2 x)} + (1 fx) exp{β 4 (β 2 x)}) 1 + fx exp{β 3 (β 2 x)} + (1 fx) exp{β 4 (β 2 x)} **** Explanation of Coefficients This model is very flexible in that it allows the logistic curve to have asymmetry curvature. However some data sets may not follow a logisitic fit and it is then recommended to use the Spline approach. 11

12 4.1.1 Logistic Model Results Figure 8: Summary of Results from the Logistic Model Figure 9: Logistic Fit to the Data Figure 8 provides MCMC details and resulting DIA breakpoint distribution. Figure 9 displays the resulting fit to the data. This graph is useful for seeing how well the logistic function fits the data. Note a graph of the distribution of DIA breakpoints, similar to Figure 6, is also displayed but not shown here. 12

13 4.2 Tab 2: Spline Model The Spline Model page fits a spline model to the data. If the data do not follow a logistic curve the spline model will provide more accurate estimates of the DIA breakpoints. The spline model is fit via I-splines (Ramsay, 1988). I-splines ensure the relationship is monotonically decreasing given the spline coefficients are positive. An example of I-spline basis functions is given below in Figure 10. Figure 10: I-spline bases for knots: 0,.2,.4,.6,.8, 1 When using a spline procedure, one must select the appropriate knot locations. Our procedure places knots at every half unit increment and uses a random walk prior (Christensen et al. 2012) to constrain the coefficients to prevent overfitting. 13

14 4.2.1 Spline Model Results Figure 11: Summary of Results from the Spline Model Figure 12: Spline Fit to the Data Figure 11 provides MCMC details and resulting DIA breakpoint distribution. Figure 12 displays the resulting fit to the data. Note a graph of the distribution of DIA breakpoints, similar to Figure 6, is also displayed but not shown here. 14

15 4.3 Tab 3: Compare Model Fits This page graphically compares the logistic and spline fits to the data. Note this procedure can only be called after both the Logistic Model page and the Spline Model page have been called. Figure 13: Both Model Fits to the Data 15

16 4.4 Tab 4: DIA Classification Probability This page computes the DIA probability classification for one or two sets of DIA breakpoints. It can be used to compare the performance of DIA breakpoints selected by the various methods. Figure 14: Summary of Classification Probability Figure 15: Probability Classification across MIC Values.myp Figure 14 displays classification results for the MIC as well as two sets of DIA breakpoints: 33, 40 and 34, 39. Figure 15 graphically displays the classification probabilities across MIC values. Note the classification probabilities are calculated within ± 2 of the intermediate range. 16