The framework of the BCLA and its applications

Transcription

1 NOTE: Please cite the references as: Zhu, T.T., Dunkley, N., Behar, J., Clifton, D.A., and Clifford, G.D.: Fusing Continuous-Valued Medical Labels Using a Bayesian Model Annals of Biomedical Engineering, 5 DOI:.7/s Zhu, T.T., Pimentel, M.A.F., Clifford, G.D., and Clifton, D.A.: Bayesian Fusion of Algorithms for the Robust Estimation of Respiratory Rate from the Photoplethysmogram IEEE Engineering in Medicine & Biology Conference, Milan, Italy, 5 The framework of the BCLA and its applications Tingting Zhu Supervised by David Clifton and Gari Clifford 7 th May 5

2 35 3 Ground truth Algorithm Algorithm Algorithm 3 5 RR (bpm) x

3 Motivation Improve continuous valued annotations through aggregating labels from novices to reach an expert level. Why continuous-valued Heart rate; Blood pressure; ECG QT Interval; Respiratory rate; Timestamps of event. Chicken and egg problem : If we know the ground truth labels then we can measure how precise and accurate each annotator is. If we know the performance of each annotator then we can infer a reliable ground truth. 3

4 Bayesian Continuous- valued Label Aggregator (BCLA) mean for Bias for mean of bias precision of bias λ λ For i =,,N Unknown ground truth for annotations for, jth annotator λ For j =,,R

5 mean for Unknown ground truth for ith record For i =,,N Raykar s model: [] For j =,,R Unknown ground truth for Prior of For i =,,N Previous works Bias for λ For j =,,R The true annotation for the : ( ;, ) where a is the mean and is the variance. a can be expressed as a linear regression function f(w,x), where w are the regression coefficients. The annotation from jth annotator for, : ( ;, ) where λ is the precision of the jth annotator as: = + References: Continuous version of Simultaneous Truth and Performance Level Estimation (STAPLE): [] The annotation from jth annotator for, : ( ; +, ) where is the bias and λ is the precision of the jth annotator. P A flat prior is assumed in STAPLE. [] Vikas C. Raykar, Shipeng Yu, Linda H. Zhao, Gerardo Hermosillo Valadez, Charles Florin, Luca Bogoni, and Linda Moy.. Learning From Crowds. Journal of Machine Learning Research (August ), [] Warfield SK, Zou KH, Wells WM. Validation of image segmentation by estimating rater bias and variance. Philosophical transactions Series A, Mathematical, physical, and engineering sciences. 8;366(87):

6 Bayesian Continuous- valued Label Aggregator (BCLA) mean for Bias for mean of bias precision of bias λ λ For i =,,N Unknown ground truth for annotations for, jth annotator λ For j =,,R 6

7 Simulation - Investigation of the bias values Mean of the biases = Mean of the biases = Mean of the biases = - 7

8 Bayesian Continuous- valued Label Aggregator (BCLA) The true annotation for the : ;, = ; f(w,x), where a is the mean and is the variance. a can be expressed as a linear regression function f(w,x) and w are the regression coefficients. The annotation from jth annotator for, : ( ; +, ) where is the bias and λ is the precision of the jth annotator. The bias can defined as: ;, where is the variance of the bias for the jth annotator. λ,, and are real positive numbers from a Gamma distribution: ( ; λ, λ ) & ( ;, ) & (;, ) where is the shape and is the scale of the distribution. mean for ith record Unknown ground truth for For i =,,N Precision for ith record mean of bias precision of bias Bias for λ annotations for, jth annotator λ λ For j =,,R 8

9 The likelihood of parameters for a given dataset D: Bayes Rule: = h Solving it using the maximum-a-posteriori (MAP) estimation: ( ; +, ) ( ;, ) ( ; f(w,x), ) ( ; λ, λ ) (;, ) (;, ) 9

10 Expectation Maximization (EM) - Solving parameters when the ground truth is not available E step: Expected ground truth M step: variance for jth annotator Regression coefficients Bias for jth annotator Variance of the biases from all annotators Variance of the ground truth

11 Convergence criteria for the MAP-EM approach Precision in the MAP-EM model has a convergence issue as it goes to infinity. Possible solutions: Set a scalar threshold value (i.e.:. where the best annotator is 5ms away from the unknown ground truth); Set the threshold using the extreme value theorem 99th quantile of the inverse CDF in the generalised extreme value distribution (GEV). where k is the shape parameter, is the scale parameter, and is the location parameter.

12 Applications

13 ECG QT interval estimation Physionet 6 adult QT database automatic estimation of the QT interval in electrocardiogram Gold standard - Manual & semi-manual experts D D RMSE (ms) of the inferred true QT intervals for 58 records Best Algorithm Mean Median ML STAPLE BCLA (MAP) Automated (8 D) D3 Automated ( D3) Automated (69 D)

14 Capnobase respiratory rate estimation Data ( collected during elective surgery and routine anaesthesia. PPG recordings and capnometry data (Fs = 3 Hz); Only subjects considered -- 9 children and 3 adults, each has 8 min recordings. The annotations were estimated using 6 modulations (3 FFTs and 3 ARs) which are derived from FFT and AR, each produced RR estimated from signals: Baseline wander (or "respiratory-induced intensity variation"); Amplitude modulation (or "respiratory-induced amplitude variation"); Frequency modulation (or "respiratory-induced frequency variation"). Gold standard - invasive measurements Each RR was computed over a 3-second window, overlapped by 9 seconds resulted 5 RRs per subject.

15 Results across subjects: ML Raykar s model using Maximum Likelihood; MAP - Maximum a Posteriori proposed by the BCLA; TB - Theoretical Best algorithm with highest precision after bias correction, and selecting best algorithm per subject; TB(mae) - Theoretical Best algorithm with least mean absolute error, and selecting best algorithm per subject; BA - Best Algorithm with highest precision after bias correction or with least mean absolute error across all subjects. STAPLE - similar to MAP but no assumption of distribution on the bias term. Benchmark Algorithm (Smart fusion) - mean voting and discard windows when respiratory rate std>. W. Karlen et al., Multiparameter respiratory rate estimation from the photoplethysmogram, IEEE TBME, vol. 6, no. 7, pp , 3 5

16 Current work 6

17 mean for Bias for mean of bias precision of bias λ λ Unknown ground truth for For i =,,N annotations for ith record, jth annotator λ For j =,,R 7

18 BCLA with physiological features feature for Linear regression function: f(x) = f(w,x) and w are the regression coefficients. ( ) mean for For i =,,N Unknown ground truth for Bias for mean of bias precision of bias annotations for, jth anntoator λ λ λ For j =,,R Physiological features account for patient specific estimation 8

19 BCLA with signal quality features feature for ( ) mean for For i =,,N Unknown ground truth for Bias for mean of bias precision of bias λ annotations for ith record, jth anntoator λ λ For j =,,R T signal quality as a prior for precision Use signal quality to describe task difficulty 9

20 BCLA with physiological feature and signal quality as a prior Use heart rate and signal quality (as a prior for precision) on ECG QT dataset: Signal quality metric: bsqi The percentage of beats on which two different QRS detectors agree in terms of the R peak position. bsqi ==: perfect agreement; bsqi <: disagreement due to presence of noise. D ) Find annotations where i) bsqi == (365 recordings); ii) >bsqi >=.5 (7 recordings); iii) <bsqi<.5 (8 recordings; iv) bsqi== (8 recordings). ) Fuse those annotations using the BCLA to obtain precision value for each annotator; D3 D D3 D 8 PDF 6 Gamma prior for precision x

21 BCLA with physiological feature and signal quality as a prior RMSE (ms) Best Algorithm PCinC 6 dataset with beat specific features No features Normalised HR (nhr) nhr + bsqi defined prior Mean Median ML BCLA ML BCLA ML BCLA D D D MAE (ms) Best Algorithm PCinC 6 dataset with beat specific features No features Normalised HR (nhr) nhr + bsqi defined prior Mean Median ML BCLA ML BCLA ML BCLA D D D

22 BCLA with signal quality(task difficulty) extension feature for ( ) mean for For i =,,N Unknown ground truth for Bias for mean of bias precision of bias λ annotations for ith record, jth anntoator λ λ For j =,,R If all algorithms label the same segment -- Signal quality for is same for all annotators (i.e. ) If algorithms label at different segments of a record -- Signal quality for is different for all annotators (i.e. ) ( ; +, ) ( ; +, ) Signal quality for from jth annotator

23 Signal quality metric: bsqi The percentage of beats on which two different QRS detectors agree in terms of the R peak position. == means perfect agreement and the segment is clean; < < means disagreement due to presence of noise. ( ; +, ) ( ; +, ) RMSE of the inferred ground truth on ECG QT dataset: D D3 D The lower bound o The lower bound o The lower bound o Effect of introducing bsqi 3

24 Finally

25 BCLA with correlation extension - remove the assumption that annotators are independent correlation of annotations Y Y7 Y7 - Ref Y - Ref Overlap correlation of annotations - reference Annotators can be highly correlated but do not have identical annotations; - Annotators with correlation = indicates: may or may not be repeated submission and can not be directly removed. - Challenges: () Missing values in the dataset; () Missing ground truth. 5

26 Before thresholding PCinC 6 ECG QT dataset RMSE (ms) Best algorithm Mean Median ML MAP D D D thresholding = (i.e. correlation ==) PCinC 6 ECG QT dataset RMSE (ms) Best algorithm Mean Median ML MAP annotator numbers D D D ; 6-;-;9-7;38- -7;-;3-;3- thresholding >=.99 (i.e. correlation >=.9) PCinC 6 ECG QT dataset RMSE (ms) Best algorithm Mean Median ML MAP annotator numbers D ; 6-;-;9-7;38-;- 9;5-8;7-8;8-7;7-5;3-8;3-5;3-7 D ;-;3-;3-;3- D

27 BCLA with correlation extension Alternatively, the covariance matrix can defined as: = Where is an R by R diagonal matrix with diagonal entries being =,, =. is the precision of annotation for jth annotator, and is the correlation matrix with size R by R annotators. Precision: is modelled from a Gamma distribution with parameters k and. Correlation: is modelled from a Beta distribution with parameters and. 7

28 Simulation Simulated Results using 5 annotators for 58 records Best Algorithm Mean Median ML (Raykar-R) BCLA - MAP BCLA - MHc Raykar-R BCLA-MAP BCLA-MHc RMSE (ms) MAE (ms) Estimated std of annotations 8 6 Original Correlation Simulated std of annotations Estimated Correlation Estimated bias of annotators (ms) BCLA-MAP BCLA-MHc Simulated bias of annotators (ms) 8

29 Real datasets Original Correlation Original Correlation Original Correlation Estimated Correlation Estimated Correlation Estimated Correlation Original Correlation Original Correlation Original Correlation Estimated Correlation Estimated Correlation Estimated Correlation

30 Acknowledgement Supervisors David Clifton and Gari Clifford CHI and IPM groups RCUK Digital Economy Programme grant number EP/G3686/ ARM Scholarship in Sustainable Healthcare Technology through Kellogg College 3

31 5 3 8 Estimated True 8 Estimated True Mean QT intervals (ms) Std QT intervals (ms) 6 Bias(ms) - - varaince of bias (ms) 6 Simulation Iteration (every ) x Iteration (every ) x Iteration (every ) x.5.5 Iteration (every ) x 3 BCLA-MAP BCLA-MHc Likelihood (ms) 3 5 x Iteration (every ) Original Correlation x std of annotations (ms) Iteration (every ) x Mean absolute difference in annotations Iteration (every ) x Simulated Results using all 5 annotators Best Algorithm Mean Median variance of absolute difference in annotations ML (Raykar -R) Iteration (every ) BCLA - MAP BCLA - MHc x Estimated bias of annotators (ms) Estimated std of annotations Simulated bias of annotators (ms) Raykar-R BCLA-MAP BCLA-MHc 6 8 Simulated std of annotations Estimated Correlation.5 RMSE (ms) MAE (ms)