Estimation of Inferential Uncertainty in Assessing Expert Segmentation Performance from STAPLE

Transcription

1 1 Estmaton of Inferental Uncertanty n Assessng Expert Segmentaton Performance from STAPLE Olver Commowck, Smon K. Warfeld, Senor Member IEEE Computatonal Radology Laboratory, Department of Radology, Chldren s Hosptal, 300 Longwood Avenue, Boston, MA, 02115, USA E-mal: {Olver.Commowck, Smon.Warfeld}@chldrens.harvard.edu Abstract The evaluaton of the qualty of segmentatons of an mage, and the assessment of ntra- and nter-expert varablty n segmentaton performance, has long been recognzed as a dffcult task. For a segmentaton valdaton task, t may be effectve to compare the results of an automatc segmentaton algorthm to multple expert segmentatons. Recently an Expectaton Maxmzaton (EM) algorthm for Smultaneous Truth and Performance Level Estmaton (STAPLE) was developed to ths end to compute both an estmate of the reference standard segmentaton and performance parameters from a set of segmentatons of an mage. The performance s characterzed by the rate of detecton of each segmentaton label by each expert n comparson to the estmated reference standard. Ths prevous work provdes estmates of performance parameters, but does not provde any nformaton regardng the uncertanty of the estmated values. An estmate of ths nferental uncertanty, f avalable, would allow the estmaton of confdence ntervals for the values of the parameters. Ths would facltate the nterpretaton of the performance of segmentaton generators, and help determne f suffcent data sze and number of segmentatons have been obtaned to precsely characterze the performance parameters. We present a new algorthm to estmate the nferental uncertanty of the performance parameters for bnary and multcategory segmentatons. It s derved for the specal case of the STAPLE algorthm based on establshed theory for general purpose covarance matrx estmaton for EM algorthms. The bounds on the performance parameters are estmated by the computaton of the observed Informaton Matrx. We use ths algorthm to study the bounds on performance parameters estmates from smulated mages wth specfed performance parameters, and from nteractve segmentatons of neonatal bran MRIs. We demonstrate that confdence ntervals for expert segmentaton performance parameters can be estmated wth our algorthm. We nvestgate the nfluence of the number of experts and of the segmented data sze on these bounds, showng that t s possble to determne the number of mage segmentatons and the sze of mages necessary to acheve a chosen level of accuracy n segmentaton performance assessment. Index Terms Covarance Matrx, Informaton Matrx, Confdence Intervals, Expectaton-Maxmzaton, Valdaton, STAPLE. I. INTRODUCTION The evaluaton of mage segmentaton has long been recognzed as a dffcult problem. Many methods have been Copyrght (c) 2009 IEEE. Personal use of ths materal s permtted. However, permsson to use ths materal for any other purposes must be obtaned from the IEEE by sendng a request to pubs-permssons@eee.org. proposed n the lterature to deal wth t. These can be classfed nto two groups. Frst, the evaluaton can be based on dstances between surfaces extracted from the segmentatons. For example, these can be the Hausdorff dstance [1] or a mean dstance between the two surfaces [2]. The other class of measures contans voxel-based measures,.e. overlap measures based on voxelwse computatons. Among those, the Dce smlarty coeffcent [3] or the Jaccard smlarty coeffcent [4], [5] have been wdely used to measure the overlap between two segmentatons. These two classes of measures have ther advantages and drawbacks. Both may be used to provde nsght nto the qualty of a segmentaton [6] and to allow the comparson of segmentatons. However, when valdatng a segmentaton algorthm, usng only one expert segmentaton as the reference standard may be napproprate as any ndvdual manual segmentatons have large or small errors. In ths artcle, we therefore focus on usng several expert segmentatons to estmate a reference standard and utlze t for the comparson of segmentatons (from experts or algorthms). One algorthm for ths, called STAPLE [7], uses an Expectaton-Maxmzaton (EM) algorthm to estmate teratvely, from a set of J expert segmentatons, the hdden reference standard segmentaton and performance parameters for each segmentaton. These parameters characterze the agreement of a gven expert wth the reference standard, expressed as rates of detecton of labels. The STAPLE algorthm generates only pont estmates of the performance parameters, and provdes no nformaton about the amount of uncertanty n the values of the estmates of the parameters. Precse knowledge of the nferental uncertanty would enhance our ablty to nterpret the performance of segmentaton generators, and could be used to determne f suffcent data sze and number of segmentatons have been obtaned to precsely characterze the performance parameters. For example, consder plannng to evaluate a new segmentaton algorthm for a new data set or patent populaton. A reference standard for assessng the segmentaton algorthm could be developed usng repeated nteractve segmentaton of some mages of a data set. When desgnng such an experment, an estmate of the nferental uncertanty, f avalable, would descrbe confdence ntervals for the values of the parameters and provde a way to determne how many experts should nteractvely delneate the data set and how many voxels or slces should be delneated so that the STAPLE estmates of the

2 2 parameters are precse enough (.e. the confdence ntervals are tght). Such confdence ntervals ndeed descrbe the certanty wth whch we know the value of the parameter. A dfferent concept s the confdence nterval for rater performance, whch descrbes the range of performance we expect to see across repeated segmentatons by the same rater. If the nferental uncertanty of the values of performance parameter estmates are very small, then a confdence nterval for rater performance can be estmated smply as the sample varance over repeated segmentatons. We propose to estmate the covarance matrx of the performance parameters from STAPLE by computng the observed Informaton Matrx. Ths computaton has been descrbed n the general EM framework [8]. In ths paper, we buld upon [9] and derve, both for bnary and mult-category segmentatons, analytc closed form expressons necessary to compute the covarance of the performance parameters obtaned from STAPLE. We then demonstrate factors nfluencng the uncertanty n the estmated performance parameters wth smulated segmentatons. Then, we apply our algorthm to characterze the segmentatons of brans of newborn nfants, comparng the bnary and mult-category expressons, showng that our algorthm provdes gudance for the desgn of future valdaton studes. A. The STAPLE Algorthm II. METHOD We frst recall brefly the prncple of the STAPLE algorthm [7]. It uses as an nput a set of segmentatons from J experts (ether manual delneatons or automatc segmentatons). These segmentatons can ether be bnary segmentatons or mult-category segmentatons,.e. several structures are delneated each one gettng a specfc label. Ths nformaton s avalable as decsons d j, ndcatng the label gven by each expert j for each voxel. The goal of STAPLE s then to estmate both a reference standard segmentaton T, and parameters θ = {θ 1,..., θ j,..., θ J } descrbng the agreement between the experts and the reference standard. Each of the parameters θ j s an L L matrx, where L s the number of labels n the segmentaton, and θ js s s the probablty that the expert j gave the label s to a voxel when the label of the reference standard s s,.e. θ js s = P (d j = s T = s). If the reference standard was known, then estmatng the performance parameters for each expert would be straghtforward. However, as ths reference standard s unknown, an Expectaton-Maxmzaton approach [10], [8] s used to estmate T and the expert performance parameters. The EM algorthm proceeds by teratng two steps: E-Step: Compute the expected value of the complete data log-lkelhood Q(θ θ (k) ) knowng the expert parameters at the precedng teraton: θ (k). Evaluatng ths expresson requres the knowledge of the posteror probablty of T : P (T D, θ (k) ), whch s suffcent n ths case to perform the Maxmzaton step. M-Step: Estmate the performance parameters at teraton k + 1, θ (k+1) by maxmzng the complete data loglkelhood, usng the current estmate of the reference standard. B. Covarance and Informaton Matrx 1) General Maxmum-Lkelhood Case: We are nterested n the computaton of confdence ntervals, llustrated on Fg. 1, on the performance parameters estmated from STAPLE,.e. a lower bound and upper bound on each estmated parameter ˆθ jl l. Ths reles on the computaton of the covarance matrx Σ(θ) of the expert parameters. Ths s done va the computaton of the Informaton Matrx I(θ) of the parameters obtaned after convergence of the Expectaton Maxmzaton algorthm, I(ˆθ). Fg. 1. Illustraton of the confdence nterval on one parameter. We am at computng the lower (LB) and upper bound (UB) for each parameter ˆθ jl l estmated by STAPLE. In the case of a known ground truth (experments on smulated data), ths range can be compared to the true value θ jl l to check for the accuracy of parameter estmaton n STAPLE. If all the data was known, the computaton of the Informaton Matrx would be smply the matrx of the second dervatves of the log-lkelhood functon, estmated at ˆθ: I(ˆθ) = 2 p 1 q 1 p 1. q J p 1 p 1 q 1 p 1 q J q 1 q J 2 q q J q 1 2 q J (θ)=(ˆθ) Then, the covarance matrx s obtaned usng the wellknown result [11] Σ(θ) = I 1 (θ), under the assumpton of a large number of samples: (1) σ 2 (ˆp 1 ) σ(ˆp 1, ˆq 1 ) σ(ˆp 1, ˆq J ) σ(ˆq 1, ˆp 1 ) σ 2 (ˆq 1 ) σ(ˆq 1, ˆq J ) Σ(ˆθ) = (2) σ(ˆq J, ˆp 1 ) σ(ˆq J, ˆq 1 ) σ 2 (ˆq J ) The confdence bounds of the estmated parameters are n turn computed from these values by assumng that ˆθ jl l θ jl l follows a Normal dstrbuton N(0, σ(ˆθ jl l)). A two-sded 100(1 α)% confdence nterval can then be constructed as [ˆθ jl LB UB l; ˆθ jl l ] = ˆθ jl l ± z 1 α/2 σ(ˆθ jl l) (3) where z 1 α/2 corresponds to the z-score related to the desred confdence nterval (for a 95% confdence nterval, z 1 α/2 = 1.96). Moreover, f the Normal assumpton does not hold, whch may be the case when the performance parameter values are very close to 0 or 1, a functon g (such as the Box- Cox transform [12] or the logt functon,.e. logt(x) = log(x) log(1 x)) may be used to transform the parameters to obtan a Normal dstrbuton. Assume that g(ˆθ jl l) g(θ jl l) follows a Normal dstrbuton N(0, σ(g(ˆθ jl l))). Then, the confdence nterval can be estmated as [g(ˆθ LB jl l ); g(ˆθ UB jl l )] =

3 3 g(ˆθ jl l) ± z 1 α/2 σ(g(ˆθ jl l)). The covarance matrx computed wth Eq. (2) may then[ be used ] to compute the confdence ntervals: σ(g(ˆθ jl l)) g θ (ˆθ jl l) σ(ˆθ jl l) [13] (page 626). 2) Estmatng the Varance-Covarance Matrx n Mssng Data Problems: In the case of an EM algorthm such as STA- PLE, the hdden varables are unknown and ther value may only be estmated. Therefore, only the observed Informaton Matrx I(θ) can be computed. The expresson of I(θ) has been derved for a general EM algorthm n [8] (page 100). We proceed by frst computng the expected value of the complete data Informaton Matrx I c (θ) usng the expected complete data log-lkelhood Q(θ θ (k) ) estmated n the EM algorthm. Then, to account for the uncertanty from the mssng data, the expected mssng data Informaton Matrx I m (θ) s subtracted from I c (θ) to obtan the observed Informaton Matrx,.e. I(θ) = I c (θ) I m (θ). These two matrces (I m and I c ) are computed from Q(θ θ (k) ) once the estmates of the parameters have converged,.e. when θ (k+1) θ (k). We now present the dervaton of these two terms for STAPLE n the bnary case,.e. when only one structure and the background were delneated by each expert. Then, we present an extenson of the observed Informaton Matrx computaton to the multcategory case. C. Computaton of the Observed Informaton Matrx n the Bnary Case In the bnary case, each expert has delneated one structure by attrbutng the value 1 to a voxel belongng to the structure and 0 otherwse (background). In ths partcular case, the θ parameters can be represented entrely by two parameters for each expert j: p j = P (d j = 1 T = 1) and q j = P (d j = 0 T = 0). p j s also known as the senstvty of the expert j whle q j s also known as the specfcty. To smplfy as much as possble the notaton for the followng equatons, we use the general notaton θ js s for the performance parameters, keepng n mnd that only p j = θ j11 and q j = θ j00 are the meanngful parameters (θ j01 and θ j10 beng completely determned as θ j01 = 1 p j and θ j10 = 1 q j ). Then, the EM algorthm s used to compute teratvely the expected value of the complete data log-lkelhood functon Q(θ θ (k) ): Q(θ θ (k) ) = j ( W (k) log(θ j,dj,1) ) (4) + (1 W (k) ) log(θ j,dj,0) where θ j,dj,s corresponds to ether θ j0s or θ j1s dependng on the decson d j of the expert j at the voxel. W (k) s the probablty that, at teraton k, the voxel of the reference standard T s labeled as 1. Usng ths functon, we now derve the observed Informaton Matrx of the parameters θ. 1) Dervaton of the Expected Complete Data Informaton Matrx: Ths matrx, denoted I c (θ), s expressed as the second dervatves of the expected value of the complete data loglkelhood functon [8],.e. 2 I c (θ) = θ θ T Q(θ θ(k) ) (5) Eq. (4) and Eq. (5) demonstrate that the non-dagonal terms of I c are zero as the parameters are ndependent of each other. Therefore, I c s a dagonal matrx composed of the followng terms: I c;pj = I c;qj = W (k) θ 2 j,d j,1 1 W (k) θ 2 j,d j,0 2) Dervaton of the Expected Mssng Data Informaton Matrx: Once I c has been computed, the observed Informaton Matrx s obtaned by subtractng from I c the expected mssng data Informaton Matrx I m. Computng ths matrx s generally more dffcult than computng the expected complete data Informaton Matrx. When no analytcal expresson can be derved, t can be estmated usng the EM algorthm tself to compute the Jacoban matrx va numercal dfferentaton (see [11], [8]). In the general case of any EM algorthm, an analytc expresson of I m may also be obtaned by the followng equaton [14] f the requred dervatves exst: I m (θ) = 2 Q(θ θ (k) ) θ T (8) In the case of the STAPLE algorthm, the expected value of the complete data log-lkelhood functon Q(θ θ (k) ) can be dfferentated. We have therefore derved the analytc expresson of I m elements as follows: θ jtt nss = ( 1) 1+d j θ j,dj,t W (k) nss where t and s are ether 1 or 0, to derve the expressons for p j and q j. Ths expresson gves I m as a functon of the dervatves of the probabltes of the reference standard W (k). These W (k) have been derved by Warfeld et al. [7] as: (6) (7) (9) W (k) f(t = 1) j = θ(k) j,d j,1 1 m=0 (f(t = m) ) (10) j θ(k) j,d j,m For smplcty of notaton, we wll consder that the pror probablty f(t = 1), respectvely f(t = 0), s constant over the entre mage and wll abbrevate t by π 1, respectvely π 0. However, all the derved expressons are stll vald for spatally varyng pror probabltes by replacng π m n the followng equatons by π m (). Knowng the expresson of W, ts dervatve wth respect to the expert parameters p (k) n and q n (k) = θ (k) n00 can be derved: W (k) nss = ( 1) 1+dn π 0 π 1 ( l n θ(k) l,d l,s ) ( l θ(k) = θ (k) n11 l,d l,1 s ) 2 ( 1 m=0 π m l θ(k) l,d l,m (11) where s s ether 0 or 1. Therefore, the expected mssng data Informaton Matrx, defned n Eq. (8), can be computed by substtutng Eq. (11) nto Eq. (9). In practce, these values are )

4 4 computed easly by evaluatng the dfferent expressons at each voxel. D. An Extenson to Mult-Category Labels We now present an extenson of the computaton of the observed Informaton Matrx to the mult-category STAPLE. We therefore now consder that each expert delneates L structures labeled from 0 to L 1. Each expert s also assocated wth an L L matrx of parameters: θ j, as explaned n Secton II-A. In ths case, the expected value of the complete data log-lkelhood functon Q s expressed as follows (see [7]): Q(θ θ (k) ) = j W (k) s log(θ j,dj,s) (12) As n the bnary case, the performance parameters are related by the constrant that s θ js s = 1. Ths constrant on the sum of the parameters on each row of the performance parameter matrx ensures that for L labels, there are only L 1 free varables. In the bnary case, t was straghtforward to select the senstvty and specfcty parameters as the varables to compute the bounds for. In the mult-category case, t s agan possble to compute bounds on the L (L 1) free parameters n each row, but ths mples selectng one of the varables n each row as a fxed parameter entrely determned by the row constrant. Rather than arbtrarly select any one parameter n each row n ths way, we have preferred to estmate the bounds for all L L varables and to not utlze the row sum constrant to reduce the number of parameters. 1) Dervaton of the Expected Complete Data Informaton Matrx: The analytcal expresson of I c (θ) can be obtaned from Eq. (5) and s expressed from the second dervatves of the expected value of the complete data log-lkelhood Q wth respect to θ. Agan, only the dagonal terms are not zero due to the ndependence of the performance parameters and I c s therefore composed of the terms: I c;θjs s (θ) = s (k) Ws θ 2 :d j =s js s (13) 2) Dervaton of the Expected Mssng Data Informaton Matrx: Once the expected complete data Informaton Matrx s derved, we need to subtract the expected mssng data Informaton Matrx from t to obtan the observed Informaton Matrx of the parameters. We derved the analytcal expresson of I m from the general equaton proposed n [14] for a general EM algorthm (rewrtten n Eq. (8)). In the mult-category case, these second dervatves are expressed as follows: As for the bnary case, ths requres to derve the expresson θ js s (k) Ws 1 :d j=s θ js s = (14) W (k) s expresson of W (k) s for all parameters. Frst, we know from [7] the as a functon of θ (k) parameters: W (k) s = m π s j θ(k) j,d j,s (π m j θ(k) j,d j,m ) (15) where π s correspond to the pror probablty of havng the structure s. From Eqs. (14) and (15), a frst observaton can be made on the dervatves to be computed: f d n t, then W (k) s = 0. Otherwse, two cases arse: t = s (both the numerator and denomnator depend on θ (k) ) and t s (only the denomnator depends on θ (k) ). These two cases lead to the followng expressons. If t = s, the dervatve s expressed as follows: W (k) s ( ( ) π t l n θ(k) l,d l,t) m t π m l θ(k) l,d l,m = ( m π ) 2 (16) m l θ(k) l,d l,m If t s, then the equaton changes slghtly: W (k) s ( ) ( ) π t π s l n θ(k) l,d l,t l θ(k) l,d l,s = ( m π ) 2 (17) m l θ(k) l,d l,m By substtutng Eqs. (16) and (17) nto Eq. (14), we are then able to compute the mssng data Informaton Matrx and therefore the observed Informaton Matrx of the parameters. 3) Relatonshp between Mult-Category and Bnary Segmentaton Formulaton: As mentoned above, the assumpton of ndependence between the parameters s not true because of the constrant on some parameters to sum up to 1: s θ js s = 1. There does not exst to our knowledge a way to take nto account ths nterdependency n the computaton of the observed Informaton Matrx. The two dervatons presented n ths paper are therefore dfferent from each other. However, these two expressons are stll related. If we consder n the bnary case the full 2 2 matrx of parameters as ndependent, then the expresson of the observed Informaton Matrx wll be the mult-category expresson. Conversely, consderng the mult-category expresson n the case of L = 2, f we consder the off-dagonal terms as exact (whch can be done as they are entrely determned by the dagonal terms), then the formulaton of the observed Informaton Matrx s exactly the bnary case expresson. In the mult-category case, there s no clear choce for reducng the number of free varables, and we prefer to compute the bounds for all of the varables. III. RESULTS To llustrate our formulaton for dervng confdence ntervals of the estmated segmentaton parameters, we wll present two applcatons. Frst, we demonstrate the estmaton of nferental uncertanty of the values of the parameters estmated from a dataset of smulated mages. Then, we present the applcaton of our framework to obtan confdence ntervals for the performance parameters on a manually segmented neonate database.

5 5 Segmentaton # Data Sens. (est. ; true) Spec. (est. ; true) [LB;UB] [LB;UB] ; ; [ ; ] [ ; ] ; ; [ ; ] [ ; ] ; ; [ ; ] [ ; ] ; ; [ ; ] [ ; ] ; ; [ ; ] [ ; ] ; ; [ ; ] [ ; ] ; ; [ ; ] [ ; ] ; ; [ ; ] [ ; ] ; ; [ ; ] [ ; ] ; ; [ ; ] [ ; ] Data Sens. (est. ; true) Spec. (est. ; true) [LB;UB] [LB;UB] ; ; [ ; ] [ ; ] ; ; [ ; ] [ ; ] ; ; [ ; ] [ ; ] ; ; [ ; ] [ ; ] ; ; [ ; ] [ ; ] ; ; [ ; ] [ ; ] ; ; [ ; ] [ ; ] ; ; [ ; ] [ ; ] ; ; [ ; ] [ ; ] ; ; [ ; ] [ ; ] TABLE I E VALUATION OF THE EXPERT PARAMETERS CONFIDENCE INTERVALS ON SIMULATED IMAGES. E XPERIMENTS WITH SIMULATED SEGMENTATIONS SHOWING THE ESTIMATED PARAMETERS ( EST.) FOR EACH SEGMENTATION, THE TRUE VALUE OF THE PARAMETER ( TRUE ), AND THE CONFIDENCE INTERVAL AT A 95 % LEVEL ([LB;UB]) ESTIMATED BY OUR NEW ALGORITHM. R ESULTS ARE SHOWN FOR SENSITIVITY (S ENS.) AND SPECIFICITY (S PEC.). T HE KNOWN ( TRUE ) PARAMETERS FALL WITHIN THE CONFIDENCE INTERVAL OF THE ESTIMATED PARAMETERS. R ESULTS ARE PRESENTED FOR AND IMAGES, SHOWING AN INCREASE IN THE UNCERTAINTY WHEN DECREASING THE SIZE OF THE IMAGES. A. Experments on Smulated Data 1) Impact of Data Sze and STAPLE Precson: To evaluate our algorthm wth respect to a known ground truth, we created a database of ten segmentatons (2D mages, sze ), llustrated n Fg. 2, dvded nto two groups. From the ground truth n Fg. 2(a), we smulated a frst group of 5 mages wth a senstvty parameter of 0.7 and a specfcty parameter of 0.8 (llustrated n mage (b)). Then, a second group, llustrated n mage (c), was generated wth dfferent parameters: senstvty and specfcty of 0.9. In order to evaluate the nfluence of the mage sze on the confdence ntervals of the parameters estmates, we have also generated a second database usng the same parameters but wth an mage sze of (a) (b) (c) Fg. 2. Database of Smulated Images. Smulated mages used for the valdaton of our confdence ntervals estmaton method : (a): orgnal segmentaton, (b): smulated segmentaton of group 1 (senstvty: 0.7, specfcty: 0.8), (c): smulated segmentaton of group 2 (senstvty: 0.9, specfcty: 0.9). We have then run STAPLE to convergence (so that θ(k+1) θ(k) ) on the mages of both databases to estmate a reference standard and utlzed our algorthm to estmate the confdence ntervals of the values of the parameters. The results are presented n Table I for the two databases. Our frst observaton was that the non dagonal terms of the covarance matrx were always much smaller than the dagonal terms. Ths comes from the fact that only the expected mssng data Informaton Matrx Im s non-dagonal. If the reference standard was known, then the covarance matrx would be computed only as the expected complete data Informaton Matrx Ic, whch s dagonal (see Eqs. (4) and (5)). In the STAPLE algorthm, the reference standard s not known and ths leads to non zero off-dagonal terms. The fgures n Table I show that all the true values of the parameters (senstvty and specfcty) fall wthn the 95% confdence nterval around the estmated values of the parameters. Dervng the confdence bounds on these parameters therefore allows us to show that the estmaton performed by STAPLE s very precse. The second observaton that can be made on these fgures s on the nfluence of the mage sze on the confdence bounds of the parameters. Our experments ndeed show a clear correlaton between the mage sze and the wdth of the confdence nterval, whch s ncreasng when the mage sze s smaller. 2) Impact of the Performance Parameters Intalzaton: To evaluate the nfluence of ntalzaton on the estmated parameters and the confdence ntervals, we have used a database

6 6 Seg # True Sens True Spec Intalzaton Intalzaton 0.3 Sens. (est.) Spec. (est.) Sens. (est.) Spec. (est.) [LB;UB] [LB;UB] [LB;UB] [LB;UB] [ ; ] [ ; ] [ ; ] [ ; ] [ ; ] [ ; ] [ ; ] [ ; ] [ ; ] [ ; ] [ ; ] [ ; ] [ ; ] [ ; ] [ ; ] [ ; ] [ ; ] [ ; ] [ ; ] [ ; ] [ ; ] [ ; ] [ ; ] [ ; ] [ ; ] [ ; ] [ ; ] [ ; ] [ ; ] [ ; ] [ ; ] [ ; ] [ ; ] [ ; ] [ ; ] [ ; ] [ ; ] [ ; ] [ ; ] [ ; ] TABLE II EVALUATION OF THE INFLUENCE OF INITIALIZATION ON PERFORMANCE PARAMETERS ESTIMATES AND CONFIDENCE INTERVALS. EXPERIMENTS WITH SIMULATED DATA FOR TWO DIFFERENT INITIALIZATIONS OF THE PARAMETERS (0.3 AND ). RESULTS SHOW THE ESTIMATED PARAMETERS (EST.) FOR EACH SEGMENTATION, THE TRUE VALUE OF THE PARAMETER, AND THE CONFIDENCE INTERVAL AT A 95 % LEVEL ([LB;UB]) ESTIMATED BY OUR NEW ALGORITHM. RESULTS ARE SHOWN BOTH FOR SENSITIVITY (SENS.) AND SPECIFICITY (SPEC.). THIS TABLE SHOWS THAT THE CONFIDENCE INTERVALS DO NOT DEPEND STRONGLY ON THE INITIALIZATION BUT THE ESTIMATED VALUES OF THE PERFORMANCE PARAMETERS DO. of ten segmentatons (2D mages, sze ) based on the same ground truth as above. In ths experment, we have generated 9 mages wth relatvely low qualty segmentatons (senstvty and specfcty at 0.3) and one wth good qualty (senstvty of 0.8 and specfcty of 0.9). Then, we ran STAPLE on ths database wth two dfferent ntalzatons: one close to the true parameters (all estmates are ntalzed at 0.3) and one where we suppose all experts are good (all parameters ntalzed at ). We present the results of these experments and the confdence ntervals estmated n Table II. Ths table clearly shows an nfluence of the ntalzaton on the estmated performance parameters. When the parameters are ntalzed far away from ther true values (0.9999), the estmated parameters converge to erroneous values for all experts. On the contrary, when the parameters are better ntalzed (all at 0.3), the algorthm converges to values close to the true senstvtes and specfctes (whch are ncluded n the confdence ntervals around the estmated performance parameters). Another very mportant result s that, despte ths great change n the estmated values, the confdence nterval wdths are very smlar. Ths demonstrates that our formulaton for the computaton of the confdence ntervals estmates how precse the estmaton of the parameters s, not the actual accuracy of these estmates. B. Evaluaton of Inferental Uncertanty of Parameters on a Neonate Database 1) Image Database: We have appled our algorthm to fve datasets of neonate MRI segmentatons (one of them llustrated n Fg. 3) selected from MRI scans from prevous studes. Each of these datasets conssted of a T1 and a T2 weghted mage. After regstraton of the T2 mage on the T1 mage, fve tssue classes were delneated nteractvely on one slce: cortcal gray matter, sub-cortcal gray matter, unmyelnated whte matter, myelnated whte matter and cerebrospnal flud (CSF). Ths process was repeated fve tmes by three experts so that for each newborn MRI, 15 segmentatons nto fve structures were avalable. 2) Evaluaton of the Confdence Bounds of the Parameters: To evaluate ntra-expert segmentaton varablty, we have used STAPLE for each patent on the fve segmentatons of one expert to determne the reference standard for ths expert, together wth parameters of senstvty and specfcty for each manual segmentaton. We have then used our analytcal formulaton to effcently compute the observed Informaton Matrx for these parameters, and evaluated the covarance matrx of the parameters by smply nvertng the Informaton Matrx. We computed the confdence ntervals of the parameters usng the bnary case formulaton separately on all patents and all structures. We only present n Fg. 4 a representatve

7 7 Unmyelnated Whte Matter Senstvty (a) Unmyelnated Whte Matter Specfcty (b) Gray Matter Senstvty (c) Gray Matter Specfcty (d) Fg. 4. Confdence bounds of the senstvty and specfcty parameters. Expert parameters and ther confdence ntervals ((a, c): Senstvty, (b, d): Specfcty) for the whte matter segmentaton (a, b) and the gray matter segmentaton (c, d). Each chld segmentatons were treated separately, each column for each chld represents an expert s segmentaton. The results on fve datasets (each column of each graph) show that the confdence ntervals of the estmated senstvtes and specfctes are very tght. (a) (b) (c) (d) (e) (f) Fg. 3. Illustraton of one mage from the database. Coronal slce of (a) newborn T1 MRI and (b-f) ts repeated manual segmentaton n 5 classes done by one expert (cortcal gray matter - grey, sub-cortcal gray matter - whte, unmyelnated whte matter - red, myelnated whte matter - orange - and CSF - blue). Other mages n the database were smlar to ths specfc example. example of the results on the unmyelnated whte matter and the gray matter for fve patents usng the fve segmentatons of one expert (each cluster n the fgure llustrates ndependent experments on each patent), showng for each performance parameter ts confdence nterval as an error bar. Ths fgure shows that even wth only fve segmentatons to estmate the reference standard, the estmaton of the expert performance parameters s stll very precse. The maxmum relatve standard devaton s ndeed of 1.3 %. 3) Influence of the Number of Voxels on the Confdence Intervals: We also wanted to confrm wth data from a subject prevous results on smulated data on the nfluence of mage sze on the confdence ntervals of the performance parameters. To ths end, we subsampled the segmentatons of one patent. Because the subsamplng s done usng nearest neghbor nterpolaton, the subsamplng amounts to takng one row and column every two n the mage. We then ran STAPLE untl convergence on the subsampled segmentatons of one expert for one patent and computed the confdence ntervals on the parameters. We present n Fg. 5 the results of senstvty, specfcty and confdence ntervals (as error bars) on a patent n ts orgnal resoluton (n blue), subsampled once (n red) and twce (n green). Frst, we can see on some experts that the confdence ntervals of ther parameters become 0 when the mage sze s dvded by 4 n each drecton. Ths s due to the fact that the mage becomes so small that the whole regon of nterest for a gven expert s only composed of the delneated structure, thereby removng the varablty for the correspondng expert parameter. Apart from ths effect, these results confrm a clear nfluence of the mage sze on the parameters bounds.

8 8 (a) Senstvty (b) Specfcty Fg. 5. Influence of the mage dmenson on confdence bounds of the parameters. 95 % confdence ntervals on the estmated values of the senstvty (a) and specfcty (b) parameters for the mage at orgnal sze (blue), subsampled once (red), and subsampled twce (green). These show a decrease n the confdence n the estmated parameters as the mage s subsampled, reflectng that the confdence n the estmates decreases when the amount of avalable data s reduced. (a) Senstvty (b) Specfcty Fg. 6. Influence of the number of experts on the confdence ntervals of the performance parameters. Average relatve confdence ntervals values (n percent of the average performance parameter) as a functon of the number of experts used n STAPLE. For each number of experts, all combnatons of K experts among the 15 avalable were used to compute the average. The three curves show the results usng: the whole mages (blue), half of the mages (red), and the upper left quarter of the mages (green) to compute the STAPLE performance estmates. The standard devatons nearly double when the mage s subsampled. 4) Influence of the Number of Experts on the Confdence Intervals: Another potental cause of uncertanty of the estmated values of the parameters s the number of segmentatons used to compute the reference standard. We have studed ths property usng bnary segmentaton performance estmates on 15 manual segmentatons of one subject. We present the evaluaton of the results usng from 3 segmentatons up to 10 segmentatons. For each number K of manual segmentatons, we have performed the study over all the combnatons of K mages among the 15 avalable. We present n Fg. 6 the average relatve values (n percent of the average performance parameter) of the 95% confdence ntervals for each number of experts for senstvty and specfcty parameters. These results show that the relatve confdence nterval decreases rapdly wth the number of experts, and s stable for more than fve experts. Moreover, we also present n ths fgure three curves, usng only part of the mages to estmate the performance parameters (green: a quarter of the mage, red: half of the mage, and blue: the whole mage). Ths suggests that, usng 4 or more experts, the sze of the structure to be delneated as well as the sze of the regon of nterest for the STAPLE computaton s more nfluental upon the confdence bounds of the estmated parameters than the number of experts. Overall, both these aspects should be taken nto account when desgnng a valdaton study to ensure enough experts and a suffcent regon have been delneated to get precse estmates of the performance parameters for each expert. 5) Evaluaton of the Mult-Category Case Algorthm: Fnally, we present an applcaton of our algorthm for the multcategory case of STAPLE. The results have been computed on all structures and all patents but for clarty, we present the results on only 5 repeated segmentatons of three structures from one expert: the cortcal gray matter, the sub-cortcal gray matter and the unmyelnated whte matter. We have then run STAPLE on these segmentatons usng the multcategory case mplementaton (usng 4 classes: 3 structures plus the background). We present n Table III the results of

9 9 BG (θ j,bg,bg ) CGM (θ j,cgm,cgm ) UWM (θ j,uwm,uwm ) SCGM (θ j,scgm,scgm ) Seg. 1 Estmate Seg. 1 95% CI [ ; ] [ ; ] [ ; ] [ ; 1.0] Seg. 2 Estmate Seg. 2 95% CI [ ; ] [ ; ] [ ; 1.0] [ ; 1.0] Seg. 3 Estmate Seg. 3 95% CI [ ; ] [ ; ] [ ; ] [ ; 1.0] Seg. 4 Estmate Seg. 4 95% CI [ ; ] [ ; ] [ ; ] [ ; 1.0] Seg. 5 Estmate Seg. 5 95% CI [ ; ] [ ; ] [ ; ] [ ; 1.0] TABLE III EVALUATION OF THE MULTI-CATEGORY CONFIDENCE INTERVALS ALGORITHM. ESTIMATED PERFORMANCE PARAMETERS VALUES AND THEIR CONFIDENCE INTERVALS (CI) OBTAINED USING OUR MULTI-CATEGORY ALGORITHM ON FIVE SEGMENTATIONS FROM ONE RATER. STUDIED STRUCTURES ARE: BG: BACKGROUND, CGM: CORTICAL GRAY MATTER, UWM: UNMYELINATED WHITE MATTER, SCGM: SUB-CORTICAL GRAY MATTER. our algorthm, showng only the estmated values and 95% confdence ntervals of the dagonal parameters,.e. the θ jss, as showng the results for all parameters would produce a very large table. The mult-category bounds estmate enables us to determne the precson of a rater performance estmate. In ths precse example, the relatve standard devatons of the expert performance parameters are very tght, varyng between 1.7 % and 4.5 % of the respectve estmated parameters values, showng that the values estmated by STAPLE are also precse n the mult-category case. The estmaton of the confdence ntervals of the mult-category performance parameters wll allow the determnaton of the mnmal mage sze and the number of experts necessary to acheve a chosen level of precson n segmentaton performance assessment. IV. CONCLUSION We have presented n ths artcle the expresson of confdence ntervals of the expert performance parameters obtaned usng the STAPLE valdaton method, both n the bnary and the multple category case. These formulatons are based on the dervaton of analytc expressons for the observed Informaton Matrx of the underlyng Expectaton-Maxmzaton algorthm. Such confdence bounds wll be very mportant for future studes as they wll ad n the nterpretaton of the performance of segmentaton generators, and n determnng the mnmal sze and number of segmentatons to precsely characterze the performance parameters. We have presented examples of the applcaton of these expressons for the evaluaton of the nferental uncertanty of the expert parameters n experments on smulated mages, showng that the true values of the expert performance parameters fall wthn the confdence ntervals of the estmated values of the parameters. We have also utlzed these expressons n the context of neonate bran segmentaton, showng a dependence of the confdence ntervals wth respect to the number of voxels n the regon of nterest for the segmentaton. Moreover, we have also shown that the number of experts used n the study may nfluence the uncertanty of the estmated parameters. In our partcular case, we have shown that, ndependently of the sze of the segmentaton, the uncertanty of the parameters s stable when 5 or more experts are used n the study. These experments provde an mportant nsght on the desgn of future experments for segmentaton valdaton. It wll ndeed be very mportant to have as many experts as possble when comparng small segmentatons, n order to mnmze the potentally large uncertanty on the values of the estmated parameters. Otherwse, f the structure of nterest s large enough, usng a small number of experts wll not affect the nferental uncertanty n the values of the performance parameters. Fnally, we have presented experments llustratng the mult-category formulaton of the confdence bounds computaton. These confdence bounds are useful for the desgn of future segmentaton comparson experments. These expressons may then have many other applcatons n terms of valdaton of segmentaton or evaluaton of ntraexpert segmentaton varablty n a clncal context. In addton to provdng gudance n the nterpretaton of the parameters determned by the STAPLE valdaton algorthm, ths work could be used n the future for the development of a spatally localzed STAPLE algorthm by computng performance parameters estmates n a blockwse manner. The bounds estmated wth the algorthm descrbed here would allow us to determne the mnmal sze of the regon of nterest requred to obtan precse parameter estmates for a gven structure. Future work wll then examne usng ths approach to evaluate spatally varyng performance parameters and ther bounds. ACKNOWLEDGMENTS Ths nvestgaton was supported n part by a research grant from CIMIT, grants RG 3478A2/2 and RG 4032A1/1 from the NMSS, and by NIH grants R03 EB008680, R01 RR021885, R01 GM074068, R01 EB and P30 HD REFERENCES [1] D. Huttenlocher, D. Klanderman, and A. Rucklge, Comparng mages usng the Hausdorff dstance, IEEE Transactons on Pattern Analyss and Machne Intellgence, vol. 15, no. 9, pp , Sep

10 [2] V. Chalana and Y. Km, A methodology for evaluaton of boundary detecton algorthms on medcal mages, IEEE Transactons on Medcal Imagng, vol. 16, no. 5, pp , [3] L. Dce, Measures of the amount of ecologc assocaton between speces, Ecology, vol. 26, no. 3, pp , [4] P. Jaccard, The dstrbuton of flora n the alpne zone, New Phytologst, vol. 11, pp , [5] K. H. Zou, S. K. Warfeld, A. Bharatha, C. M. C. Tempany, C. Tempany, M. R. Kaus, S. J. Haker, W. M. Wells, F. A. Jolesz, and R. Kkns, Statstcal valdaton of mage segmentaton qualty based on a spatal overlap ndex, Acad Radol, vol. 11, no. 2, pp , Feb [6] G. Gerg, M. Jomer, and M. Chakos, VALMET: A new valdaton tool for assessng and mprovng 3D object segmentaton, n MICCAI, ser. LNCS, vol. 2208, 2001, pp [7] S. K. Warfeld, K. H. Zou, and W. M. Wells, Smultaneous truth and performance level estmaton (STAPLE): an algorthm for the valdaton of mage segmentaton, IEEE Transactons on Medcal Imagng, vol. 23, no. 7, pp , July [8] G. McLachlan and T. Krshnan, The EM Algorthm and Extensons. John Wley and Sons, [9] O. Commowck and S. K. Warfeld, Estmaton of nferental uncertanty n assessng expert segmentaton performance from STAPLE, n Proceedngs of the 21st Internatonal Conference on Informaton Processng n Medcal Imagng, ser. LNCS, vol. 5636, 2009, pp [10] A. Dempster, N. Lard, and D. Rubn, Maxmum lkelhood from ncomplete data va the EM algorthm, Journal of the Royal Statstcal Socety, vol. 39 (Seres B), [11] X. Meng and D. Rubn, Usng EM to obtan asymptotc varancecovarance matrces: the SEM algorthm, Journal of the Amercan Statstcal Assocaton, vol. 86, pp , [12] R. Nsh, Encyclopaeda of Mathematcs. Kluwer Academc Publshers, 2001, ch. Box-Cox Transformaton. [13] W. Meeker and L. Escobar, Statstcal Methods for Relablty Data. John Wley & Sons, [14] D. Oakes, Drect calculaton of the nformaton matrx va the EM algorthm, J. R. Statstcal Socety, vol. 61, no. 2, pp ,