Learning-Based Quality Control for Cardiac MR Images

Transcription

1 1 Learning-Baed Quality Control for Cardiac MR Image Giacomo Tarroni, Ozan Oktay, Wenjia Bai, Andrea Schuh, Hideaki Suzuki, Jonathan Paerat-Palmbach, Antonio de Marvao, Declan P. O Regan, Stuart Cook, Ben Glocker, Paul M. Matthew, Daniel Rueckert arxiv: v2 [c.cv] 15 Sep 2018 Abtract The effectivene of a cardiovacular magnetic reonance (CMR) can depend on the ability of the operator to correctly tune the acquiition parameter to the ubject being canned and on the potential occurrence of imaging artefact uch a cardiac and repiratory motion. In the clinical practice, a quality control tep i performed by viual aement of the acquired image: however, thi procedure i trongly operatordependent, cumberome and ometime incompatible with the time contraint in clinical etting and large-cale tudie. We propoe a fat, fully-automated, learning-baed quality control pipeline for CMR image, pecifically for hort-axi image tack. Our pipeline perform three important quality check: 1) heart coverage etimation, 2) inter-lice motion detection, 3) image contrat etimation in the cardiac region. The pipeline ue a hybrid deciion foret method - integrating both regreion and tructured claification model - to extract landmark a well a probabilitic egmentation map from both long- and hort-axi image a a bai to perform the quality check. The technique wa teted on up to 3000 cae from the UK Biobank a well a on 100 cae from the UK Digital Heart Project, and validated againt manual annotation and viual inpection performed by expert interpreter. The reult how the capability of the propoed pipeline to correctly detect incomplete or corrupted can (e.g. on UK Biobank, enitivity and pecificity repectively 88% and 99% for heart coverage etimation, 85% and 95% for motion detection), allowing their excluion from the analyed dataet or the triggering of a new acquiition. Index Term Image quality aement, Magnetic reonance imaging, Motion compenation and analyi, Heart I. INTRODUCTION CARDIOVASCULAR magnetic reonance (CMR) imaging preent a wide variety of different application for the anatomical and functional aement of the heart. The ucce of a CMR acquiition relie, however, on the ability of the MR operator to correctly tune the acquiition parameter to the ubject being canned [1]. Moreover, CMR can be negatively affected by a long lit of imaging artefact (caued for intance by repiratory and cardiac motion, blood flow and magnetic field inhomogeneitie) [2]. Therefore, a quality control tep i required to ae the uability of the acquired G. Tarroni i with the Department of Computing, Imperial College London, SW7 2AZ London, UK, giacomo.tarroni@gmail.com. O. Oktay, W. Bai, A. Schuh, J. Paerat-Palmbach, B. Glocker and D. Rueckert are alo with the Department of Computing, Imperial College London. H. Suzuki and P. M. Matthew are with the Diviion of Brain Science, Faculty of Medicine, Imperial College London. P. M. Matthew i alo with the UK Dementia Reearch Intitute, London, UK. A. de Marvao, D. P. O Regan and S. Cook are with the MRC London Intitute of Medical Science, Faculty of Medicine, Imperial College London. Thi work ha been ubmitted to the IEEE for poible publication. Copyright may be tranferred without notice, after which thi verion may no longer be acceible. Optimal Non-optimal Heart Coverage Inter-lice Motion Image Contrat Fig. 1: Potential iue affecting CMR image acquiition. In the firt two column, the uperimpoition of long-axi twochamber view (red) and hort-axi tack (gray) i hown, while, in the lat one, hort-axi lice are diplayed. image. In the clinical practice thi tep i performed by viual inpection, uually carried out by the ame operator who et up the acquiition, thu leading to highly ubjective reult. In the lat decade, everal initiative for the acquiition of open acce large-cale population tudie have been launched. For example, the UK Biobank (UKBB) i a population-baed propective tudy, etablihed to allow detailed invetigation of the genetic and non-genetic determinant of the dieae of middle and old age. Of the 500,000 ubject enrolled in the tudy, CMR will be collected from 100,000 of them [3]. At the time of ubmiion the acquiition i ongoing, with cloe to 20,000 ubject already canned. Together with thi trend toward the implementation of large-cale multicentre imaging dataet, the need for fat and reliable quality control technique for CMR image ha become evident, a highlighted alo by everal tudie aiming to define tandardized criteria for thi tak [4]. In thi cenario, quality control through viual inpection i not only ubjective, but imply infeaible due to the very high throughput demanded by the acquiition pipeline. On the other hand, failure to correctly identify corrupted or unuable image could affect the reult of automated analyi performed on the dataet, with undeirable effect. Conequently, the need for fully automated quality control pipeline for CMR image ha arien. Many reearch effort have been dedicated to the automated identification of quality metric from MR image. Mot of

2 2 thee effort have focued on the automated etimation of noie level [5], [6]. Still, many apect related to the uability of the acquired image are inherently modality-pecific. Several automated pipeline for quality control have been propoed for brain MR imaging [7]. However, to our knowledge, no comprehenive automated quality control pipeline have been propoed o far for CMR image, in particular for the hortaxi (SA) cine image tack, which are the reference image for the tructural and functional aement of the heart. One crucial apect of the acquiition of SA image tack i that it require the MR operator to identify the direction of the left ventricular (LV) long axi - the line going from the apex to the centre of the mitral valve - and to define a region of interet: the correct planning will generate a SA tack encompaing both thoe landmark with lice perpendicular to the LV long axi. If thi election i incorrect, the acquired SA tack may include an inufficient number of SA lice to fully cover the LV (ee firt column of Fig. 1). A a conequence, any functional analyi performed on the tack (e.g. ventricular volume etimation) may be compromied. Another important apect involved in CMR acquiition i that SA cine tack are generated during multiple breath-hold (with uually 1-3 lice acquired per each breath-hold). Although the ubject are intructed to hold their breath at the ame breath-holding poition, in practice the heart location can vary coniderably. If the difference between the breath-holding poition are too pronounced, the acquired image tack will be affected by inter-lice motion and thu will not correctly repreent the cardiac hape, introducing potential error in the following analye and viualization (ee econd column of Fig. 1). Finally, the contrat of the obtained CMR image i directly affected by the choen acquiition parameter (a well a by potential artefact). If the different tructure of the heart are not properly contrated, the aement of the cardiac function can be hampered (ee third column of Fig. 1). In thi paper, we preent a fully-automated, learning-baed quality control technique for CMR SA image tack. Our approach ue a hybrid deciion foret method to extract at once both landmark poition (LM) and probabilitic egmentation map (PSM) from long-axi (LA) and SA image. LM and PSM are then ued to perform three quality check: 1) heart coverage etimation, 2) inter-lice motion detection, 3) image contrat etimation in the cardiac region. Our hybrid foret method i thu not intended a a novel technique for landmark detection and egmentation per e, but rather a an integral component of our pipeline. The extraction of LM from multiple LA view and the probabilitic nature of PSM allow to ae the reliability of the pipeline for each can uing dedicated anity check. The technique wa teted on two dataet (up to 3000 cae from the UKBB tudy and 100 cae from the UK Digital Heart Project [8], UKDHP) and validated againt manual annotation and viual inpection. II. RELATED WORK To the bet of our knowledge, differently from brain MRI [7], no comprehenive quality control technique for cardiac CMR image have been reported in the literature. One of the few tudie in thi direction ha been recently preented by Alba et al. [9], who however focued on aeing egmentation quality rather than image quality. On the other hand, automated heart coverage etimation alone ha been the aim of everal tudie. Zhang et al. [10], [11] propoed to ue convolutional neural network (CNN) to perform lice claification in order to detect the preence or abence of the baal and apical lice. In their firt work [10] they propoed a 2D CNN trained on UKBB data, while in their more recent one [11] they improved their previou reult by uing a generative adverarial network. Differently from thee technique, our approach to heart coverage etimation i baed on the detection of landmark: in our previou preliminary work [12], we propoed a deciion foret method to detect the cardiac apex and the mitral valve on long-axi 2-chamber (LA 2CH) view image, and ued the poition of thee landmark with repect to the pace encompaed by the acquired tack to etimate the coverage. The technique wa applied to 3000 cae extracted from the UKBB, and wa able to detect SA tack with inufficient coverage with relatively high accuracy. Motion detection and modeling in the thoracic area ha been a highly invetigated ubject for more than a decade [13]. A far a inter-lice repiratory motion in CMR i concerned, mot of the approache reported in the lat decade have focued on motion correction rather than motion detection [14], [15], [16], [17]. All of the cited tudie focued on the compenation of inter-lice motion and in the generation of a corrected SA tack by mean of rigid in-plane regitration. Unfortunately, however, repiration caue a complex roto-tranlation of the heart in all three dimenion [18]: while mot tranlation happen in the cranio-caudal direction (thu approximately almot perpendicularly to the long axi of the LV), big difference in ubequent breath-holding poition can caue out-of-plane motion, which would lead to an inaccurate repreentation of the heart in the tack. A a conequence, it i important to etimate the amount of motion occurred during the acquiition of the tack in order to decide whether there are the ground for the application of a motion correction technique or it i intead adviable to repeat the can (or exclude it from ubequent analye). In the pat, everal reearch effort have been made toward the correct quantification of ignal-to-noie (SNR) or contratto-noie (CNR) ratio in MR image [5]. However, modern acquiition technique making ue of parallel imaging produce image with patially-varying noie ditribution, rendering image-baed etimator unreliable [19]. To overcome thi limitation, more elaborate method have been propoed exploiting information about coil enitivity or recontruction coefficient [20]. Unfortunately, thee data are very often not available, making the etimation of noie, and conequently of SNR and CNR, practically unfeaible in mot cenario. At the ame time, image contrat between two object - imply defined a the difference between their ignal intenity - ha long been ued to determine their viual differentiability in the acquired MR image [21]. In CMR imaging, image with poor contrat between the LV cavity and myocardium can potentially hinder the aement of cardiac tructure and function: conequently, contrat etimation in the cardiac region can provide a ueful

3 3 Fig. 2: Overview of the propoed pipeline. Probabilitic egmentation map (PSM) and landmark poition (LM) are extracted from LA and SA image uing hybrid random foret and exploited to perform three eparate quality control check. metric for quality control purpoe, either triggering the ue of contrat-enhancing technique or a new acquiition. In thi paper, we preent a fully-automated, learning-baed quality control pipeline for CMR SA tack. The propoed approach build upon our previou work [12], which ued a hybrid deciion foret method [22] to extract LM from LA 2CH view image in order to perform heart coverage etimation. With repect to our previou approach a well a to tate-of-the-art technique, the main contribution of the preent work can be lited a follow: We preent the firt comprehenive, fat, fully-automated quality control pipeline pecifically deigned for CMR SA image tack. The check incorporated in the pipeline are 1) heart coverage etimation, 2) inter-lice motion detection, 3) image contrat etimation in the cardiac region. To the bet of our knowledge, motion detection and cardiac image contrat for the ake of quality control have not been invetigated before. A for heart coverage etimation, we build on our previouly publihed tudy [12] by extending LM extraction to all long-axi view. LM are then combined together to ubtantially increae the robutne and the reliability of thi quality check (for detail pleae refer to the Dicuion ection); We propoe a different implementation of the previouly publihed hybrid deciion foret [22] (adopted in our previou work [12]) which allowed the joint extraction of LM and probabilitic edge map (PEM). The new implementation (baed on a novel mapping) allow intead the extraction of LM and PSM: PSM are required to perform both inter-lice motion detection and cardiac image contrat etimation, and enable anity check to ae the reliability of the pipeline; We validate thi pipeline by applying it to up to 3000 cae extracted from the UKBB tudy and to 100 cae from the UKDHP, howing it accuracy and robutne in real world cenario. The pipeline could be both applied retropectively on large-cale dataet to improve the reliability of clinical tudie or deployed propectively at acquiition ite to allow almot real-time aement of the acquired can. III. METHODS The propoed quality control pipeline i ummarized in Fig. 2. All of the three quality control tep are baed on the information extracted by hybrid deciion foret model from the acquired image. Thi ection of the paper tart with a brief recap on the theory behind deciion foret and i followed by the decription of the implementation adopted in the propoed pipeline, which allow the joint extraction of LM and PSM. Finally, each pecific quality control tep i decribed in detail. A. Hybrid Deciion Foret A deciion tree conit of a combination of plit and leaf node arranged in a binary tree tructure [23]. Tree route a ample x X (in our cae an image patch) by recurively branching left or right at each plit node j until a leaf node k i reached, where the poterior ditribution p(y x) for the output variable y Y i tored. Each plit node j i aociated

4 4 LA 2CH Landmark Segmentation Sample Label Hybrid Foret Structured Claification Regreion Training Teting Tet Image PSM + LM Fig. 3: Hybrid random foret. During training, randomly extracted ample with aociated label - coniting of egmentation and vector diplacement - are fed to the foret, and the learnt aociation are tored in the leaf node. During teting, each ample extracted from the tet image i ent to the model, extracting both PSM and LM at once. with a binary plit function h(x, θ j ) {0, 1} defined by the et of parameter θ j : if h = 0 the node end x to the left, otherwie to the right. Uually h i a deciion tump, i.e. a ingle feature dimenion n of x i compared with a threhold τ: θ = (n, τ) and h(x, θ) = [x(n) < τ]. A deciion foret i an enemble of T independent deciion tree: during teting, given a ample patch x, the prediction of the different tree are combined into a ingle output by mean of an enemble model. During training, at each node the goal i to find the et of parameter θ j which maximize a previouly defined information gain I j, uually defined a I j = H(S j ) i {0,1} Si j / S j H(Sj i), where S j, Sj 0 and Sj 1 are repectively the training et (compriing of ample x and aociated label y) arriving at node j, leaving the node to the left and to the right. H(S) i the entropy of the training et, whoe contruction depend on the tak at hand (e.g. claification, regreion). Different type of node (maximizing different information gain) can be interleaved within a ingle tree tructure (hence named hybrid ) in order to perform multiple tak. A in previou approache [24], [22], in the propoed technique tructured claification node (aiming at the detection of an object cloe to the deired landmark, in our cae uually the LV cavity) and regreion node (aiming at landmark localization) are combined (ee Fig. 3). In particular, in the propoed framework, landmark localization i conditioned on the reult of the detection of the cavity [24]. Thi not only lead to the extraction of two different type of information (PSM and LM) with only one model, but improve landmark localization by implicitly incorporating complementary information about cardiac poition and hape. Structured Claification and PSM Extraction: Structured claification extend the concept of claification by uing tructured label for Y intead of integer label. In our cae, each label y Y (aociated with the image patch x) conit of a egmentation of the LV cavity within x. To train a tructured claification node it i neceary to find a way to cluter tructured label at each plit node into two ubgroup depending on a imilarity meaure. The olution to thi problem wa firt propoed by Dollar et al. [25] and conit of two tep. Firt, Y i mapped to an intermediate pace Z by mean of the function Π : Y Z where the ditance between label can be computed. Importantly, Π mut be choen o that imilar label y will be aociated with vector z cloe to each other with repect to the ditance defined in Z. Then, PCA i applied to the vector z to map the aociated label y into a binary et of label c C = {0, 1}, which i achieved by applying a binary quantization to the principal component of each z vector. Finally, the Shannon entropy can be adopted [25]: H c (S) = c C p(c)log ( p(c) ), (1) with p(c) indicating the empirical ditribution extracted from the training ubet at each node. In our previou work [22], thi approach ha been adopted for tructured label Y coniting of edge map (EM) highlighting the contour of the myocardium. In the cae of EM, the mapping Π can imply encode for each pair of pixel whether they belong to the ame egment in the label y or not: Π EM : z = [y(j 1 ) = y(j 2 )] j 1 j 2, (2) where j 1 and j 2 are indice panning every pixel in y [25]. The reulting long binary vector z (which ha a number of dimenion equal to the number of pixel pair in y) can be ued to compare thi particular label to the other one by imply computing the Euclidean ditance in Z. However, the ame choice for Π cannot be adopted for our tak, which aim at uing tructured label coniting of egmentation map

5 5 (SM) of the LV cavity. For example, let imagine two label y 1 and y 2, the former completely outide the LV cavity and the latter completely inide: uing the mapping Π EM, we would obtain z 1 = z 2, which contradict the requirement by which only imilar label will be mapped cloe to each other in Z. Conequently, we implemented a different mapping: Π SM : z =[y(j 1 ) = y(j 2 ) = 0]......[y(j 1 ) = y(j 2 ) = 1] j 1 j 2, which encode for each pair of pixel in y whether they are both equal to 0, whether they are both equal to 1 and then concatenate the two obtained binary vector. Thi formulation enure the proper computation of the ditance between label, and thu their clutering at each node baed on their imilarity. At the end of the training proce, the label ŷ tored in each leaf node i the one whoe ẑ i the medoid (i.e. that minimize the um of ditance to all the other z at the ame node). At teting time, each ample patch of the tet image i ent down each tree of the foret, and the egmentation map tored at each elected leaf node are averaged, producing a mooth egmentation map (PSM) of the LV cavity. The value in the PSM are actual probabilitie (proportional to the certainty in LV cavity detection), and can be ued to ae the reliability of the prediction. Of note, the introduced formulation for Π SM in Eq. 3 could be eaily extended to multi-label PSM generation by concatenating additional binary vector computed for each label c i and by performing a channel-baed averaging operation at teting time. Regreion and Landmark Detection: To train regreion node, it i neceary to aociate with each ample patch x an additional label D = (d 1, d 2,..., d L ), where d l repreent for each of the L landmark the N-dimenional diplacement vector from the patch centre to the landmark location. Intead of the Shannon entropy defined in Eq. 1, regreion node are trained by minimizing the determinant of the covariance matrix Λ(S) defined by the landmark diplacement vector: (3) H r (S) = 1 2 log( (2πe) d Λ(S) ). (4) Landmark poition are aumed to be uncorrelated, thu only the diagonal element of Λ(S) are ued in Eq. 4 [26]. The location prediction are tored at each leaf node k uing a parametric model following a N L-dimenional multivariate normal ditribution with d l k and Σl k mean and covariance matrice, repectively. At teting time, Hough vote map are generated for each landmark by umming up the poterior ditribution obtained from each tree for each patch (applying normalization factor) [24]. Auming that pixel belonging to the LV are more informative for cardiac landmark detection than background one, the PSM value for the LV cavity are ued for each patch a weighting factor during the generation of the L Hough vote map, effectively retricting voting right only to pixel likely to belong to the LV cavity itelf [22]. Finally, the location of a landmark i determined by identifying the pixel with the highet value on each Hough vote map. Model Training: Each patch x i repreented by everal feature: multi-reolution image intenity, hitogram of gradient (HoG) and gradient magnitude. For a detailed decription, pleae refer to [22]. The decribed hybrid random foret approach i ued to build five different model (I-V) for our application (ee Fig. 2): PSM etimation of LV cavity and LM extraction for apex and mitral valve for LA image, PSM of LV cavity and LV myocardium for SA tack. For LA 3CH and 4CH image (model II and III) only one mitral valve point i identified becaue in thee image the LV outflow tract of the aorta can partially occlude one ide of the mitral valve, making it localization inaccurate. Alo, the training of the model uing SA image (model IV and V) i performed by feeding the random foret with all the lice extracted from the SA image tack: conequently, at teting time, the model are applied to each lice of the tack independently. Algorithm 1: Heart Coverage Etimation Input landmark: Apex: l1 2CH, l1 3CH, l1 4CH Mitral Valve point: l2 2CH, l3 2CH, l2 3CH, l2 4CH Change coordinate ytem: 2CH 3CH 4CH Apex: ˆl 1, ˆl 1, ˆl 1 2CH 2CH 3CH 4CH Mitral Valve point: ˆl 2, ˆl 3, ˆl 2, ˆl 2 Compute median landmark: l a = median (ˆl2CH ) 3CH 4CH 1, ˆl 1, ˆl 1 l m = median (ˆl2CH ) 2CH 3CH 4CH 2, ˆl 3, ˆl 2, ˆl 2 with z-component l a and l m, repectively Extract SA tack extenion in the z direction: Apex: r a Bae: r m Compute coverage CV: ( ) max 0,min(r a,l a) max(r m,l m) CV = l a l m if (condition) r a r m otherwie l a l m (condition): r a < l a or r m > l m B. Heart Coverage Etimation Heart coverage i etimated exploiting the landmark identified on LA 2CH, 3CH and 4CH image uing the previouly trained hybrid foret model. The rationale i that a properly canned SA tack hould encompa the whole portion of pace between the apex and the mitral valve. A highlighted in Fig. 2, for a pecific ubject we identify three landmark for the apex (one per each LA image: l 2CH 1, l 3CH 1 and l 4CH 1 ) 2 ) and four for the mitral valve (l 2CH 2, l 2CH 3, l 3CH 2, l 4CH with value in the coordinate ytem of each repective LA image. Uing the orientation matrix extracted from the DICOM header of the acquired SA and LA image, it i poible to define the coordinate of thee landmark in the coordinate ytem of the SA tack itelf. Two new median landmark (l a and l m ) are then defined taking the median of the coordinate of the landmark for the apex and for the mitral valve, repectively, in the SA coordinate ytem. The extenion in the z direction (i.e. along the LV long axi) of the SA tack can be eaily computed from the lice thickne

6 6 and lice number, which are tored in the DICOM header of the tack itelf: the two extrema along thi direction are defined r a and r m, repectively. Finally, the relative coverage can be computed by comparing the relative poition along the z direction of l a and l m (i.e. the pace that i uppoed to be covered by the SA tack) to the portion of pace between r a and r m (i.e. the pace that i actually covered). The tep for coverage etimation are lited in Algorithm 1, including the formula for the computation of the coverage (under the aumption that the apex i located at higher z compared to the mitral valve). Importantly, thi technique can eamlely be applied even if only one LA image i available. Alo, while minor motion can occur between the acquiition of LA image and of the SA tack, it i generally negligible in the z direction (the only one influencing coverage) [18] and thu regitration procedure between thee image were found to be unneceary. Finally, a anity check i performed to detect cae in which landmark detection failed: for each LA view, when either of the relative ditance between the landmark wa greater or maller than reference value by a certain threhold, the landmark from that image were dicarded, and the automated coverage etimation wa performed only on the remaining landmark (if available). Algorithm 2: Inter-Slice Motion Detection Input PSM: LA image: P SM 2CH, P SM 3CH, P SM 4CH SA lice: P SM SA Cav, = (1,..., numslice) Perform rigid regitration of LA PSM to SA PSM: Output: P SM 2CH, P SM 3CH, P SM 4CH for = 1 to numslice do Reample LA PSM: Output: P SM 2CH, P SM 3CH Combine reampled LA PSM: LA comb Output: P SM Perform in-plane rigid regitration of P SM SA Cav LA comb to P SM : Output: Tranlation magnitude T end SA, P SM 4CH LA 2CH LA 3CH LA 4CH PSM PSM PSM PSM C. Inter-Slice Motion Detection Inter-lice motion detection relie on the PSM extracted from the acquired image. The rationale i that while LV cavity PSM of motion-corrupted SA lice are mialigned, PSM extracted from the LA image repreent ection of the true hape of the LV cavity and can conequently be ued a reference. Moreover, the amount of mialignment between the SA PSM and LA PSM can be ued a an indicator of motion. To perform thi aement, the LA PSM are initially rigidly regitered (by 3D tranlation only, uing um of quared difference a diimilarity metric) to the SA PSM tack to compenate for potential motion between different acquiition. Then, for each lice of the SA PSM tack, the three regitered LA PSM are reampled and combined into a ingle image (referred to a combined LA PSM) containing the ection of the LA PSM with repect to a pecific lice (ee Fig. 4). Finally, in-plane rigid regitration (by tranlation only, uing um of quared difference a diimilarity metric) i performed between each SA PSM lice and the aociated combined LA PSM, and the magnitude of the tranlation T ued a a metric for motion (i.e. difference in breath-holding poition). Of note, thi tep i performed only on the lice which are effectively covering the LV, condition aeed uing the LA LM a in Algorithm 1. The probabilitic nature of PSM allow the application of a anity check performed to detect lice with a failed PSM etimation: SA PSM lice (whoe value range between 0 and 1024) with a peak probability value below a uer-defined threhold are conidered unreliable, and thu their T dicarded. Alo, thi technique could be applied even if only two LA image were available. The tep for motion detection are lited in Algorithm 2. Slice Selection LA PSM comb Reampling SA PSM In-plane rigid regitration Reampling Color Map Reampling Fig. 4: Motion detection technique. For each lice of the SA tack, the correponding portion of pace in each LA PSM i reampled and combined, producing the aterik-haped LA PSM comb image. In-plane rigid regitration i then performed between each SA PSM and LA PSM comb, and the tranlation magnitude T ued a proxy for inter-lice motion for that lice. A color map, with the intenity of each lice proportional to the repective T, can be alo generated. D. Cardiac Image Contrat Etimation Cardiac image contrat i etimated uing the LV cavity and LV myocardium PSM extracted from the SA tack. The rationale i to tranform the PSM into hard egmentation (SM) and to ue them to etimate the difference between average pixel intenity in the LV cavity and in the LV myocardium. Each cavity PSM lice i threholded electing the N cav pixel with the highet probability value: thi will maximize the probability of meauring the intenity in the actual cavity. The ame happen to each myocardium PSM, threholded electing N myo pixel. To exclude potential puriou region

7 7 from the obtained egmentation, the average centroid for the cavity egmentation i computed among the different lice, and for each lice only the connected component cloet to the average centroid i kept, both for the cavity and the myocardium egmentation. Of note, thi tep i performed taking into account the lice-by-lice rigid tranformation etimated uing Algorithm 2, which amount to performing the average centroid computation and connected component analyi on a motion-compenated tack. Finally, in order to exclude potential papillary mucle from the cavity intenity computation, a Gauian mixture model i fitted to the ditribution of intenity value inide the cavity egmentation. Since only ome lice how papillary mucle, both a twocomponent and a one-component model are ued, and only one i elected baed on the Akaike information criterion [27]. If the two-component model yield the bet fit, ince the cavity ditribution i alway higher than that of papillary mucle, the mean of the component with the highet mean i ued a average intenity value for the cavity. For the myocardium, the mean intenity of the pixel maked by the egmentation i computed. Cardiac image contrat i finally defined a the difference between thee two value. A double anity check i performed leveraging the probabilitic nature of PSM: if either the peak value of either the cavity or the myocardium PSM wa below a uer-defined threhold or the ize of either of the final hard egmentation for the cavity or the myocardium wa le than a defined number of pixel, the obtained contrat wa deemed unreliable. The tep for cardiac image contrat etimation are alo lited in Algorithm 3. Algorithm 3: Cardiac Image Contrat Etimation Input PSM: LV cavity: P SM SA Cav LV myocardium: P SM SA Myo with = (1,..., numslice) for = 1 to numslice do Threhold P SM SA Cav Output: SM SA Cav and P SM SA Myo and SM SA Myo : Etimate centroid P for SM SA Cav end Etimate mean centroid: P = mean(p ) for = 1 to numslice do Exclude all but one connected component per SM baed on ditance to P: end Output: SM SA Cav and SM SA Myo Fit Gauian Mixture Model to SM SA Cav exclude papillary mucle: Output: µ SA BP Compute contrat CT: (SA ( CT = µ SA BP - mean with SA the -lice of the SA tack to SM SA Myo )) E. Performance Evaluation Image Acquiition: To train and tet the propoed quality control pipeline, image from two different dataet were ued: the UKBB [3] and the UKDHP [8]. CMR imaging for the UKBB wa performed uing a 1.5T Siemen MAGNETOM Aera ytem equipped with a 18 channel anterior body urface coil (45 mt/m and 200 T/m/ gradient ytem). 2D cine balanced teady-tate free preceion (b-ssfp) SA image tack were acquired with in-plane patial reolution mm, lice thickne 8 mm, lice gap 2 mm, image ize and average number of lice 10. 2D cine b-ssfp LA image were acquired with in-plane patial reolution mm, lice thickne 8 mm and image ize Further acquiition detail can be found in [3]. CMR imaging for the UKDHP wa performed on healthy volunteer uing a 1.5T Philip Achieva ytem equipped with a 32 element cardiac phaed-array coil (33 mt/m and 160 T/m/ gradient ytem). 2D cine balanced teady-tate free preceion (b-ssfp) SA image tack were acquired with in-plane patial reolution mm, lice thickne 8 mm, lice gap 2 mm, image ize and average number of lice 12. 2D cine b-ssfp LA image were acquired with in-plane patial reolution mm, lice thickne 8 mm and image ize In both dataet, only end-diatolic frame were conidered. Experimental deign: A erie of experiment wa conducted to ae the accuracy of each portion of the pipeline. Firt of all, the five hybrid random foret model were trained uing a randomly-generated ubet of 500 cae from the UKBB. For each LA image-baed model, the 500 image were ued together with manually-annotated landmark and egmentation of the LV cavity. The egmentation were obtained with a CNN-baed automated tool proven to reach human-level performance [28], and then viually checked for accuracy. Each training et wa quadrupled in ize through data augmentation applying random recaling (following a normal ditribution with µ = 1, σ = 0.1) and random rotation (µ = 0, σ = 30 ). For each of the two SA tack-baed model, the lice extracted from the 500 tack were ued (for a total of 5165 image) together with egmentation of the LV cavity and of the LV myocardium, repectively (obtained uing the ame proce decribed for LA image). Detail regarding foret training include image patch ize px for LA model and px for SA one, egmentation label ize px, number of ample , number of tree T = 8. A firt erie of experiment wa performed by evaluating the trained pipeline on a eparate teting et coniting of 3000 cae randomly extracted from the UKBB. To evaluate the accuracy of the propoed heart coverage etimation technique, two experiment were conducted. Firt, for each of the three LA view, the poition of the landmark were manually annotated on 100 randomly elected cae. The automatically detected LM were compared to the manually identified one by meauring the Euclidean ditance between the two et of point. Then, the 3000 SA tack were viually inpected (ometime uing LA image a reference) to identify cae with inufficient coverage, defined a uch when at leat one full lice wa miing. Automated heart coverage etimation

8 8 wa then performed on the ame dataet. To intruct the previouly decribed anity check, the mean and tandard deviation of the relative ditance between manually annotated landmark were computed on the 100 image (l 2 l 1 = 89 ± 12 mm, l 3 l 2 = 32 ± 5 mm, l 3 l 1 = 87 ± 12 mm): then, for each LA view, when either of the relative ditance between the automatically detected LM wa over 2 tandard deviation greater or maller than the repective mean ditance value (thu covering roughly 95% of the meaured variability), the LM from that image were dicarded and the automated coverage etimation wa performed only on the remaining one (if available). Finally, the accuracy of the technique wa aeed againt the performed viual inpection performing a tandard binary claification tet uing a threhold for inufficient coverage optimized automatically with an ROC analyi. To evaluate the accuracy of the motion detection technique, two experiment were conducted. Firt, for each of the three LA view a well a for the SA tack, the automatically extracted PSM were compared to hard egmentation obtained uing the previouly-decribed CNN-baed automated tool [28] on 1000 randomly elected cae. While thi experiment wa aimed at aeing the accuracy of the PSM, it i worth noting that the PSM are never directly threholded for egmentation purpoe in the pipeline, which on the contrary exploit their probabilitic nature. For the ake of thi comparion, the PSM were turned into hard egmentation by applying a global threhold and compared to the reference one by computing the Dice coefficient (DSC). The global threhold wa optimized automatically uing an ROC analyi. Then, 1500 SA tack were viually inpected (ometime uing LA image a reference) to identify cae with noticeable motion corruption. Automated motion detection wa then performed on the ame dataet. To implement the previouly decribed anity check, PSM lice with peak probability value below 600 were conidered not reliable for motion detection, and thu their T (i.e. the etimated tranlation magnitude) dicarded; if le than 2 T value were left, the motion detection analyi wa not performed on the pecific tack. Accuracy of the automated technique wa aeed againt viual inpection with a tandard binary claification tet uing the following criterion: a tack wa deemed motion-corrupted if either the average T wa above a firt threhold T A or at leat two T were above a econd threhold T B. Thi double criterion aimed at the detection of both tack with a few, clearly mialigned lice a well a tack with poor general alignment. Both T A and T B were optimized automatically uing an ROC-like approach. To evaluate the accuracy of the cardiac image contrat etimation technique, 100 random lice from a many random SA tack were manually annotated electing region of interet (ROI) within the LV cavity and the LV myocardium. Cardiac image contrat wa etimated both from the original image and from the image after contrat normalization uing a randomly elected reference image tack. Automated contrat etimation wa then performed on the ame dataet, both before and after normalization, uing N cav = 450 px and N myo = 200 px. To implement the previouly decribed anity check, contrat extracted from lice with PSM (either for the cavity or for the myocardium) with peak value below 150 or with repective hard egmentation with a ize of le than 32 mm 2 (i.e. 10 pixel) wa deemed unreliable and excluded from the analyi. Automatically etimated and manually computed contrat value were compared uing Pearon correlation coefficient, linear regreion and Bland-Altman analye. A econd erie of experiment wa then performed by evaluating the pipeline trained on UKBB on a eparate teting et coniting of 100 cae randomly extracted from the UKDHP. Since the can in UKDHP were acquired with a different canner and with different parameter from thoe ued for UKBB, thee experiment were aimed at aeing the generalization propertie of the propoed pipeline. To harmonize the difference between training and teting dataet, the image in UKDHP were pre-proceed through intenity normalization [29], patial reampling and image reorientation. The 100 cae were then viually inpected and manually annotated following the ame criteria decribed for the previou experiment to provide the ground truth for etimation coverage, motion detection and contrat etimation. Since the viual aement for heart coverage etimation returned no ub-optimal cae, a procedure wa implemented to imulate coverage iue and allow a more meaningful evaluation of the pipeline. Stack were randomly picked following a uniform ditribution (10% chance of being picked), and a number of lice were deleted (either from the top or the bottom of the tack with equal probability), with thi number randomly elected from a normal ditribution (µ = 1, σ = 2). Coverage wa then viually re-aeed on the whole dataet. It i important to note that while thi corruption procedure altered the propertie of the dataet with repect to coverage, it did not affect the image on which the learning-baed portion of the pipeline i applied (i.e. the LA image) but only the SA tack, which influence the coverage etimation by mean of their ize and patial orientation. The pipeline wa applied with the ame etting ued for the previou dataet except for the threhold for the anity check for contrat etimation relative to the peak PSM value, which wa moved from 150 to 100 to account for the lightly lower overall repone in the PSM. The evaluation trategy for the three check wa the ame a for the previou et of experiment. For all the experiment, manual annotation and viual inpection ued a ground truth were performed internally by G. T. (medical imaging reearcher with 10 year of experience in cardiac imaging) and H. S. (experienced cardiologit), both blinded to the reult of the automated analye: more pecifically, H. S. viually inpected the 3000 SA tack from the UKBB dataet to identify cae with inufficient coverage, and G. T. performed all of the remaining aement. IV. RESULTS The experiment were initially run on a ingle core of an Intel Xeon CPU E GHz with 64 GB of memory to ae the peed of the current pipeline. Average time required to extract PSM and LM (when included in the model) wa 1.3 per SA tack (of roughly 10 lice) and 0.85 per LA image. Average time required to perform the quality control check were 0.26 per SA tack for coverage

9 9 Fig. 5: Reult for heart coverage etimation in two cae, one with ufficient (cae #1) and one with inufficient coverage (cae #2). In the firt three column, the reult for landmark detection in the three LA view. In the lat column, a mix view with the LA two-chamber view and the SA tack together with the median landmark for the apex and the mitral valve. Landmark Detection Localization Error LA 2CH LA 3CH LA 4CH Apex 4.2 ± ± ± 3.1 Mitral Valve (Side I) 3.6 ± ± ± 2.3 Mitral Valve (Side II) 3.9 ± 2.7 TABLE I: Landmark localization error in mm (mean ± td). etimation, 9 per SA tack for motion detection (in thi cae uing parallelization on 6 core to evaluate multiple lice from one tack at once) and 0.6 per lice for contrat etimation. The localization error for landmark detection on UKBB for the three LA view are reported in Table I and in Fig. 9 (Supplementary Material). Of note, the landmark extracted from one image per LA view were identified a outlier and thu excluded from the reported reult. Mean DSC value between threholded PSM (uing a threhold of 450) and reference egmentation were repectively 0.90 ± 0.07 for the SA tack, 0.94 ± 0.08 for LA 2CH, 0.94 ± 0.08 for LA 3CH, and 0.94 ± 0.07 for LA 4CH. Firt are reported the reult on UKBB. For accuracy aement of heart coverage etimation, 3 of the 3000 cae were excluded from the analyi: one due to the lack of LA image, and two for failing the anity check on all the LA image. The ROC analyi performed on the remaining 2997 image returned an optimal threhold of 90%. The reult of the binary claification tet are reported in Table II. For accuracy aement of motion detection, 3 of the 1500 cae were excluded from the analyi: one due to the lack of the SA tack and two for failing the anity check. An ROC-like analyi wa performed on the remaining 1497 image to elect the threhold T A and T B. The reult of the binary claification tet, obtained for T A = 3.4 mm and T B = 6 mm, are reported in Table III. For accuracy aement of contrat etimation, 3 of the 100 image were excluded from the analyi for failing the anity check. Reult for Pearon correlation coefficient, linear regreion and Bland-Altman analye between automatically and manually etimated contrat value are reported in Table IV and in Fig. 11 (Supplementary Material). Example of the reult obtained for the three check on UKBB are hown in Fig. 5, 6, 7 and 10 (Supplementary Material). Quality Control on UKBB Heart Coverage Etimation Senitivity Specificity Viual Aement 88% 99% Propoed 49 (TP) 15 (FP) Technique 7 (FN) 2926 (TN) TABLE II: Claification reult for heart coverage etimation uing a 90% coverage threhold. Poitive cae correpond to cae with inufficient coverage. Inter-Slice Motion Detection Senitivity Specificity Viual Aement 85% 95% Propoed 213 (TP) 58 (FP) Technique 39 (FN) 1187 (TN) TABLE III: Claification reult for motion detection uing T A = 3.4 mm and T B = 6 mm. Poitive cae correpond to motion-corrupted cae. Cardiac Image Contrat Etimation R Bia Std a b Mean Original Image Normalized Image TABLE IV: Correlation coefficient (R), bia and td for Bland-Altman analyi, linear regreion coefficient (a and b) and mean meaured value between automatically and manually etimated cardiac image contrat, both on original image and after hitogram normalization, in a.u.. Then are reported the reult on UKDHP. For accuracy aement of heart coverage etimation, all cae paed the

10 10 anity check. The ROC analyi returned an optimal threhold of 92% coverage, and the reult of the ubequent binary claification tet are reported in Table V. For accuracy aement of motion detection, 1 of the 100 cae wa excluded from the analyi for failing the anity check. The ROC-like analyi wa performed on the remaining 99 image to elect the threhold T A and T B. The reult of the binary claification tet, obtained for T A = 3 mm and T B = 6 mm, are reported in Table VI. For accuracy aement of contrat etimation, 9 of the 100 image were excluded from the analyi for failing the anity check. Reult for Pearon correlation coefficient, linear regreion and Bland-Altman analye between automatically and manually etimated contrat value are reported in Table VII and in Fig. 11 (Supplementary Material). Quality Control on UKDHP Heart Coverage Etimation Senitivity Specificity Viual Aement 100% 100% Propoed 5 (TP) 0 (FP) Technique 0 (FN) 95 (TN) TABLE V: Claification reult for heart coverage etimation uing a 92% coverage threhold. Poitive cae correpond to cae with inufficient coverage. Inter-Slice Motion Detection Senitivity Specificity Viual Aement 78% 90% Propoed 14 (TP) 8 (FP) Technique 4 (FN) 73 (TN) TABLE VI: Claification reult for motion detection uing T A = 3 mm and T B = 6 mm. Poitive cae correpond to motion-corrupted cae. Cardiac Image Contrat Etimation R Bia Std a b Mean Original Image Normalized Image TABLE VII: Correlation coefficient (R), bia and td for Bland-Altman analyi, linear regreion coefficient (a and b) and mean meaured value between automatically and manually etimated cardiac image contrat, both on original image and after hitogram normalization, in a.u.. V. DISCUSSION The reult obtained for the landmark localization experiment how that the average localization error i around 3.9 mm (roughly two pixel) and i thu mall compared to the recontructed lice thickne in both dataet (10 mm), uggeting the reliability of the landmark detection technique for the ake of heart coverage etimation. The propoed hybrid deciion foret method i baed upon a previou implementation for landmark detection [22] which conited of a multitage approach devied to increae the robutne to large variation in ditance and orientation of the landmark. It i worth mentioning that initial experiment performed uing thi approach howed no meaurable improvement with repect to the ingle-tage one (perhap due to the ize of the training et and to the conitency of the orientation of the image), which wa thu preferred (Fig. 12, Supplementary Material). The high DSC value obtained for the PSM ugget their reliability for both motion detection and contrat etimation. The fact that the PSM of the SA tack are lightly wore than thoe of the LA image (0.90 v 0.94) i mainly due to a lower repone of the model in the apical lice, where a different threholding value would have been beneficial. However, thi doe not caue a direct problem on the propoed pipeline, which never threhold PSM for egmentation purpoe and intead exploit their probabilitic nature. The firt et of experiment involving the whole pipeline wa aimed at aeing it accuracy on UKBB. The binary claification tet on coverage etimation performed on 2997 cae from UKBB indicate the high accuracy of the propoed technique, with enitivity = 88% and pecificity = 99%. The interpretation of thee reult i hindered by the trong cla imbalance between cae with ufficient and inufficient coverage, and thu a more detailed analyi of the reported confuion matrix i required. By applying the propoed automated technique, it i poible to correctly detect 88% of the cae with inufficient coverage, and thu to lower the percentage of undetected wrongly imaged cae from 1.9% to 0.2%. Thi come at the price of having to viually check an additional 0.5% of cae that actually featured a ufficient coverage. Notably, everal of the 15 FP cae actually had a ub-optimal coverage, but not of the amount required to be conidered a wrongly imaged following the criterion adopted during viual inpection. Compared to our previou work [12], the preent approach make ue of three LA image intead of jut one. The redundancy offered by exploiting all the available LA view allow a more robut and reliable etimation: thi i uggeted by the higher enitivity and pecificity achieved (88% v 73% and 99% v 98%, repectively, although a direct comparion i not completely fair ince the UKBB ubet ued in [12] wa different from the preent one) and by the lower number of cae excluded due to failing the anity check (down from 89 to 3). Of note, thi check i able to indirectly detect and exclude LA image with high noie level, wrong acquiition planning or wrong file naming that make the landmark localization unreliable, and in the preent implementation only cae in which all the three LA view yielded bad landmark detection had to be excluded from the coverage aement. Zhang et al. [10] addreed coverage etimation by performing fully-upervied CNNbaed lice claification to detect tack with miing baal (MBS) or apical lice (MAS). In their later work [11], they acknowledged the need for a large amount of labelled data during training to achieve good generalization: to mitigate thi iue, the author have increaed the ize of the training et uing generative network (reaching average accuracie of 93% for MAS and 89% for MBS on a dataet of 3400 cae from UKBB). The ue of different ubet of data from the UKBB and the different validation trategie (detection of miing lice eparately in the apical and in the baal

11 11 LA 2CH + SA Color Map Cae #2 Cae #1 Average T : 4.7 mm Average T : 1.6 mm Fig. 6: Reult for motion detection on UKBB in two cae, one with (cae #1) and one without motion corruption (cae #2). In the econd column, the color map of the tranlation magnitude for each lice are overlaid on top of the SA tack. region v detection of overall non-optimal cae) make the comparion between the two approache not traightforward. The main advantage of the approach of Zhang et al. i that it can detect problematic can uing only the SA tack, while our pipeline relie on the preence of at leat one of the LA view (which are, however, routinely acquired in mot CMR protocol). On the other hand, we believe there i a clinical and practical advantage in meauring the relative coverage intead of performing binary claification: cae with only lightly ub-optimal coverage could till be included in the following analye, epecially when the lack of coverage i in the apical area. Moreover, while their approach completely relie on feature extraction from ingle lice and thu mall image perturbation can potentially lead to miclaification, our approach i deigned to exploit the redundancy offered by the multiple LA view for greater robutne. The reported reult on UKBB for motion detection indicate that the propoed approach achieve enitivity = 85% and pecificity = 95% over 1497 cae. By applying the propoed automated technique, it i poible to lower the percentage of undetected motion-corrupted cae from 16.8% to 2.6%. Thi come at the price of having to viually check 3.9% cae that were viually deemed motion-free. It i worth to note that the binary claification of tack baed on the viual aement of motion i a difficult tak in itelf, limiting the meaurable accuracy of any technique. A more thorough examination would require a lice-by-lice viual claification, which i however impractical for dataet of thi ize. The accuracy of the contrat etimation technique on UKBB i indicated by very high correlation coefficient and regreion line near unity both for image before and after contrat normalization. Bland-Altman analye how negligible biae and narrow limit of agreement with repect to the mean meaured value, uggeting the high accuracy of the technique. The econd et of experiment wa aimed at aeing the accuracy of the pipeline (trained on UKBB) on the UKDHP dataet, thu teting it generalization propertie, and yielded encouraging reult. Regarding heart coverage etimation, our Fig. 7: Reult for contrat etimation on UKBB in two cae, one with high (cae #1) and one with low contrat (cae #2). The ROI from which the mean intenitie are etimated are hown in red and cyan. technique wa able to correctly identify all ub-optimal cae. Regarding motion detection, it returned lightly lower value for enitivity and pecificity than thoe obtained on UKBB: while thi might be due to a lower accuracy of the extracted PSM, we noted that motion in the UKDHP dataet i coniderably le pronounced than on UKBB, o it i eaier to miclaify borderline cae. Regarding contrat etimation, the technique howed again very high correlation coefficient and regreion line near unity. The increaed difficulty in dealing with a teting dataet different from the training one can be een in the lightly higher number of cae failing the anity check (up from 3 to 9) and in bigger biae, till however negligible when compared to the mean meaured value. In general, the mall ize of the UKDHP dataet hould be taken in conideration when evaluating thee reult, epecially for binary claification tet where the miclaification of a ingle cae can have a very large influence on the accuracy figure. However, we believe the reported reult how that the propoed approach generalize well to previouly uneen dataet, coping with difference in the acquiition protocol. Our approach to quality control doe not attempt to directly claify ub-optimal cae for two reaon. Firt, thi allow the complete circumvention of any cla-imbalance iue, ince the only learning-baed portion of our pipeline aim at the identification of tructure that are preent in every image. Second, the implemented pipeline doe not contitute a blackbox approach: each quality check produce quantitative metric with a clear meaning, which can be of great value in informing the MR operator on the type and the entity of the identified iue. Importantly, the propoed pipeline could be adopted alo uing different technique for landmark detection and probabilitic egmentation. One major requirement for thee alternative method would be the generation of fuzzy egmentation map providing a probabilitic repreentation of the target tructure: thi allow the aement of their reliability for both motion detection and contrat etimation, otherwie unfeaible with tandard, hard egmentation. The main limitation affecting our approach i that no quality check i performed on the manual election of the imaging plane for LA and SA image, which can be ubject to error.