Carotid Lumen Segmentation and Stenosis Grading Challenge Reinhard Hameeteman Maria Zuluaga Leo Joskowicz Moti Freiman Theo van Walsum version 1.00 may 07, 2009 This document contains a description of the data, data processing and challenges for the Carotid Lumen Segmentation and Stenosis Grading Challenge, that was organized as part of the 3 rd 3D Segmentation in the Clinic: a Grand Challenge workshop at the MICCAI 2009.
Contents 1 Introduction 3 2 Challenges 3 2.1 Lumen Segmentation.......................... 3 2.2 Stenosis Grading............................ 4 3 Data acquisition 5 3.1 Erasmus MC protocol.......................... 5 3.2 Hadassah protocol............................ 6 3.3 Louis Pradel protocol.......................... 6 3.4 Datasets for training, testing and on-site challenge........... 7 4 Manual annotations 7 5 Reference standard 8 5.1 Preprocessing.............................. 8 5.2 Observer contour processing...................... 8 5.3 Reference standard lumen segmentation................ 10 5.4 Stenosis values............................. 10 6 Evaluation measures and ranking 12 6.1 Lumen segmentation.......................... 12 6.2 Stenosis grading............................. 13 7 Datasets, fileformats and software 13 7.1 Testing and on-site challenge data................... 13 7.2 Training data.............................. 13 7.3 Participant partial volume segmentations................ 14 7.4 Stenosis grades............................. 15 7.5 Software................................. 15 2
1 Introduction This document describes the Carotid Lumen Segmentation and Stenosis Grading Challenge (CLS). The CLS is organized as one of the challenges of the 3rd MICCAI Workshop in the series 3D Segmentation in the Clinic: a Grand Challenge. The CLS has two seperate challenges: 1. Carotid bifurcation lumen segmentation 2. Internal carotid artery stenosis grading Each team can participate in either one of the challenges, or in both. This document describes respectively the challenge (i.e. the tasks to be performed), the data used, the manual annotation, the reference standard and the evaluation criteria. 2 Challenges 2.1 Lumen Segmentation The Common Carotid Artery (CCA) and Internal Carotid Artery (ICA), see Fig. 1, are clinically the most relevant arteries of the Carotid Bifurcation. Therefore, the segmentation challenge focuses on these two arteries. A small part of the External Carotid Artery (ECA) is also included, to prevent evaluation issues at the location where the ECA bifurcates from the ICA. Additionally, it allows us to include a complete bifurcation in the challenge. The goal of this challenge is to accurately segment the lumen of the Carotid Bifurcation in a Computed Tomography Angiography (CTA) dataset. There are two versions of this challenge: a fully automated version, and a semi-automated version where three initial points are provided. The region to be segmented is defined around the bifurcation slice, which we define as the first (caudal to cranial) slice where the lumen of the CCA appears as two separate lumens: the lumen of the ICA and the lumen of the ECA. The segmentation must contain the CCA, starting at least 20 mm caudal of bifurcation slice, the ICA, up to at least 40 mm cranial of bifurcation slice, and the ECA, up to between 10 and 20 mm cranial of the bifurcation slice, see also Fig. 1. The performance measures are only determined over the region of interest as specified above. However, the bifurcation slice is not communicated to the participants of the challenge. Therefore, the participants should make sure that there segmentation at least includes this region. Our definition of the bifurcation slice, and the specified regions, should be sufficient to determine a suitable region of interest for the segmentations. For the External Carotid Artery, the segmented lumen should be cut between 10 and 20 mm cranial of the bifurcation slice. To allow for some flexibility in cutting of the ECA, the region around the ECA between 10 and 20 mm cranial of the bifurcation slice is a masked region, where the evaluation measures will not be evaluated, see also Fig. 1. The input data for participants of the challenges is (see Section 7 for more detailed information): 3
Figure 1 Schematic depiction of the region of interest that is relevant for the challenge, and a rendering of this region for one of the datasets. the CTA dataset (including header information such as voxel sizes and world coordinate system), and three points if you join the semi-automatic method challenge: 1. a point in the Common Carotid Artery, at the level of the cranial side of the thyroid gland 2. a point in the Internal Carotid Artery, just before the artery enters the skull base 3. a point in the External Carotid Artery, where the artery is close to the mandible The participant will be asked to return the segmented lumen. This segmentation must be respresented as a (partial volume) segmentation, i.e. an image with floating point numbers, where each voxel value contains the occupancy of the voxel by the vessel lumen, where a value of 0 means no lumen present, and a value of 1 means fully occupied with lumen. The voxel value must thus be in the range [0,1]. 2.2 Stenosis Grading Two different stenosis grades have to be determined for each ICA that needs to be segmented in the segmentation challenge. We use the following NASCET-like definitions for stenosis grading: ( S a = 100% 1 a ) m (1) ( S d = 100% a r ) 1 d m d r where S a is an area-based stenosis grade, and S d is a diameter-based stenosis grade. The stenosis grade is a value in the range [0... 100], where 0 implies no stenosis, and 100 implies a fully occluded vessel. In the above formulations, a m is the minimal cross-sectional area along the CCA and ICA, and a r is the average cross-sectional area over a distal reference part of the (2) 4
Figure 2 Minimal diameter lines for various cross-sectional contours. Internal Carotid Artery. The default reference part has a length of 10 mm, and is 20 mm distal of the location of minimal area measured along the vessel centerline. However, observers are free to change the location and length of the reference area, with the restriction that it must be distal to the minimal area location, and not extend outside the segmented region, i.e. beyond 40 mm cranial of the bifurcation plane. The second stenosis grade is determined using minimal diameters. The minimal diameter of a cross-section is defined as the shortest straight line that divides the contour in two equal-sized areas, see Fig. 2 for examples of minimal diameters for various contour shapes. Similar to the lumen segmentation challenge, there are two versions of the stenosis grading challenge: a fully automated version, where the stenosis grading uses only the CTA dataset and the specification whether the left or right side needs to be graded, and a semi-automated version, where the algorithm also may use the three points in each of the arteries of the bifurcation (as supplied with the data). The input data available for this challenge is identical to the data for the lumen segmentation challenge. 3 Data acquisition The CTA datasets used in this challenge have been acquired at the Erasmus MC, University Medical Center Rotterdam, The Netherlands (36 datasets), Hôpital Louis Pradel, Bron, France (10 datasets) and the Hadassah Hebrew University Medical Centre, Jerusalem, Israel (10 datasets). The datasets have been selected such that the contain a large range of stenoses: from fully open to severe stenotic. Below details of the scanning protocols are provided for each of the medical centers that provided data for this challenge. The CTA scanning parameters are summarized in Table 1. 3.1 Erasmus MC protocol The CTA data were acquired on a 16-row CT scanner (Sensation 16 - Siemens Medical Solutions, Forchheim, Germany) with a standard scan protocol using the following parameters: 120 kv, 180 mas, collimation 16 0.75 mm, table feed per rotation 12 mm, pitch 1.0, rotation time 0.5 seconds and scan time 10 14 seconds. The CTA scan range 5
Table 1 Overview of scanning parameters of CTA datasets, EMC = Erasmus MC, LP = Louis Pradel, Hd = Hadassah CT Z- Slice Scanner Dims Pixel size spacing thick. Recon. (type) (voxels) (mm) (mm) (mm) filter EMC Sensation 16 512 512 395 579 0.23 0.26 0.6 1 B30f Hd Brilliance 64 512 512 750 0.55 0.5 1 B LP Brilliance 64 512 512 636 827 0.414 0.547 0.45 0.9 B is from the ascending aorta to the intracranial circulation (2 cm above the sella turcica). All patients received 80 ml contrast material (Iodixanol 320 mg/ml, Visipaque Amersham Health, Little Chalfont, UK), followed by 40 ml saline bolus chaser, both with an injection rate of 4 ml/sec. Synchronization between the passage of contrast material and data acquisition was achieved by real time bolus tracking at the level of the ascending aorta. The trigger threshold was set at an increase in attenuation of 75 Hounsfield Units (HU) above baseline attenuation (approx. 150 HU in absolute HU value). Image reconstructions were made with in-plane pixel sizes of 0.23 0.26 0.23 0.26 mm 2, matrix size 512 512 (real in-plane resolution 0.6 0.6 mm), slice thickness 1.0 mm, increment 0.6 mm and with an intermediate reconstruction kernel (B30f). 3.2 Hadassah protocol The CTA data were acquired on a 64-row CT scanner (Brilliance 64 Philips Healthcare, Cleveland OH) with a standard scan protocol using the following parameters: 120 kv, 251 mas, collimation 64 0.625 mm, pitch 1.20, rotation time 0.75 seconds and scan time 7.30 seconds. The CTA scan range is from the ascending aorta to the intracranial circulation (2 cm above the sella turcica). All patients received 75 ml contrast material (Iopamiro, Bracco Diagnostics, Milano Italy), with an injection rate of 3.5 ml/sec. Image reconstructions were made with in-plane pixel sizes of 0.55 0.55 mm 2, matrix size 512 512, slice thickness 1.0 mm, increment 0.5 mm and with an intermediate reconstruction kernel (B). 3.3 Louis Pradel protocol The CTA data were acquired on a 64-row CT scanner (Brilliance 64 Philips Healthcare, Cleveland OH) with a standard scan protocol using the following parameters: 120 kv, 300 mas, collimation 52 1.5 mm,rotation time 0.35 seconds and scan time 10 14 seconds. The CTA scan range is from ascending aorta to the intracranial circulation (2 cm above the sella turcica). All patients received 80 ml contrast material (Iomeron 4000 mg/ml, BRACCO, Milano, Italy) followed by 40 ml saline bolus chaser, both with an injection rate of 4 ml/sec. Synchronization between the passage of contrast material and data acquisition was achieved by real time bolus tracking at the level of the ascending aorta. The trigger threshold was set at an increase in attenuation of 75 HU above baseline attenuation. Image reconstructions were made with in-plane pixel sizes of 0.414 0.547 0.450.414 0.547 mm 2, matrix size 512 512 (real inplane resolution 0.6 0.6 mm), slice thickness 0.9 mm, increment 0.45 mm with an intermediate reconstruction kernel (B). 6
Table 2 Datasets per center with their three-digit ids, and selection for training and testing Training Testing On-site Total (ids) (ids) (ids) (#) Erasmus MC 000 008 009 029 030 035 36 Hadassah 100 102 103 107 108 109 10 Louis Pradel 200 202 203 207 208 209 10 Total (#) 15 31 10 56 3.4 Datasets for training, testing and on-site challenge In total 56 datasets have been acquired for this challenge. Each dataset has a unique three-digit id. The first digit shows in which center the data were acquired (0 = Erasmus MC, 1 = Hadassah, 2 = Louis Pradel), and the other two digits are a sequence number, starting at 00. The distribution of the datasets over the training, testing and on-site sets, with their numbering, is shown in Table 2 4 Manual annotations Three different observers annotated the carotid lumen boundary and graded the stenosis in the ICA. Each contributing center performed the annotations on the data it provided, thus the observers for each of the three centers were different. The manual annotations for the lumen segmentation and stenosis grading are performed with an in-house build tool, based on MeVisLab. The annotation procedure is performed as follows: 1. After clicking a point for the bifurcation slice, positions along the centerlines for both the ICA and ECA are clicked, starting in the CCA, 20 mm caudal of the bifurcation slice, and extending to 40 mm cranial of the bifurcation slice (ICA) or ending 20 mm cranial of the bifurcation slice (ECA), see Fig. 3. 2. The resampled centerlines are used to generated Curved Multi Planar Reformatted images (CMPRs), in which longitudinal contours are drawn for three different orientations (each 60 o apart) of the CMPRs. Cross-sectional contours orthogonal to the centerline are created at one mm intervals along the centerline. These spline-interpolated contours are initialized from the (six) positions where the cross-sectional plane intersects the longitudinal contours. The contours can be edited and updated if they do not match the luminal area, see also Fig. 4. 3. Based on the corrected cross-sectional contours, a graph for the contour area (or diameter) along the centerline is created. In the graph, the position of the minimal area (or diameter) can be selected, after which a default reference area is shown (20 mm distal, 10 mm length). This reference area can also be manually edited. The stenosis grade is determined using the values from these graphs, see also Fig. 5. 7
The first two steps are performed both for the ICA and the ECA. The ICA (and CCA contours) are used in the stenosis grading step, the ECA contours are not used for the stenosis grading. The contours are drawn with using standard window level settings (center = 176 HU, width = 800 HU). Figure 3 Screenshot of tool for center line drawing. 5 Reference standard The reference standard is created from the cross-sectional contours that result from the manual annotations. Below we describe how the contours of each of the three observers are turned into a signed distance map, and how the observer results are combined into the reference standard. 5.1 Preprocessing The bifurcation slices are averaged, giving a reference bifurcation slice. This slice, and the bounding boxes of the contours, are used to determine the region of interest for the evaluation. The region of interest is the bounding box of the contours, extended with 15 mm both in x- and in y-direction. The z-range is determined from the reference bifurcation slice, and ranges from 20 mm caudal of the bifurcation slice to 40 mm cranial of the bifurcation slice. 5.2 Observer contour processing The observer contours are processed as follows, see also Fig. 6: 8
Figure 4 Screenshot of tool for lumen contouring. Figure 5 Screenshot of tool for lumen stenosis grading. 9
The contours (both for the ICA and the ECA) are turned into partial volume segmentations, using a Thin Plate Spline interpolation between the contour points [1], pv-i and pv-e. Signed distance maps sdm-i and sdm-e are generated from the partial volume segmentations pv-i and pv-e. The ICA and ECA signed distance maps are combined, giving a signed distance map for the complete bifurcation, sdm-b. The partial volume segmentations pv-i and pv-e are also combined (voxel wise maximum) to obtain the partial volume segmentation of the bifurcation, pv-b. This partial volume segmentation is used to rate the observer in the same way as the contestants segmentations are rated. All signed distance maps use world distances in mm. 5.3 Reference standard lumen segmentation The reference standard consists of a partial volume segmentation, signed distance map and an isosurface, and is constructed in the following way: sdm The observers signed distance maps of the bifurcation are averaged, to obtain an average signed distance map. iso The zero-crossing of this reference signed distance map gives the reference lumen surface. pv From the reference standard signed distance map, a partial volume representation is generated, by interpolating the distance map on super-resolution, and averaging the voxels that have a negative distance (i.e. that are inside) ext From the observer signed distance maps of the ICA and ECA, average distance maps of the ICA and ECA are created. The mask of the distal part of the external contains all voxels that satisfy all of the following three criteria: 1. the voxel is in the 10 20 mm range cranial of the bifurcation 2. the ECA signed distance map value of the voxel is less than 2 mm, i.e. the voxel is inside or close to the ECA 3. the ECA signed distance map value is less than the ICA signed distance map value, i.e. the voxel is closer to the ECA than to the ICA. 5.4 Stenosis values The observer values for the stenosis are averaged to obtain the reference standard stenosis values. 10
Figure 6 Processing of observer annotations, left 3D visualization and right 2D visualization. From top to bottom: initial contours, partial volume from contours (left: isosurface at 0.5) and signed distance map from partial volume (left: isosurface at 0.0). 11
6 Evaluation measures and ranking 6.1 Lumen segmentation The partial volume lumen segmentations will be evaluated using the following four performance measures: 1. the Dice similarity index D si : D si = 2 pv r pv p pv r + pv p, (3) where pv r and pv p are the reference and a participants partial volumes, the intersection operation is the voxelwise minimum operation, and. is the volume, i.e. the integration of the voxel values over the complete image. 2. the mean surface distance D msd : D msd = 1 ( 2 S r sdm p ds + S r S p sdm r ds S p ), (4) where sdm r and sdm p are the signed distance maps of the reference and a participants segmentation, and S r and S p are the lumen boundary surfaces (isosurfaces of the signed distance map at the value 0), and S i is the surface area of surface S i, i.e. S i = S i ds. 3. the root mean squared surface distance: D rmssd : D rmssd = 1 2 S r sdm 2 p ds S + p sdm 2 r ds, (5) S r S p 4. maximum surface distance D max : D max = 1 ) (max 2 sdm p (x) + max sdm r (x) x S r x S p, (6) All distance measures are symmetric, and all these measures are only evaluated in a the region of interest that is specified in 2.1. Furthermore, the mask for the distal part of the ECA is also used in all the above measures. The measures above will lead to one performance value of a participants for each dataset and for each performance measure. Per dataset and per performnce measure a ranking of the participants will be made, i.e. with N datasets, and the 4 measures, N 4 rankings will be obtained. The final ranking for a participant is obtained by averaging the ranks of all these N 4 rankings. 12
6.2 Stenosis grading The evaluation of the stenosis grade is straightforward: the absolute difference between the reference standard value and the value determined by a participant is the error in stenosis grade. As revealing the (exact) error per dataset also more or less reveals the reference stenosis grades, the stenosis errors are not communicated per dataset, but only per ensemble (testing or on-site). The same holds for the ranking. The final ranking, however, is determined by averaging the (hidden) errors per dataset and stenosis grade (diameter and area). 7 Datasets, fileformats and software This section describes the datasets that are available via the website, the format of the data, etc. 7.1 Testing and on-site challenge data The testing data contains all data that might be needed to compute the lumen segmentation and/or stenosis grading. It is distributed in several archives that need to be extracted in the same directory. Extraction of the archives will give one subdirectory for each challenge dataset, named challenge<cid>, where <cid> stands for the challenge id, being a three-digit id for the challenge (e.g. 003, 100,... ). Per directory, the following files will be present: cta<cid>[l r].mhd and cta<cid>[l r].raw: the header information, such as the world coordinate system and the voxel sizes and the pixel data, in 16-bits unsigned integers (Hounsfield units plus 1024). If the last character of the filename without extension is an l the left side (left carotid artery bifurcation) needs to be processed, if it is an r the right side needs to be processed. points<cid>.txt: a text file with three lines, containing the three coordinate positions that can be used for the initialization if you join one of the semi-automatic challenges. The first line contains the spatial coordinates of the point in the CCA, the second line contains the spatial coordinates of the point in the ICA, and the third line contains the spatial coordinates of the point in the ECA. All coordinates are in the world coordinate system of the CTA image. side<cid>.txt: a text file that contains left when the left carotid bifurcation needs to be processed, and right when the right carotid bifurcation needs to be processed. 7.2 Training data The training data is distributed in the same structure, and contains per challenge directory the following files: roi<cid>.txt: the region of interest (in voxels) The first line contains the minimum voxel index, and the second line contains the maximum (inclusive!) voxel index. 13
ext range<cid>.txt: the region of interest (in voxels) The first line contains the minimum voxel index, and the second line contains the maximum (inclusive!) voxel index. pv<cid>.mhd and pv<cid>.raw: a floating point volume (in the size of the region of interest) containing the reference partial volume segmentation. sdm<cid>.mhd and sdm<cid>.raw: a floating point volume (in the size of the region of interest) containing the signed distance map of the segmentation (negative values are inside the lumen, positive values are outside the lumen). ext<cid>.mhd and ext<cid>.raw: a byte volume (in the size of the bounding box) containing the mask around the distal part of the external, where the performance measures will not be evaluated. A voxel with value 0 is a background voxel, a voxel with value 1 belongs to the mask. iso<cid>.vtp: a vtkpolydata file containing the isosurface of the signed distance map at the value 0.0. The surface is in world coordinates of the original dataset. The world coordinate system of the above images that only contain a region of interest corresponds to the world coordinate system of the original CTA dataset. 7.3 Participant partial volume segmentations The participants are required to upload their segmentation results as a floating point partial volume segmentation, using the same directory structure (challenge000... challenge209) and file naming conventions: roi<cid>.txt: the region of interest (in voxels). The first line contains the minimum voxel index, and the second line contains the maximum (inclusive!) voxel index. pv<cid>.mhd and pv<cid>.raw: the floating point volume (in the size of the bounding box) containing the reference partial volume segmentation. The participants are encouraged to only upload the relevant portion of the data, to reduce the data bandwidth of our website. In case the participants region of interest does not overlap completely with the reference region of interest, the missing voxels are assumed to have a value of 0 (not containing any lumen). The world coordinate system in the participants image data will be ignored, the roi text file is the definitive source for the spatial relationship between the submitted data and the original CTA image. Upon evaluation, a signed distance map and an isosurface representation of this partial volume segmentation will be generated for determining the performance measures. 14
7.4 Stenosis grades The stenosis grades as provided in the training data are in two text files: s area<cid>.txt: a text file with the area stenosis measure s diam<cid>.txt: a text file with the diameter stenosis measure We require the same structure for submission of stenosis grades by participants of the stenosis grading challenge: provide these two files, with the correct names, in the right challenge directory. 7.5 Software The following software tools will be available for evaluating training segmentations with the reference standard. These are the same tools as will be used for the evaluation of the testing and on-site challenge datasets: clip-pv: tool to clip the participants dataset to the region of interest of the reference standard normals: determines gradient in partial volume image, required for signed distance map pv2sdm: converts the partial volume segmentation to a signed distance map sdm2iso: converts the signed distance map to an isosurface dice: evaluates the dice measure surfdist: determines the mean, root mean squared and maximum surface distance Additionally, some scripts will be supplied to perform the actual performance measurements. References [1] Greg Turk and James F. O Brien. Shape transformation using variational implicit functions. In Proceedings of ACM SIGGRAPH 1999, pages 335 342, Aug 1999. 15