CONTOURING ACCURACY. What Have We Learned? And Where Do We Go From Here? BEN NELMS, PH.D. AUGUST 15, 2016

Transcription

1 CONTOURING ACCURACY What Have We Learned? And Where Do We Go From Here? BEN NELMS, PH.D. AUGUST 15, 2016

2 FIRST THINGS FIRST Happy Medical Dosimetrist s Week!

3 OUTLINE 1. Objectives 2. The importance of accurate anatomy contouring 3. Is there variation in contouring? If so, which structures vary the most? 4. What is the impact of contouring variation? 5. Can we increase accuracy and reduce variation? 6. Introduction to the Contouring Accuracy Program 7. Q & A Session

4 OBJECTIVES We will study the variability in contours created by multiple observers for the same image sets. We will examine the variability of structures relative to others. Which structures vary the most? Which are most consistent? We will investigate the potential impact of contouring inaccuracy in terms of dose optimization, plan review, and ultimately quantifying clinical outcomes. We will learn how to use an online system to: 1. Interactively contour critical structures; 2. Measure performance against gold standards; 3. Access training material; 4. Practice and re-contour in order to improve accuracy and consistency, and 5. Earn CE credits as well as performance-based badges for your professional résumé.

5 THE IMPORTANCE OF CONTOURING ACCURACY PER PLAN PER FRACTION Imaging (CT, MR, etc.) ANATOMY CONTOURING Dose Optimization & Calculation Plan Review & DVH Analysis Pretreatment Dose QA Patient Setup IGRT The 3D patient anatomy model is THE CRITICAL INPUT to the radiation therapy process. Adaptive Planning Dose Delivery Volumetric imaging and anatomy contouring are two critical inputs that have an impact on all subsequent steps in the radiation therapy process.

6 THE IMPORTANCE OF CONTOURING ACCURACY What will be wrong if there are contouring errors? - Plan Optimization will be compromised. The dose will be shaped to the wrong volumes. - Plan Acceptance criteria will not be valid. The plan acceptance criteria (DVH points, max ROI dose, mean ROI dose, etc.) will not be trustworthy. - IGRT and Adaptive RT will also suffer. IGRT setup based on ROI shape/location will have errant volumes as the reference. Adaptive based on morphing target and OAR volumes will be wrong. - The patient may be mistreated. Dose will be delivered to the wrong place or not delivered to the right place. No TPS and no modality will correct for contouring errors.

7 IF IT S SO IMPORTANT, WHY NO ATTENTION? If contouring accuracy is so important, why does the industry overall give it much less attention than new technology or algorithms? Possible explanations: 1. Big money and curb appeal has been in new technology, not in continual improvement. 2. People agree it is important but there is no quality standard or proof-of-performance required for accreditation, reimbursement, etc. 3. Some people do not know (or acknowledge) that there is a need for improvement.

8 IS THERE VARIATION IN CONTOURING? We all agree that contouring is important. - But is there significant variation in contouring? - Or is this much ado about nothing? Those are great questions, and we have lots of data. - Contouring Challenge 1.0 (2010) - Contouring Challenge 2.0 (2011) - Red Journal Paper (2012) - AAMD Contouring Workshops 2014, 2015, & 2016

9 METHOD TO STUDY VARIATION PARTICIPANTS CT Dataset Participant downloads and/or views the CT images Expert (or panel of experts) creates reference structures Participant defines the requested structures Participant submits data Generate Population Statistics Measure Accuracy Gold ROIs

10 METHOD TO MEASURE ACCURACY Compare vs. Gold StructSure patented method Missing Voxel Extra Voxel 1 mm Error No Penalty (Forgiven) Comparison Each Missing or Extra Voxel is penalized as a function of Distance Away from Primary Volume Penalty per Error Voxel = 0.5 x (Distance Error mm 1 mm Forgiveness) 2 mm Error Penalty = mm Error Penalty = 1.0 Accuracy Score = 100 x [# Gold Voxels S Voxel Penalties] / # Gold Voxels

11 METHOD TO MEASURE ACCURACY Distance-Error Volume Histogram This example shows a contour that was drawn too small (many missing voxels) This example shows a contour that was drawn too large (many extra voxels)

12 # Participants # Participants HOW TO LEARN FROM VARIATION Histograms of Accuracy Scores (Max Score = 100) This distribution shows: Many high scores No low scores Some variation This distribution shows: Many low scores No high scores High variation Accuracy Score Accuracy Score

13 WHAT STRUCTURES EXHIBIT THE MOST VARIATION? Low Variability Medium Variability High Variability Brain Spinal Cord Brainstem Brachial Plexus Mandible Parotid Larynx Glottis Lung Bladder Seminal Ves Submandibular Cochlea Hippocampus Chiasm Prostate Rectum Skin Small Bowel Sigmoid Large Bowel Penile Bulb Nelms BE, Tomé WA, Robinson G, Wheeler J. Variations in the contouring of organs at risk: test case from a patient with oropharyngeal cancer, Int J Radiat Oncol Biol Phys Jan; 82(1). Kusano A., Robinson G., and Nelms BE. Contouring Workshops, AAMD Annual Meetings Contouring Challenge 1.0 and 2.0 (2010 and 2011).

14 AAMD CONTOURING WORKSHOP (2014) 40 workshop attendees contoured: - Left and Right Parotid - Left and Right Submandibular Glands - Glottic Larynx Processed all contours and created 3D frequency grids - Each voxel s value represents the % of people who contoured that voxel as part of the structure - For example, a voxel with value of 100 (%) means everybody thought that voxel was in the structure - For example, a voxel with value of 50 (%) means half of the people thought it was in the structure - etc. Go to Interactive 3D Display of Variation

15 CONTOURING CHALLENGE 1.0* (2010) Prostate study - 72 CT slices - 3 mm slice spacing Regions of Interest - Prostate - Seminal Vesicles - Bladder - Rectum - Penile Bulb Time Limit - 60 min stated (allowed 90) Clinicians/Contestants - 59 registered contestants - 27 successful uploaded datasets Auto-segmentation * Organized by: ROR, Standard Imaging, and Canis Lupus

16 CONTOURING CHALLENGE 1.0 (2010)

17 CONTOURING CHALLENGE 1.0 (2010) Metric Scores (N=27) Structure Average Low High Bladder Seminal Vesicles Prostate Rectum Penile Bulb

18 CONTOURING CHALLENGE 2.0* (2011) Head/Neck Study CT slices mm slice spacing Regions of Interest - Spinal Cord Clinicians/Contestants - 42 registered contestants - 21 successful uploaded datasets Auto-segmentation - Brainstem - Larynx (Whole) - No entries - Parotid (Left and Right, scored as Total ) - Submandibular (Left and Right, scored as Total ) - Brachial Plexus (Left and Right, scored as Total ) * Organized by: ROR, Standard Imaging, and Canis Lupus

19 CONTOURING CHALLENGE 2.0 (2011)

20 CONTOURING CHALLENGE 2.0 (2011) Metric Scores (N = 21) Structure Mean Low High Spinal Cord Brainstem Parotid (Total) Submandibular (Total) Larynx Brachial Plexus (Total)

21 OUR RED JOURNAL PAPER (2012) BRAIN MANDIBLE SPINAL CORD IPSILATERAL PAROTID CONTRALATERAL PAROTID BRAINSTEM

22 OUR RED JOURNAL PAPER (2012)

23 AAMD CONTOURING WORKSHOP (2016) Rectum Sigmoid Small Bowel Large Bowel

24 WHAT STRUCTURES EXHIBIT THE MOST VARIATION? Low Variability Medium Variability High Variability Brain Spinal Cord Brainstem Brachial Plexus Mandible Parotid Larynx Glottis Lung Bladder Seminal Ves Submandibular Cochlea Hippocampus Chiasm Prostate Rectum Skin Small Bowel Sigmoid Large Bowel Penile Bulb Nelms BE, Tomé WA, Robinson G, Wheeler J. Variations in the contouring of organs at risk: test case from a patient with oropharyngeal cancer, Int J Radiat Oncol Biol Phys Jan; 82(1). Kusano A., Robinson G., and Nelms BE. Contouring Workshops, AAMD Annual Meetings Contouring Challenge 1.0 and 2.0 (2010 and 2011).

25 WHAT IS THE IMPACT OF CONTOURING ACCURACY? You draw these. You plan this. But these were correct.

26 WHAT IS THE IMPACT OF CONTOURING ACCURACY? Axial plane, VMAT dose gradient: 38 Gy to 13 Gy in 1 cm 25 Gy per cm (2.5 Gy per mm) 62.5% of Rx per cm (6.25% of Rx per mm)

27 WHAT IS THE IMPACT OF CONTOURING ACCURACY? Sagittal plane, VMAT dose gradient: 39 Gy to 4 Gy in 1 cm 35 Gy per cm (3.5 Gy per mm) 87.5% of Rx per cm (8.75% of Rx per mm)

28 WHAT IS THE IMPACT OF CONTOURING ACCURACY? We have seen from the studies on contouring variation that differences (i.e. errors) can be very large. But even errors on the order of ~mm in the critical (high gradient) locations can cause drastic dosimetric changes to sub-volumes. Quickie Quiz. Could one be conservative and purposely contour all organs-at-risk (OAR) a little too large?

29 WHAT IS THE IMPACT OF CONTOURING ERRORS? Int. J. Radiation Oncology Biol. Phys., Vol. 82, No. 1, pp , 2012 There was significant inter-clinician variation in OAR contouring. The degree of variation is organ-dependent. We found substantial dose differences resulting strictly from contouring variation (differences ranging from 289% to 56% for mean OAR dose; 22% to 35% for maximum dose).

30 METHODS: DATA COLLECTION DICOM RT Plan DICOM RT Structure Set DICOM RT Dose DICOM RT Plan DICOM RT Structure Set DICOM RT Dose Controlled Inputs CT Images & Target Volumes DICOM RT Plan DICOM RT Structure Set DICOM RT Dose 32 Treatment Planners: Contour OARs and generate IMRT plan & dose Expert, M.D.: Create gold OARs DICOM RT Structure Set

31 METHODS: CONTOURING DIFFERENCES StructSure Metric Scores (Similarity to Gold Data) Gold vs. 1 Gold vs i Gold vs. i Contouring Variability (Over All Datasets) Other Stats (Volume, etc.)

32 METHODS: DOSIMETRIC EFFECTS Plan & Dose 1 Plan & Dose 2 Plan & Dose i Gold vs. 1 Gold vs. 2 Gold vs. i

33 LET S REVISIT THE QUICKIE QUIZ Quickie Quiz. Could one be conservative and purposely contour all organs-at-risk (OAR) a little too large? Yes. Adding a margin to OARs is an effective conservative approach. No. Adding a margin to OARs is not conservative. It Depends. What s the dosimetric/dvh endpoint? Answer. It Depends! A larger OAR is conservative with respect to max dose, but can make mean dose or other DVH points lower than actual.

34 WHAT IS THE IMPACT OF CONTOURING ACCURACY? Right Parotid contours just a little different Right Parotid contours a lot different

35 WHAT IS THE IMPACT OF CONTOURING ACCURACY? Mean OAR Dose: Differences ranging from 289% to 56% Max OAR Dose: Differences ranging from 22% to 35% Contouring Score > 75: Differences all within 10%; most within 5%

36 ANOTHER RELEVANT PAPER ON THE TOPIC Int. J. Radiation Oncology Biol. Phys., Vol. 73, No. 3, pp , 2009 Differences in target and OAR delineation for breast irradiation between institutions/observers appear to be clinically and dosimetrically significant. A systematic consensus is highly desirable, particularly in the era of intensitymodulated and image-guided RT.

37 CAN WE INCREASE ACCURACY & DECREASE VARIATION? Int. J. Radiation Oncology Biol. Phys., Vol. 83, No. 5, pp. e , 2012 The results of this study indicate the feasibility of applying the PDCA cycle to assess competence in the delineation of individual organs With testing, guidance, and reevaluation, contouring consistency can be obtained across multiple dosimetrists Continual quality improvement using the PDCA approach will ensure more accurate treatments and dose assessment in treatment planning and delivery.

38 CAN WE INCREASE ACCURACY & DECREASE VARIATION? StructSure OAR Accuracy Scores Initial vs. Re-Testing

39 THE CONTOURING ACCURACY PROGRAM (CAP)

40 THE CONTOURING ACCURACY PROGRAM (CAP) The ProKnow CAP: - Is online (cloud-based) & fully interactive - Is a partnership with the AAMD - Earns 1.0 CE credit per structure (up to 12 credits/year) - Includes built-in contouring tools - Includes StructSure scoring method for both the initial measurements and all re-do and practice measurements - Provides PDF refresher course education materials along with video walkthroughs with the expert

41 THE CONTOURING ACCURACY PROGRAM (CAP) Go to ProKnow Sign In Select Contouring Accuracy Select Structure

42 THE CONTOURING ACCURACY PROGRAM (CAP) Review instructions Open and review the general education, refresher course document (PDF)

43 THE CONTOURING ACCURACY PROGRAM (CAP) Contour the structure Method 1: Use the built-in contouring tools Method 2: Download images, contour with your own software, then upload the DICOM RT Structure Set

44 THE CONTOURING ACCURACY PROGRAM (CAP)

45 THE CONTOURING ACCURACY PROGRAM (CAP) Generate score Review results

46 THE CONTOURING ACCURACY PROGRAM (CAP) Watch video walkthrough Interview with the expert source of the gold contours Tips and commentary on: - window/level - super and inferior location - slice-by-slice analysis of gold contours ProKnow Feedback - Rate our materials with our star system - Write specific comments >> VIDEO WALKTHROUGH

47 THE CONTOURING ACCURACY PROGRAM (CAP) Re-do and practice as much and as often as you want Clear all to start over, or just continue to edit your current attempt No limit to how many times you try and no expiration of the tools or materials

48 THE CONTOURING ACCURACY PROGRAM (CAP) ProKnow plans to release one new structure per month New instances of each structure (i.e. same structure on different dataset) will be crowdsourced from members of the ProKnow community who proved excellent at that particular organ These Peer Experts will receive publication kudos and advanced merit badges, and even a subscription coupon for a colleague.

49 PROKNOW BADGES & ACHIEVEMENT RECORDS ProKnow will provide a Digital Badge system - Track your achievements - Strive for higher learning and skills - Benchmark your skills with established and scientifically valid testing mechanisms Enhance your professional résumé - Your achievements (i.e. your badges ) go with you no matter where you work. - Set yourself apart based on your advanced learning and proven, practical skills.

50 FROM W. E. DEMING TO THE JAPANESE GOVERNMENT National Trade Skill Testing & Certification (Japan) Goals - To appraise and certify professional skills - To encourage skilled workers to yet higher achievement - To: 1) enhance public s appreciation and 2) raise the profile and economic status of the skilled worker Successful Implementation - Started in 1959, now applied to 137+ different trades - Grade Advanced High Proficiency Superior

51 CERTIFICATION AND DIGITAL BADGES A proper evaluation of an individual s vocational abilities allows them to market their abilities and also provides a rough guide to set goals for future career development. For companies, it serves as a guideline for proper personnel evaluation. NTSTC Statement Japanese Government In a crowded market, we need ways to differentiate between those that are qualified and those that are not It is still tied to learning the people have to earn the credentials through specific learning activities but the badges go with them beyond the walls of the current employer. Brandon Hall Group Research and Analysis in Human Capital Comments on Open Badge System

52 GOALS OF THE CONTOURING ACCURACY PROGRAM Short Term (Immediate) - Skills improvement and CE credits - Digital badges as incentive towards continual improvement & proof-of-skills to add to professional résumé Near Horizon (1-2 Years) - Vast library of CT and MR image sets - Advanced practice certification for Medical Dosimetrists - Requirement for ACR, ASTRO site accreditation Big Picture Future (2+ Years) - The entire world of anatomy experts in radiation oncology is interacting via ProKnow: sharing data & techniques, encouragement, and knowledge. DREAM BIG.

53 THANK YOU. Questions?