ABR Exams of the Future Examination Details Impact on Training Duane Mezwa, MD, FACR ABR Trustee, GI Section Professor and Chair, Radiology Oakland University William Beaumont SOM
Disclosure The material presented today best represents the current thoughts and processes in developing the Core and Certifying exams of the ABR.
What We ve Done Until Now Two comprehensive written examinations Physics (first offered September of 2 nd year) Diagnostic (first offered September of 3 rd year) Candidates must pass both to qualify. One comprehensive oral examination First offered in June of 4 th year 11 categories examined
Written Examinations: Details Physics Diagnostic, Nuclear, Radiobiology Recall and application of physics concepts No associated images Diagnostic ~20 MCQs in each category Evaluated information recall No associated images Scored as a whole (not by category)
Written Examinations: More Details Criterion-based exams Theoretically, failure rate could be 0-100%. Failure rates (first-time takers): Physics: 10-15% Diagnostic: 8-10% Both exams offered retake opportunities.
Oral Examinations: Details 25 minute oral examinations Centered on specific case images Minimal fact recall Focus on observation, synthesis, management Communication skills and judgment paramount Specific diagnosis less important than logical approach
Oral Examination: More Details Criterion-based exam Complex grading system Each candidate discussed in detail Potential for raising marginally failing scores Could fail 1-3 categories, but must re-test Statistics for first-time takers: Pass: 85% Fail: 5-7% Condition: 8-10%
Criticisms of Examination Process: Written Exams Physics examination is better suited to physics students, not radiologists. Rarely relevant to everyday practice Diagnostic exam: No images Fact retrieval (trivia?) only Not scored categorically candidates could pass knowing nothing about (fill in the blank).
Criticisms of Examination Process: Subjective Oral Exams No two tests the same For some candidates, pathologically stressful Exam reliability incalculable Expensive For ABR: ~400 examiners X 5 days For candidates: single inconvenient location
Response by the ABR: Changes in Exam Structure Qualifying (Core) exam: Comprehensive: Covers all of diagnostic radiology Includes physics, patient safety, noninterpretive skills Categorical: candidates must pass all sections. Candidates and training programs receive data. Opportunities for retakes: every six months Knowledge level: basic to intermediate Appropriate for ending third-year resident Timed for June of third year (except 2013) About 40% fact recall, 60% higher-level evaluation
Core Exam Certifying Exam Internship R1 R2 R3 R4 Fellowship/ employment 12 mos 12 mos 12 mos 12 mos 12 mos 12 mos 3 mos
Core Exam in Diagnostic Radiology Will be given September 30-October 4, 2013 Subsequent years will be offered in late June. Last full Oral exam in June 2013 Covers all of diagnostic radiology Comprehensive, categorical exam Candidates must pass all categories.
What Are These Categories? Organ systems MSK, Thoracic, GI, Urinary, Neuro, Pediatrics, Cardiac, Reproductive/Endocrine, Mammography, Vascular Modalities CT, MRI, RF/Fluoro, Nucs, Interventional, Ultrasound Fundamentals Physics, Safety
Exam Goals: Core Overarching goal: to protect the public by determining that individual candidates have attained competence in basic diagnostic radiology Specific goal: to create examinations that are relevant to current radiologic practice Specific goal: to make a reliable pass/fail decision about candidate performance in each category For this high-stakes exam, reliability requires ~60 questions/category. Emphasis on unique importance of Physics content = 110 questions
Breast Cardiac GI MSK Neuro Peds Thorax Repro / Endo Urinary Vascular Q# CT 60 IR 60 MR 60 NM/Molecular 60 Rad/Fluoro 60 US 60 Physics 110 Safety 60 Q# 60 60 60 60 60 60 60 60 60 60
Core Exam: Content Image-rich (unlike current written) ~40% fact recall (like current written) ~60% higher level (like current oral) Differential diagnosis Management Mostly MCQs, may include new question types
Example: Extended Matching For each patient whose clinical and imaging information is shown, select the most appropriate diagnosis from the list below. Each option may be used once, more than once, or not at all. A. Focal nodular hyperplasia B. Liver cell adenoma C. Cavernous hemangioma D. Inflammatory pseudotumor E. Pyogenic abscess F. Fungal abscess G. Nodular focal fat H. Biliary cystadenoma I. Lymphoma J. Solitary metastasis K. Intrahepatic cholangiocarcinoma L. Hepatocellular carcinoma M. Fibrolamellar carcinoma
1: 35 year old woman who underwent sonographic evaluation for mild abdominal discomfort. She was referred for MR imaging to characterize a solitary liver mass seen on that ultrasound. Images are obtained 30 seconds (A), 70 seconds (B) and 1 hour (C) after administration of gadobenate intravenously. A B C A. Focal nodular hyperplasia B. Liver cell adenoma C. Cavernous hemangioma D. Inflammatory pseudotumor E. Pyogenic abscess F. Fungal abscess G. Nodular focal fat H. Biliary cystadenoma I. Lymphoma J. Solitary metastasis K. Intrahepatic cholangiocarcinoma L. Hepatocellular carcinoma Key = A
2. 60 year old man with abdominal pain. Imaging performed elsewhere showed a liver mass, and he is referred for CT imaging to characterize it. Images are obtained before (A), 20 seconds after (B), 50 seconds after (C), and 5 minutes after (D) intravenous administration of iodinated contrast material. A. Focal nodular hyperplasia B. Liver cell adenoma C. Cavernous hemangioma D. Inflammatory pseudotumor E. Pyogenic abscess F. Fungal abscess G. Nodular focal fat H. Biliary cystadenoma I. Lymphoma J. Solitary metastasis K. Intrahepatic cholangiocarcinoma L. Hepatocellular carcinoma M. Fibrolamellar carcinoma Key = L
a b c d e f g h A coronal CT image obtained in a patient with ascites is shown. Label the following structures: a) Left subphrenic space b) Lesser sac, inferior recess c) Lesser sac, superior recess d) Transverse mesocolon e) Gastrohepatic ligament f) Morison s pouch g) Left paracolic gutter h) Root of intestinal mesentery
Core Exam 2013+ Nuts and Bolts ~660 items Two half-day sessions during a single week Examination center(s) One large Chicago center Smaller center in Tucson ABR commitment: distributed exam by 2018 Condition exams in June/October
ABR FastPass ~30,000 square feet Palm Vein Biometrics 5440 N. Cumberland Ave., 4½ miles and the second train stop
Results: Test Scoring Each category (18) will be scored separately. Score will be based on pre-test assessment of item difficulty (Angoff method). Each test item will be scrutinized for its performance (p-value, r- value), and adjustments made for problem items. Three possible results: Pass (all 18 categories passed) qualify for Certifying Exam Fail (6 or more categories failed) must retake whole exam Condition (1 to 5 categories failed) must retake those categories failed
Core Examination: Impact on Training Candidates must be exposed to all basic diagnostic radiology by end of third year. Core anxiety and review sessions will occur early in third year. Content of review sessions will change.
Core Exam: Review Sessions Physics content Should focus on practical applications. Physics resource: rsna.org/education/physics.cfm Diagnostic content 40% fact recall even though image-rich, it s reasonable to study topics prevalent on previous written exam. 60% resembles oral boards standard board review. Resource: Core study guide on theabr.org Resource to come: practice exam 90 minute simulation
Practice Exam A taste of the full exam Shortened version of all sections A WIP To be on line by end of 2012 90 minutes -- to simulate the real exam
RSNA/AAPM Physics Modules 46 complete, 1 in progress 30,081 enrolled, 10,467 completed Popular modules: Basic Radiation Biology Basic Concepts in Radiography Radiation Effects Basic Principles of Nuclear Magnetic Resonance Atoms, Radiation and Radioactivity Radiation Risks
BEST Way to Prepare Come to work and PARTICIPATE Fully
Piloting the Core Exam First core pilot: May 2011 Aims: Nuts and bolts: software, image quality, time allotted, interface Psychometric item data: p, r values; reliability
First Core Pilot Population: Residents taking Oral Exam Incentive: Passing score on electronic exam could raise condition in that category. Caveat: Would not raise if 68 on Ora.l Would not raise if failed 4 categories. Only six categories: MSK, Neuro, VIR, Peds, Breast, and Cardiac
Core Pilot 2011: Results 1,117 (81.5%) took exam. Neuro 26%, Breast 7%, rest ~18% p-value ranged from 0.66 0.81 (mean 0.72) Average passing score 69.2% Post-test survey: good questions, enough time, appropriate difficulty, interface good
Core Pilot 2012 Will again be given at this year s May Oral exams. Two categories will be offered. One will be chosen by ABR, and other is candidate s choice. Can be used to raise 2 sections in Orals.
Next Expanded Core Pilot Full Exam to be given to PGY4 residents in ~ June 2013. Will be given at both Chicago and Tucson. Will try to accommodate all residents. Feedback will be given to help guide exam to be taken in September. Does NOT Count in place of real exam. For now a one-time only event in 2013
Core Exam: How Are We Doing? 13 item-writing committees (physicist embedded in each) Scorable units submitted: 5580 Scorable units approved: 3284 All study guides completed and revised X 1. First test assembly meeting: January 2012
Exam Week Logistics
After the Core Candidates can decide (within constraints of their chosen training program) on their future practice. General vs. subspecialized Which subspecialty(ies) Begin the process of continuous learning.
Certifying Exam in Diagnostic Radiology Will be first administered in October 2015 Contains five parts (but questions will appear in random order) Noninterpretive skills Essentials Things every physician should know Things every diagnostic radiologist should know Clinical Practice Areas (3 self-selected CPAs) Things this specific radiologist should know
Clinical Practice Areas (CPA) Twelve categories Organ system: MSK, Cardiac, Thoracic, GI, Urinary, Neuro, Peds, Breast Technology: US, VIR, Nuclear Radiology General Candidates can choose any combination of CPAs. Items will vary in both difficulty and scope. Level 1: fundamental Level 2: advanced Candidates selecting a CPA more than once will receive a higher proportion of level 2 items.
Certifying Exam: Content (CPA) Emulates clinical practice Will include normals and variants. Will include important findings outside chosen area. Appropriateness, clinical vignettes, management decisions Item types Familiar: MCQs, extended matching Unfamiliar*: Structured reporting, script concordance testing *Examples posted on ABR Website >1 year before use
Certifying Exam: Goals To confirm candidate has acquired and maintained necessary skills to practice independently Dual role Final ABR Certification Exam First exam of practice-based learning (like MOC)
Certifying Exam: Nuts and Bolts ~300 items 5-hour exam (one half-day) Exam center(s): Chicago, Tucson Exams offered October/February
Certifying Examination: Scoring Each of the five modules scored separately. Passing standard established prior to test administration (Angoff method). All problem items analyzed, adjustments made. Two possible results: Pass: all 5 modules passed certified 10 years Fail: any module failed Retakes possible every six months
Certifying Exam: Impact on Training After Core passed, some candidates may begin self-directed specialty training. Some part of 4 th year available. Extent depends on program needs. Supplement, but will not replace Fellowship. May impact first 3 months of practice.
Certifying Examination: Candidate Preparation Good professional practice Medical judgment Communication skills Lifelong learning Critical thinking Self-assessment Recognize the six competencies? Study Guides
Certifying Exam: Resources Noninterpretive skills and Essentials Study guides to be posted on theabr.org, Clinical practice modules Study guides at theabr.org CME, SAMs Practice-based learning Society-produced educational modules
Certifying Exam: How Are We Doing? 15 item-writing committees Submitted scorable units: 4439 Approved scorable units: 3323
Questions? dmezwa@beaumont.edu or mezwa@theabr.org