AAMC Standardized Video Interview Update

Size: px

Start display at page:

Download "AAMC Standardized Video Interview Update"

Chrystal Melton
5 years ago
Views:

1 AAMC Standardized Video Interview Update Society of Academic Emergency Medicine Renee Overton, MBA, Sr. Director, Residency & Fellowship Program Solutions Rebecca Fraser, Ph.D., Manager, Admissions and Selection Research May 17, 2018

2 Agenda SVI Overview Evaluation Plan Research Partnerships What s New for SVI Progress Report on SVI Pricing Questions 2

3 The Medical Education Continuum Premedical Medical School Residency and Fellowships Practice Today s Discussion 3

4 What s the rationale for the SVI? PDs consistently rate behavioral competencies as important for trainees to demonstrate when they enter residency Current information about behavioral competencies is hard to use and interpret Most information about applicants behavioral competencies comes from interviews Specialties and programs are developing their own local tools 4

5 5 AAMC Standardized Video Interview (SVI)

6 Academic Emergency Medicine and AAMC Partnership SAEM Andra Blomkalns, MD Steve Bird, MD AACEM Richard Wolfe, MD CDEM Rahul Patwari, MD Kathy Hiller, MD CORD Fiona Gallahue, MD Gene Hern, MD EMRA Zach Jarou, MD AAEM-RSA Ashely Alker, MD Mary Calderone Haas, MD Michael Wilk, MD GSA Nicole Deiorio, MD Yolanda Haywood, MD Christopher Woleben, MD AAMC Staff John Prescott, MD Renee Overton, MBA Dana Dunleavy, PhD Rebecca Fraser, PhD Malika Fair, MD 6

7 What are the goals of the SVI? Designed for use, along with other selection data, in deciding whom to invite to interview, to: Short-term - Provide information about applicants non-academic competencies - Balance emphasis on Step scores in the selection process - Broaden the pool of applicants invited to in-person interviews Long-term - Reduce staff time required to conduct a more holistic review - Give staff more time to focus attention on applicants who may be a good fit for their program 7

8 What format does the SVI take? Online Text-based questions Audio/video response 6 questions 21 minutes of interview time 30 sec to read each question 3 min to respond each question 8

9 What does the SVI measure? Questions were reviewed by residency program directors for: o o Relevance to targeted ACGME competencies Potential for bias Only questions that survived the review were retained Interpersonal Skills and Communication (ICS) Knowledge of Professional Behavior (PROF) 9

10 How is the SVI scored? Behaviorally-anchored scoring rubrics were developed for each target competency Rating scales were reviewed by Subject Matter Experts from residency programs o o Relevance to target competencies Appropriateness for each level on the rating scale Responses to each question are made on a 5-point scale and summed to create a total score 10

11 Raters Rater cohort o HR professionals with expertise in recruiting and interviewing o Diverse In-person training (12 hours) o Emergency Medicine resident job o Unconscious Bias o Standardized Rating Process, including focus on content o Trained to an EM faculty true score standard Practice and feedback on ratings (8 hours) 20 hours of training before making ratings of EM applicant interviews 11

12 12 Evaluation Plan

13 Validity The process of validation involves accumulating relevant evidence to provide a sound scientific basis for the proposed score interpretations Ongoing Multiple sources of evidence Relationship with relevant performance Response process Sources of Evidence Consequential Relationship with unrelated variables Content Relationship with related variables Internal structure 13 Source: American Educational Research Association, American Psychological Association, National Council on Measurement in Education. Standards for Educational and Psychological Testing. 3rd ed. Washington, DC: American Educational Research Association; 2014.

14 We will evaluate each source of evidence by conducting research in different areas Psychometrics PD and Applicant Reactions Fairness & Preparation Predicting Non- Academic PGY-1 Performance 14

15 More data will be available each year 2017 Spring 2017 Fall 2018 Spring 2018 Fall 2019 Spring 2019 Fall 2020 Spring Psychometrics SVI SVI SVI PD & Applicant Reactions SVI SVI SVI Fairness & Prep. SVI SVI SVI Predicting PGY-1 Performance SVI SVI

16 First, let s review results for applicant and PD reactions Psychometrics PD and Applicant Reactions Fairness & Preparation Predicting Non- Academic PGY-1 Performance 16

17 Applicant Survey Methods Survey #1 Survey #2 Administration Online Immediately following completion of the SVI Jun 6 to July 31 Sample 2,906 participated 83% response rate Content 11 questions Preparation Applicant reactions Open response Online After scores were released Oct 18 Nov 13 2,074 participated 58% response rate 29 questions Perceptions of the Selection Process SVI Experience Future of the Selection Process Open response 17

18 Applicant Reactions 38% Satisfied with the SVI overall 84% SVI instructions were clear 18 67% 50% professionalism & interpersonal/communication skills (ACGME) Content of SVI questions are related to the types of activities required of residents Drew on past experiences to answer questions

19 Program Director Survey Methods Administration Online November 2017 Sample 125/175 ACGME-accredited EM programs participated 71% response rate 92% of respondents were program directors Content 27 questions User reactions to SVI scores and videos Perceptions on the future of the SVI 19

20 Program Directors used SVI scores cautiously 54% Used in ERAS 2018 selection 20 70% 33% 57% SVI scores were not important in deciding whom to invite to the in-person interview professionalism & interpersonal/communication skills (ACGME) SVI helped me compare applicants ICS and PROF Somewhat or more likely to use in ERAS 2019 selection

21 Program Directors Use of SVI Videos 80% PDs who viewed videos out of curiosity 111 Median # of videos viewed by PDWS users 71% Applicants with 3+ videos viewed by PDWS users Programs were more likely to view videos from: US-MDs Applicants with higher Step 1 scores Applicants with higher SVI scores There were not differences in views for applicants by race/ethnicity or gender. 21

22 Next, let s review results for fairness and preparation Psychometrics PD and Applicant Reactions Fairness & Preparation Predicting Non- Academic PGY-1 Performance 22

23 Women had slightly higher SVI scores than men (3.1) Male (N=2,311) 19.5 (3.1) M-F d = -.21 Small Effect Female (N=1,219) 23

24 SVI scores did not differ by race and ethnicity (3.0) 19.3 (3.0) 18.9 (3.1) 19.1 (3.1) White (N=2,001) W-B d = -.06 No Effect Black (N=247) W-H d =.09 No effect Hispanic (N=286) W-A d =.01 No Effect Asian (N=613) 24

25 SVI scores have small correlations with scores from Step 1, Step 2CK, or Step 2CS Threshold for a small correlation 0.0 USMLE Step 1 (N=2,977) USMLE Step 2 CK (N=2,596) USMLE Step 2 CS (N=1,058) 25

26 Finally, let s talk about predicting performance Psychometrics PD and Applicant Reactions Fairness & Preparation Predicting Non- Academic PGY-1 Performance 26

27 27 Local Validity Study Participants

28 Local Study Design Follow 2 cohorts (2018 and 2019) through their first year of training. SVI Scores Correlation Local Selection Data File review scores Invited/not invited to the in-person interview In-person interview scores ROL, etc. Intern performance (possible outcomes) Milestone ratings End-of-Shift ratings 28

29 Timeline for the Validity Study (SVI cohort) Date Activity Spring 2018 Set up infrastructure Confirm programs Finalize research design IRB, DUA Late Summer/Early Fall 2018 Recruit interns Identify outcomes Exchange selection data Fall 2018 Set-up data exchange process Winter 2018 Exchange performance data (round 1) Winter 2018/Spring 2019 Analyze and disseminate results Summer 2019 Exchange performance data (round 2) Fall 2019 Analyze and disseminate results Repeat for the SVI 2018 cohort 29

30 Other planned research for 2018 Relationship between SVI and esloe ratings (partnership with CORD) in progress Relationship between SVI scores and Step 2 Clinical Skills subscores (partnership with NBME) in progress Relationship between SVI scores and ACGME milestones (partnership with ACGME) preliminary conversations 30

31 Research Partnership Review research questions, methods, and analyses Connect you to colleagues pursuing similar questions Data steward obligation Per the SVI Terms and Conditions, Program Users who wish to conduct research utilizing SVI data are required to submit their manuscript or poster presentation to the AAMC for review and approval before submitting to a conference, journal, or other outlet. 31

32 Research Partnership (cont.) Provide the following information: Researchers (and the institution(s)) involved Contact Information ( & Phone #) of Researcher Purpose and research questions Sample description and size Data to be used Methodology and analysis plan 32

33 Research Partnership (cont.) We are NOT trying to suppress research. We want to collaborate to ensure high quality research on the SVI. Six research requests received to date: 5 Approved 1 Deferred Action Increase sample size to support research question(s) 33

34 34 What s New for SVI

35 SVI Administration Timeline Date Activity February 7 th Announced SVI and launched new website May 16 th SVI Registration for Programs opens June 7th MyERAS and SVI open July 15 th at 11:59 p.m. ET SVI window closes September 1 st Scores released to applicants and medical schools (expected) (applicants who interviewed by July 15, 2018) September 15 th Scores and videos released to programs (applicants who (expected) interviewed by July 15, 2018) October 15 th Scores and videos released to programs (applicants who were granted an extension and interviewed by September 15, 2018) For a detailed schedule, refer to the AAMC SVI Essentials for the ERAS 2019 Season. 35

36 What s New for SVI Applicants Early June to July 15 Revised website More flexible extensions process Updated Tips for Applicants Online Practice Interviews Scores released sooner Medical Schools Score reports available Programs Registration open in April PD training open in July Filters Export capability User s Guide 36

37 Program Director Training Self-paced online training modules: available starting in July 2018 SVI Content (10 min) Interpreting SVI scores (15 min) Incorporating SVI scores into the selection process (15 min) Unconscious Bias Training (15 min) Webinar (late-august/ early-september 2018) User s Guide: available starting in August 2018 Completing the training will help you and your team understand SVI scores and will give you ideas on how to incorporate them into your selection process. 37

38 38 SVI Pricing Update

39 What criteria will be used to evaluate pricing models? Value to user Operational costs Market comparison 39

40 Roadmap for finalizing SVI pricing Focus Groups and Interviews (Summer 2018) DIOs Program Directors Chairs and Program Directors Survey to Stakeholders (Summer 2018) AAMC Deliberates (Summer 2018) AAMC Decision (Fall 2018) 40

41 41 Questions

Overview of ABET Kent Hamlin Director Institute of Nuclear Power Operations Commissioner TAC of ABET

Overview of ABET Kent Hamlin Director Institute of Nuclear Power Operations Commissioner TAC of ABET 1 st National Meeting on Improving Education and Training For Chinese Nuclear Power Industry Personnel