EPSO Gilles Guillard Hudson Nikola Trbovic Prometric Nicoletta Vullo. #EATPconf

Transcription

1 EPSO Gilles Guillard Hudson Nikola Trbovic Prometric Nicoletta Vullo

2 Presentation outline 1. Starting situation - About EPSO, Prometric & Hudson - Two single-provider collaborations - Original candidate trajectory 2. Need for a new candidate trajectory - Insertion of intermediate eliminatory test - New trajectory - Different providers, different solutions Hudson s E-tray Prometric 3. In practice - Multi-provider collaboration process - Reliability - Equal opportunities - What worked well - Lessons learned September

3 28-30 September

4 European Personnel Selection Office EPSO selects highly qualified staff for September

5 European Personnel Selection Office The EU Institutions September

6 European Personnel Selection Office A few figures Since the creation of EPSO in 2003: Over 600,000 candidates have participated in EPSO selection processes Over 510,000 EPSO candidates have taken a computer-based test delivered via Prometric Over 70,000 CBT exams were delivered in 2015 Between 1 % and 2 % of all candidates in EU open competitions are successful and can be recruited Some competitions have attracted up to 52,000 applicants for the approx. 300 positions offered September

7 About Prometric Present Prometric Wholly owned subsidiary of ETS A trusted market leader in technology enabled testing and assessments Comprehensive suite of services, including test development, test delivery and data management capabilities 350+ clients in every major industry segment Over 9 million exams delivered annually 160 countries 26 languages September

8 Starting situation: Two single-provider collaborations Prometric has been providing Computer-Based Admission Testing to EPSO since 2005 Over the years, Prometric has implemented innovative and tailored solutions to meet EPSO s specific testing needs Prometric exam delivery for the EPSO includes: Unproctored internet-based testing in candidates locations; Proctored Multiple Choice exams of various types, Case Studies, Audio-Video exams in the Prometric testing network; In all 24 official EU languages and across all 28 EU Member States and worldwide September

9 About Hudson September

10 About Hudson Belgium Since 1982 ( De Witte & Morel): more than 30 years experience Market leader with a strong reputation 6 regional offices: Brussels, Gent, Antwerp, Kortrijk, Hasselt & Louvain-La-Neuve Over 200 people including over 100 consultants More than 85% loyal customers Worldwide Research & Development Center located in Belgium Partnerships with universities & top business schools e.g. Vlerick Business School, Université catholique de Louvain, Louvain School of Management, UGent, RECRUITMENT & SELECTION INTERIM MANAGEMENT & CONTRACTING LEADERSHIP DEVELOPMENT JOB GRADING ASSESSMENT & DEVELOPMENT CENTERS CAREER GUIDANCE COMPETENCY & PERFORMANCE MANAGEMENT REWARD MANAGEMENT, COMPENSATION & BENEFITS September

11 Starting situation: Two single-provider collaborations Hudson as trusted advisor for EPSO Assessment Centres Develops new test material (and test variations) and accompanying documentation (marking grids, manuals, guidelines) Adapts pre-existing tests and provides off-the-shelf exercises Trains permanent and non-permanent Selection Board members, markers and EPSO support staff Provides advice to EPSO on the further development of its Assessment Centres Provides IT tools, support and services relating to the Informatics tools used by EPSO September

12 Starting situation: Original candidate trajectory (AD-cycle) 1 month 4 to 6 weeks 2 to 3 months 33,837 candidates 25,544 candidates 504 candidates 185 laureates 1 MONTHS September

13 2. Need for a new candidate trajectory September

14 New candidate trajectory Intermediate test: E-tray Decision to insert an intermediate eliminatory test between pre-selection tests (CBT) and Assessment Centre: invite 10 x the number of candidates sought to take the e-tray in order to reduce invitees to AC to 2 x the number of candidates sought Invite only the very best candidates to AC Cost effective (financial & human resources) Improved gender balance Assesses generic competencies at an early stage Automated scoring Limited scope for training September

15 New candidate trajectory: Need for a multi-provider collaboration September

16 New candidate trajectory 40,666 candidates 27,388 candidates 2,190 candidates 541 candidates 235 laureates MONTHS September

17 New candidate trajectory: Different providers; different solutions Implementing the new trajectory was not without challenges: EPSO s needs Hudson Prometric Testing facilities and administrators in 28 MS E-tray to measure competencies IT platform for test administration (need for alignment) Equal opportunities for all candidates in all MS Parallel tests and randomisation of test material Standardised testing conditions in all MS September

18 New candidate trajectory Description of Hudson E-tray Computer-based test using documentation provided in the form of an inbox Duration: 50 minutes Assesses 4 general competencies: Analysing & Problem-solving; Delivering Quality & Results; Prioritising & Organising; Working with Others September

19 New candidate trajectory Hudson E-tray Step 0: EPSO request Step 5: Selection Board validation & adaptations Step 6: Linguistic revision and translation Step 1: Development planning & check NoC Step 4: Develop exercise Step 7: Trial session Step 8: Psychometrics Step 2: Contact Subject Matter Expert Step 3: Meet Subject Matter Expert Step 9: IT programming 9/20/

20 New candidate trajectory Hudson E-tray September

21 New candidate trajectory Prometric solution Prometric s Web-launch technology supports clients who have their own browser-based test driver but who require a secure, proctored environment for test delivery Web-launch acted as a proxy for Hudson s website, providing an interface with Prometric s administrative system Hudson s website had to operate using the infrastructure in place at the Prometric testing centres In summary, using Web-launch required integration between Prometric s and Hudson s systems 9/20/

22 Prometric solution Web-launch functionality Start the browser and navigate to Hudson s start URL using agreed parameters passed in the URL Interface to the administrative system to set the status of test delivery Maintain a test seat session timer Ensure that candidates navigate only to known internet domains 9/20/

23 Prometric solution Web-launch functionality Monitor for pages that indicate conditions such as end-of-test or error Monitor for a loss of internet connectivity Support restart of a candidate s sessions Close the browser and return control back to the administrative system 9/20/

24 Prometric solution Hudson s browser Supported the Start and End control pages Supported checkpoint and recovery of testing sessions Supported test-specific timing (for ex-time extension for Testing Accommodations) 9/20/

25 3. In practice September

26 In practice: Multi-provider collaboration process Explore the capabilities of Hudson and Prometric Define the requirements for information flow between the three parties IT implementation (platform integration) Training of test administrators by Prometric Trials on dummy candidates in Brussels First rollout: 1,500 candidates 6 days 28 MS September

27 Hudson and Prometric readiness for Weblaunch implementation Prometric and Hudson determined parameters to be passed from Web-launch to Hudson s website Prometric and Hudson determined Start and End URLs for QC and Production Prometric published service scripts for Hudson tests, representing parameters, URLs, and other configuration decisions 9/20/

28 Hudson and Prometric readiness for Weblaunch implementation A test harness was created and Prometric and Hudson performed integration testing in the QC environment Four dry-runs were held in Brussels, one of which used real candidates eligibilities An escalation matrix was established between EPSO, Prometric and Hudson prior to going live. 9/20/

29 Prometric readiness for test delivery in the Prometric network Information was gathered on the ISP bandwidths of all EU28 testing centres Hudson s exams were downloaded in Prometric technical support labs in order to issue knowledge-based documentation Beta testing was conducted in all participating EU28 MS testing centres (38 locations in total) Hudson verified all results A Command Centre was put in place during go-live, and bi-hourly reports were issued during testing 9/20/

30 In practice: Challenges The process was not without challenges: A few limits to the integration of both IT platforms (example: abort functionality not possible) Candidates had some questions on the scoring (need to publish more information on scoring methods on EPSO website/ombudsman enquiry) Isolated technical issues encountered during testing (restarts, loss of connectivity, etc.) September

31 In practice: Reliability E-tray test is stable and reliable. Analysing & Problem Solving Delivering Quality & Results Prioritising & Organising Working with others Cronbach's Alpha September

32 In practice: Equal opportunities In terms of equal opportunities, no adverse impact for gender Count Gender_Female 341 Gender_Male 438 Total 779 ANOVA Sum of Squares df Mean Square APS_C Between Groups Within Groups Total DQR_C Between Groups Within Groups Total F Sig. POR_C Between Groups Within Groups Total WWO_C Between Groups Within Groups Total September

33 In practice: Equal opportunities Increased gender balanced achieved: WITHOUT E-TRAY INTERMEDIATE STAGE WITH E-TRAY INTERMEDIATE STAGE September

34 In practice: Equal opportunities In terms of equal opportunities, no adverse impact for test language Count language DE_BE 18 EN_INT 702 FR_BE 59 Total 779 APS_C DQR_C POR_C WWO_C Between Groups ANOVA Sum of Squares df Mean Square F Sig Within Groups Total Between Groups Within Groups Total Between Groups Within Groups Total Between Groups Within Groups Total September

35 In practice: Equal opportunities In terms of equal opportunities, no adverse impact for test day Count Testingday 09/21/ /22/ /23/ Total 779 APS_C DQR_C POR_C WWO_C Between Groups Sum of Squares ANOVA df Mean Square F Sig Within Groups Total Between Groups Within Groups Total Between Groups Within Groups Total Between Groups Within Groups Total September

36 EPSO s idea in practice What worked well What worked well Interaction between the three parties was very positive, prompt and transparent Dry-runs were effectively used to test and validate the exam prior to delivery and helped identify and solve a number of issues Command centre structure and daily testing reports were effective Operations worked well for a first-time delivery, with no candidates displaced and five re-tests in total Escalation path and communication with Hudson worked very well September

37 EPSO s idea in practice: Lessons learned The multi-provider collaboration was a positive experience Will be retained for future competitions Further improvements are being introduced to enhance the process A dashboard was created to facilitate follow-up in real time by EPSO September

38 28-30 September