How to Cheat on Credentialing Exams Presenters: Danny Breidenbach, Applied Measurement Professionals Stephen Hennessey, Prometric Mark Poole, Pearson Promoting Regulatory Excellence Before the Exam Danny Breidenbach, Applied Measurement Professionals 2 Security During Examination Development Internal vs. External Testing company vs. Credentialing organization and Subject Matter Expert (SME) 3 San Francisco, California 1
Security During Examination Development Physical Access Staffing Inventory / Storage Other Functions Translation, Printing, Artwork, etc. Working with SMEs 4 Development Physical Access Secure Room/Area Vault Key Access Limited Authorized Access Print copies vs. Electronic copies Chain-of-Custody Shredding including notes/drafts 5 Development Staffing Staff Confidentiality Agreements Conflict of Interest Audits 6 San Francisco, California 2
Development Inventory and Distribution Printing (internal vs. outsourced, serial # s, sealed booklets, shrink wrap) Shipping (multiple counts, traceable shipping, courier) Electronic Transmission Web Access 7 Development Other Functions Translators Artwork Supplies for Practical Exams Printers Couriers 8 Development Working with SMEs Face-to-Face Meetings Passwords Selective Access Permission Specific Time Period Read-only, non-print, non-export Screen security Web-conferences 9 San Francisco, California 3
Summary Security During Development Main threat: unintended exposure of materials Responsibility must be shared among testing company, credentialing organization, volunteer subject matter experts 10 During the Exam Stephen Hennessey, Prometric 11 Various Types and Levels of Threats Single cheater Collusion Proxy testing Item harvesting Hackers 12 San Francisco, California 4
Common Cheating Methods Copying from other examinees Hidden notes in clothes on hands or legs inside books or exhibits underneath hats or taped to brim Inside bottle/ labels inside food packages Accessing notes on a break Notes hidden in bathroom Bribery of Test Administrator Paper Cheating Notes Cheating Notes 13 Facility- Computer Based Testing Separate area for check-in & testing rooms Window into test room & surveillance system video & audio recording Signs posted notifying of recording Secure lockers for personal items Partitions between workstations or privacy screens Surveillance cameras view desktops ADA compliant Facility- Paper Based Testing Separate areas for check-in and testing All testing materials kept secured by proctor Optimum proctor to examinee ratio- no more than 1:20 Provide storage for personal belongings Exam booklets numbered, tracked by examinee Maintain examinee seating chart Monitor examinees on breaks San Francisco, California 5
Proctors - Best Defense Available Qualifications Background checks- include 5-7 year country criminal search, State Police are an inexpensive option Formal training & certification program with annual recertification required Monitoring Conflict of interest restrictions proctor agreement Examinees monitored at ALL times Conduct frequent walk-throughs Be vigilant for suspicious 16 Emerging Threats 17 Emerging Trends in Detection Technology Hand Held Metal Detectors Wireless Jamming Devices & Cell Signal Detectors Screen Monitor Obfuscation San Francisco, California 6
Biometric Identity Management Capture examinee Scan Validate ID Capture Fingerprints Informationfrom ID Scan & Validate ID Capture Fingerprints The examinee s ID is swiped in the combined magnetic strip and 2D barcode reader which verifies the legitimacy of government issued ID (North America only). The ID is then placed on a flatbed scanner which captures an electronic image of the photo ID and uses optical character recognition (OCR), to compare printed data and encoded data. The examinee then places a finger on the fingerprint scanner, which collects a unique physical characteristic which is then linked to the examinee s identity record. Day of testing - steps to mitigate risk during test delivery Mark Poole, Pearson After the Exam 21 San Francisco, California 7
Security After the Test Data Forensics Monitoring Websites Item Updating Facility Screening Audits Incident Reporting 22 p-value: pbis: After Data Forensics - Items Percentage of candidates who answer an item correctly Correlation between performance on an item and performance on the overall exam option stats: option pbis: Percentage of candidates who answer each option correctly Correlation between performance on the options and the overall exam 23 After Data Forensics - Items p-value: pbis: options: O pbis: Omits Too high, too low Too low Distractors with no responses Distractors with more responses than key Distractors higher than key Above expectations 24 San Francisco, California 8
After Data Forensics - Candidates Suspicious Candidate Activity Test for significant differences in pass rates Better performance on harder items High scores in low test times Suspicious improvement in scores 25 After Monitoring Websites Cram Schools Sample Exams False Advertising Social Networking 26 After Item Updating Monitor Item Statistics Replace Items Add New Items 27 San Francisco, California 9
After Facility Screening Look for Notes in Testing Station Screen Restrooms and Hallways 28 After - Audits Test Center Audits Secret Shopper Programs Inventory Continuing Education for Proctors 29 After Incident Reporting Documentation Evidence Reporting to Client 30 San Francisco, California 10
Examples Cheating Incidents 31 Speaker Contact Information Danny H. Breidenbach, Psychometrician dbreidenbach@goamp.org Stephen Hennessey, Director Global Security steve.hennessey@prometric.com Mark Poole, Product Strategist Mark.Poole@pearson.com Tadas Dabsys, Executive Vice President tadas@psionline.com 32 San Francisco, California 11