Michigan Department of Education

Transcription

1 Michigan Department of Education NEW REQUIREMENTS IN ESSA AND PEER REVIEW AND STATES RESPONSES ON TEST INTEGRITY AND SECURITY NATIONAL CONFERENCE ON STUDENT ASSESSMENT 2017 Pietro Semifero Online Assessment Manager Peer Review Critical Elements 2.4: Monitoring Test Administration 2.5: Test Security 2.6: Systems for Protecting Data Integrity and Privacy Test Security Audit March 2016 Caveon test security audit to identify operational risks and propose recommendations Document review Two days of on-site interview and inspection Output: 31-page report on exemplary processes and recommendations for improvement Document the old, develop the new 1

2 Test Security Audit 1. Security Plan 2. Roles and Responsibilities 3. Budget and Funding 4. Standard Forms and Agreements 5. Test and Item Design 6. Test and Item Development and Maintenance 7. Test Publication 8. Test Administration 9. Test Scores and Results 10. Physical Security 11. Information Security 12. Web and Media Monitoring 13. Security Awareness and Training 14. Security Incident Response Plan 15. Conducting Security Investigations Test Security Audit Pre-Review Preparation Evidence Outside view, not we know what we do Expands the range of the possible You really have to make the improvements Implementing best practices requires resources 2.4 Assessment Monitoring Random Contract with Measurement Incorporated Standard instructions, checklist Covers all programs tied to state accountability Targeted Contracts with former State Police detectives Standard instructions, checklist, and look around a bit Sites selected based on previous year s assessment incidents 50 sites per year. More random if fewer targeted. 2

3 2.5 Test Security, 2.6 Test Material Integrity PDIR: Prevention, Detection, Investigation, Remediation State Test Security Handbook Documented policies and procedures for: Administration manuals, training Chain of custody for paper materials Secure online testing Incident detection and reporting Investigating prohibited behavior, misadministration Consequences, contingency plans Protecting the integrity of test materials Assessment Integrity Guide MDE publishes an Assessment Integrity Guide (AIG) that all test administrators must acknowledge reading before administering assessments Goal of the AIG is to represent the expected professional conduct of educators who administer state assessments With the advent of online testing, the AIG was extensively refreshed in 2015 with input by John Olson and John Fremer, finalized after the security audit Reviewing for next round of updates, now that we have more experience with online testing Assessment Integrity Guide The Assessment Integrity Guide focuses on 4 major assessment integrity themes: Prevention standards and best practices for test integrity and security to prevent irregularities from occurring Detection guidelines for monitoring, reporting, and working with MDE when irregularities are found Follow-Up Investigations guidelines for investigating irregularities Remediation guidelines for schools to work with MDE to remediate issues and ensure valid results for all students 3

4 Assessment Integrity Guide Useful to the field: best practices, establish expectations Useful to the office: documented policies and procedures, manual to follow in the event of questions Useful for review: documentation that is polished, reviewed, and approved by the State Board of Education Test Security Specialist Staff member dedicated to test security and integrity Duties include: Developing and updating test security policies Developing and implementing test security training Responding to incident reports and tips Assessment monitoring, social media monitoring Data forensic analysis Investigating irregularities Working with schools on remediation Test Security Specialist Manual Written by the Test Security Specialist: write down what it is that you actually do on a day-to-day basis Practical, not theoretical Reference guide for consistency Training manual for turnover, backups Identifies: Duties Procedures Documentation Contacts 4

5 Document the Old: Response Plans Security incident response planning Document practices for analysis and monitoring Incident triage and investigation Incident response matrix Communications plan Document everything Document the Old: Formal Investigations Defined plans for investigating more serious incidents Define what rises to the level of an incident Establish thresholds in advance rather than ad hoc Investigation policies and procedures Standard responses, consequences Factors to consider in weighing appropriate remediation Review panel for appeals New Developments: Online Training A series of four online modules through Michigan Virtual University Walks through key topics from the Assessment Integrity Guide For staff and schools, new and refresher Focused on practitioners: preparation, administration, incident response (less on investigation and remediation) Includes examples and quizzes 5

6 New Developments: Data Forensics Ideal case of what data forensics we would like Realistic case of what we have resources for Now Development plan Division of labor between contractors and staff Identify values that trigger investigations or invalidations Identify and improve internal understanding of forensics It is not psychomagic Political support for data-based invalidations 6