INTEGRATING IRT ANALYSIS INTO LMS FOR ITEM POOL OPTIMISATION

Transcription

1 INTEGRATING IRT ANALYSIS INTO LMS FOR ITEM POOL OPTIMISATION P Fotaris, King s College London T Mastoras, University of Macedonia

2 COMPUTER-AIDED ASSESSMENT Computer-Aided Assessment (CAA) Use of computers in the assessment process Self-assessment tests Multiple Choice Questions Automatic evaluation of the responses provided by examinees Major trend in MOOCs and academic institutions worldwide

3 TEST QUALITY ISSUES Criticism of test quality: Many test items (questions) are flawed at the initial stage of their development 50% of the test items will fail to perform as intended (Haladyna, 1999) Unreliable results of examinee performance Two major approaches to item evaluation using item response data: Classical Test Theory (CTT) Item difficulty Item discrimination Item Response Theory (IRT) Item difficulty (b) Item discrimination (a) Guessing (c)

4 CTT VS IRT CTT Relatively simple to compute and understand True-score is defined in the context of a specific test Not as sensitive to items that discriminate differentially across different levels of ability Not as effective in identifying items that are statistically biased IRT Parameters are not sample- or test-dependent Allows a meaningful comparison between the ability of a person and the difficulty of an item Requires relatively complex estimation procedures Requires large sample sizes (>1,000) (Swaminathan and Gifford, 1983) Lack of widely available, user-friendly software Not included in standard statistical packages like SPSS and SAS

5 ITEM POOL OPTIMISATION Item analysis results can be used to: Evaluate performance Improve item quality Test developers can use the result to discriminate whether: an item can be reused as is should be revised before reuse should be taken out of the active item pool

6 IRT ANALYSIS TOOL ICL & LMS DOKEOS Open-source IRT Analysis Tool ICL Hanson (2002) provided a stand-alone software for estimating the parameters of IRT models called IRT Command Language (ICL) ICL is actually IRT estimation functions embedded into a fullyfeatured programming language called TCL ( tickle ) Open-source LMS Dokeos It is implemented in PHP and requires Apache acting as a web server and mysql as a Database Management System

7 INTEGRATING IRT ANALYSIS IN LMS Defining the acceptable limits for IRT parameters: a) discrimination b) difficulty, and c) guessing 3 Normalized θ 2 If the estimated parameters of an item violate any of the validity rules, the item is flagged for review of its content 3 The assessment report displays the estimated proficiency θ per examinee

8 DEFINING VALIDITY THRESHOLDS

9 PERFORMING IRT ANALYSIS

10 IRT ANALYSIS RESULTS

11 SYSTEM ARCHITECTURE Web Server Examinee Examinee LMS Dokeos 1 IRT tool ICL Examinee Assessment Profile Assessment Results Parameter Estimation Script 2 Assessment Test Developer IRT Analysis Results 4 Estimated Parameters Theta Estimation Script 3 Estimated Theta Calibration Rules

12 STEP ONE The LMS exports the assessment results to a data file and generates a TCL script to process them one row per examinee output -no_print allocate_items_dist 40 read_examinees test0140dat 40i1 starting_values_dichotomous EM_steps -max_iter 200 print -item_param release_items_dist Assessment results (test0140dat file) Parameter Estimation Script (test0140tcl file)

13 STEP TWO The LMS calls up ICL in order to create a data file containing the α, b, and c values for each test item At the same time it prepares a second TCL script to process these IRT parameters (θ estimation script) 1 1, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , one row per item output -no_print allocate_items_dist 40 read_examinees test0140dat 40i1 read_item_param test0140par set estep [new_estep] estep_compute $estep 1 1 delete_estep $estep set eapfile [open test0140theta w] for {set i 1} {$i <= [num_examinees]} {incr i} { } close $eapfile release_items_dist Estimated parameters (test0140par file) θ estimation script (test0140ttcl file)

14 STEP THREE The LMS calls up ICL to make a data file with the examinees θ values 0, , , , , , , , , , , , , , , , one row per examinee Estimated theta (test0140theta file)

15 STEP FOUR The LMS imports the two ICL-produced data files (*par and *theta) to its database for further processing in the context of the aimed assessment test calibration

16 LMS DATABASE EXTENSIONS Initial LMS database quiz_version PK,I1 id PK,I2 version 1 U1 Version quiz_id lower_a lower_b upper_b upper_c Assessment (relation m-m) Item Option PK,FK1 quiz id title description sound type random active quiz_rel_question PK,FK1,I2,I1 question_id PK,FK2,I3 exercice_id PK quiz_question id question description ponderation position type picture quiz_answer PK,I1 id PK,FK1,I3,I2 question_id answer correct comment ponderation position Examinee PK user user_id Result Score details lastname firstname username password auth_source status official_code phone picture_uri creator_id competences diplomas openarea teach productions chatcall_user_id chatcall_date chatcall_text PK FK2,I2 FK1,I1 track_e_exercices exe_id exe_user_id exe_date exe_cours_id exe_exo_id exe_result exe_weighting PK 2 track_e_answers FK1,I1 id exe_id question_id answer_id answer correct weighting Entity- Relationship diagram of LMS database extensions

17 ADDITIONAL FUNCTIONALITIES Each assessment can have multiple versions based on its revised items By monitoring the examinees performance on each item, test developers can determine whether a certain modification of a specific item affected positively its quality Each examinee s score per item is recorded for every test being administered Test developers can establish a new set of rules for each version of the assessment

18 EXPERIMENTAL RESULTS Item Discrimination Parameter Values (α) α 05 (White area) Item Difficulty Parameter Values (b) -17 b 17 (White area) Item Guessing Parameter Values (c) c 025 (White area)

19 EXAMPLES OF DEFECTIVE TEST ITEMS Initial Item Low level of difficulty; the key answer deemed too plausible: Q: In the paged memory allocation scheme, the operating system retrieves data from secondary storage in same-size blocks called: A pages B frames C segments D partitions Revised Item Q: In which memory allocation scheme does the operating system retrieve data from secondary storagein several blocksof differentsizes? A segmented B paged C demand paging D partitioned Low degree of discrimination; the key answer confused examinees of both high and low abilities: Q: The transfer layer protocol of TCP/IP is called: A TCP B UDP C IP D A and B Q: The transfer layer protocol of TCP/IP is called: A TCP/UDP B FTP C IP D HTTP High guessing value probably due to the graduated answers: Q: How many are the basic control structures in programming? A one B two C three D four Q: The control structure used to choose among alternative courses of action is called: A sequence B repetition C selection D iteration

20 CONCLUSION The suggested approach is capable of assisting test developers in their continuous effort to improve flaws test items The user-friendly interface allows users with no previous expertise in statistics to comprehend and utilize the IRT analysis results The initial experiment produced encouraging results, showing that the system can effectively evaluate item performance The proposed methodology can be easily adopted by different e-learning environments and would be ideally suited for a MOOC environment

21 QUESTIONS? References Haladyna, TM (1999) Developing and Validating Multiple-Choice Test Items (2nd edition), Lawrence Erlbaum Associates, Mahwah, New Jersey Swaminathan, H & Gifford, JA (1983) 'Estimation of Parameters in the Three-parameter Latent Trait Model', in Weiss, DJ (ed), New Horizons in Testing, Academic Press, New York Hanson, BA (2002) IRT Command Language (ICL) Obtained through the Internet: [accessed 26/6/2013] Thank you very much! Questions?