1 DATA PREPARATION BILOG-MG File menu Edit menu Setup menu Data menu...

Transcription

1 Table of Contents 1 DATA PREPARATION BILOG-MG NEW FEATURES IN BILOG-MG PHASES OF THE ANALYSIS: INPUT, CALIBRATION AND SCORING THE BILOG-MG INTERFACE File menu Edit menu Setup menu Data menu Technical menu Save menu Run menu Output menu View menu Options menu Window menu Help menu Location of keywords in interface GETTING STARTED WITH BILOG-MG A first model: 2PL model for spelling data A second model: DIF model for spelling data SYNTAX Data structures: ITEMS, TEST, GROUP and FORM commands USING THE COMMAND LANGUAGE

2 2.6.1 Overview of syntax Order of commands CALIB command COMMENT command DRIFT command FORM command GLOBAL command GROUP command INPUT command ITEMS command LENGTH command PRIORS command QUAD command QUADS command SAVE command SCORE command TEST command TITLE command Variable format statement Input and output files PARSCALE THE PARSCALE INTERFACE Main menu Workspace Run menu Output menu

3 3.1.5 Font option Window menu COMMAND SYNTAX Order of commands BLOCK command CALIB command COMBINE command COMMENT command FILES command INPUT command MGROUP command MRATER command PRIORS command QUADP command QUADS command SAVE command SCORE command TEST/SCALE command TITLE command Variable format statements INPUT FILES Specification of input files Individual level data Group-level data Key files

4 3.4 OUTPUT FILES Format of output files Combined score file Fit statistics file Item parameter file Item information file Subject scores file MULTILOG THE MULTILOG USER S INTERFACE Main menu Run menu Output menu Window menu Font option CREATING SYNTAX USING THE MULTILOG SYNTAX WIZARD New Analysis dialog box Fixed Theta dialog box Input Data dialog box Input Parameters dialog box Test Model dialog box Response Codes (Binary Data) dialog box Response Codes (Non-Binary Data) dialog box GETTING STARTED WITH MULTILOG Two-parameter model for the skeletal maturity data Three-parameter (and guessing) model for the LSAT6 data

5 4.3.3 Generating syntax for a fixed-θ model COMMAND SYNTAX Overview of syntax END command EQUAL command ESTIMATE command FIX command LABELS command PROBLEM command PRIORS command SAVE command START command TEST command TGROUPS command TMATRIX command Variable format statement TESTFACT INTRODUCTION THE TESTFACT INTERFACE Main menu Run menu Output menu Window menu Font option COMMAND SYNTAX Order of commands

6 5.3.2 Overview of syntax BIFACTOR command CLASS command COMMENT command CONTINUE command CRITERION command EXTERNAL command FACTOR command FRACTILES command FULL command INPUT command KEY command NAMES command PLOT command PRIOR command PROBLEM command RELIABILITY command RESPONSE command SAVE command SCORE command SELECT command SIMULATE command STOP command SUBTEST command TECHNICAL command TETRACHORIC command

7 TITLE command Variable format statement IRT GRAPHICS INTRODUCTION MAIN MENU The ICC option The Information option The ICC and Info option The Total Info option Matrix Plot option The Histogram option The Bivariate Plot option The Exit option MANIPULATING AND MODIFYING GRAPHS File menu Edit menu Options menu Graphs menu Axis Labels dialog box Bar Graph Parameters dialog box Legend Parameters dialog box Line Parameters dialog box Plot Parameters dialog box Text Parameters dialog box ITEM CHARACTERISTIC CURVES ITEM INFORMATION CURVES

8 6.6 TEST INFORMATION CURVES OVERVIEW AND MODELS OVERVIEW OF IRT PROGRAMS BILOG-MG PARSCALE MULTILOG TESTFACT MODELS IN BILOG-MG Introduction Multiple-group analyses Technical details Statistical tests MODELS IN PARSCALE Introduction Samejima s graded response model Masters partial credit model Scoring function of generalized partial credit model Multiple-group polytomous item response models Constraints for group parameters Test of goodness-of-fit Initial parameter estimates MODELS IN MULTILOG Introduction The graded model The one- and two-parameter logistic models

9 7.4.4 The multiple response model The multiple-choice model The three-parameter logistic model The nominal model Contrasts Equality constraints and fixed parameters OPTIONS AND STATISTICS IN TESTFACT Introduction Classical item analysis and test scoring Classical descriptive statistics Item statistics Fractile tables Plots Correction for guessing Internal consistency Tetrachoric correlations and factor analysis IRT based item factor analysis Full information factor analysis Bifactor analysis Not-reached items in factor analysis Constraints on item parameter estimates Statistical test of the number of factors Factor scores Number of quadrature points Monte Carlo integration Applications

10 8 ESTIMATION INTRODUCTION Trait estimation with Item Response Theory Information ESTIMATION IN BILOG-MG Item calibration Test scoring Test and item information Effects of guessing Aggregate-level IRT models ESTIMATION IN PARSCALE Prior densities for item parameters Rescaling the parameters The information function Warm s weighted ML estimation of ability parameters ESTIMATION IN MULTILOG Item parameter estimation USES OF ITEM RESPONSE THEORY INTRODUCTION SELECTION TESTING QUALIFICATION TESTING PROGRAM EVALUATION AND ASSESSMENT TESTING CLINICAL TESTING MEASUREMENT METHODS AND RESEARCH

11 9.7 APPROACHES TO ANALYSIS OF ITEM RESPONSE DATA Test scoring Test generalizability Item analysis Estimating the population distribution Differential item functioning Forms equating Vertical equating Construct definition Analysis and scoring of rated responses Matrix sampling Estimating domain scores Adaptive testing BILOG-MG EXAMPLES CONVENTIONAL SINGLE-GROUP IRT ANALYSIS DIFFERENTIAL ITEM FUNCTIONING DIFFERENTIAL ITEM FUNCTIONING EQUIVALENT GROUPS EQUATING VERTICAL EQUATING MULTIPLE MATRIX SAMPLING DATA ANALYSIS OF VARIANT ITEMS GROUP-WISE ADAPTIVE TESTING TWO-STAGE SPELLING TEST ESTIMATING AND SCORING TESTS OF INCREASING LENGTH COMMANDS FOR PARALLEL-FORM CORRELATIONS

12 10.12 EAP SCORING OF THE NAEP FORMS AND STATE MAIN AND VARIANT TESTS DOMAIN SCORES PARSCALE EXAMPLES ITEM CALIBRATION AND EXAMINEE BAYES SCORING WITH THE RATING-SCALE GRADED MODEL EXAMINEE MAXIMUM LIKELIHOOD SCORING FROM EXISTING PARAMETERS CALIBRATION AND SCORING WITH THE GENERALIZED PARTIAL CREDIT RATING-SCALE MODEL: COLLAPSING OF CATEGORIES TWO-GROUP DIFFERENTIAL ITEM FUNCTIONING (DIF) ANALYSIS WITH THE PARTIAL CREDIT MODEL A TEST WITH 26 MULTIPLE-CHOICE ITEMS AND ONE 4-CATEGORY ITEM: THREE-PARAMETER LOGISTIC AND GENERALIZED PARTIAL CREDIT MODEL ANALYSIS OF THREE TESTS CONTAINING ITEMS WITH TWO AND THREE CATEGORIES: CALCULATION OF COMBINED SCORES RATER-EFFECT MODEL: MULTI-RECORD INPUT FORMAT WITH VARYING NUMBERS OF RATERS PER EXAMINEE RATER-EFFECT MODEL: ONE-RECORD INPUT FORMAT WITH SAME NUMBER OF RATERS PER EXAMINEE RATERS-EFFECT MODEL: ONE-RECORD INPUT FORMAT WITH VARYING NUMBERS OF RECORDS PER EXAMINEE MULTILOG EXAMPLES ONE-PARAMETER LOGISTIC MODEL FOR A FIVE-ITEM BINARY-SCORED TEST (LSAT6) TWO-PARAMETER MODEL FOR THE FIVE-ITEM TEST THREE-PARAMETER (AND GUESSING) MODEL FOR THE FIVE-ITEM TEST THREE-CATEGORY GRADED LOGISTIC MODEL FOR A TWO-ITEM QUESTIONNAIRE THREE-CATEGORY PARTIAL CREDIT MODEL FOR THE TWO-ITEM QUESTIONNAIRE FOUR-CATEGORY GRADED MODEL FOR A TWO-ITEM INTERVIEW SCALE

13 12.7 A GRADED MODEL ANALYSIS OF ITEM-WORDING EFFECT ON RESPONSES TO AN OPINION SURVEY GRADED-MODEL SCORES FOR INDIVIDUAL RESPONDENTS FIVE-CATEGORY RATINGS OF AUDIOGENIC SEIZURES IN MICE IN FOUR EXPERIMENTAL CONDITIONS A NOMINAL MODEL FOR RESPONSES TO MULTIPLE-CHOICE ALTERNATIVES A CONSTRAINED NONLINEAR MODEL FOR MULTIPLE-CHOICE ALTERNATIVES A NOMINAL MODEL FOR TESTLETS A CONSTRAINED NOMINAL MODEL FOR QUESTIONNAIRE ITEMS A CONSTRAINED GENERALIZED PARTIAL CREDIT MODEL A MIXED NOMINAL AND GRADED MODEL FOR SELF-REPORT INVENTORY ITEMS A MIXED THREE-PARAMETER LOGISTIC AND PARTIAL CREDIT MODEL FOR A 26-ITEM TEST EQUIVALENT GROUPS EQUATING OF TWO FORMS OF A FOUR-ITEM PERSONALITY INVENTORY DIFFERENTIAL ITEM FUNCTIONING (DIF) ANALYSIS OF EIGHT ITEMS FROM THE 100-ITEM SPELLING TEST INDIVIDUAL SCORES FOR A SKELETAL MATURITY SCALE BASED ON GRADED RATINGS OF OSSIFICATION SITES IN THE KNEE TESTFACT EXAMPLES CLASSICAL ITEM ANALYSIS AND SCORING ON A GEOGRAPHY TEST WITH AN EXTERNAL CRITERION TWO-FACTOR NON-ADAPTIVE FULL INFORMATION FACTOR ANALYSIS ON A FIVE-ITEM TEST (LSAT7) ONE-FACTOR NON-ADAPTIVE FULL INFORMATION ITEM FACTOR ANALYSIS OF THE FIVE- ITEM TEST

14 13.4 A THREE-FACTOR ADAPTIVE ITEM FACTOR ANALYSIS WITH BAYES (EAP) ESTIMATION OF FACTOR SCORES: 32ITEMS FROM AN ACTIVITY SURVEY Discussion of output ADAPTIVE ITEM FACTOR ANALYSIS AND BAYES MODAL (MAP) FACTOR SCORE ESTIMATION FOR THE ACTIVITY SURVEY SIX-FACTOR ANALYSIS OF THE ACTIVITY SURVEY BY MONTE CARLO FULL INFORMATION ANALYSIS ITEM BIFACTOR ANALYSIS OF A 12TH-GRADE SCIENCE ASSESSMENT TEST Discussion of bifactor analysis output CONVENTIONAL THREE-FACTOR ANALYSIS OF THE 12TH-GRADE SCIENCE ASSESSMENT TEST COMPUTING EXAMINEE GENERAL FACTOR SCORES FROM PARAMETERS OF A PREVIOUS BIFACTOR ANALYSIS ONE-FACTOR ANALYSIS OF THE 12TH-GRADE SCIENCE ASSESSMENT TEST ITEM FACTOR ANALYSIS OF A USER-SUPPLIED CORRELATION MATRIX SIMULATING EXAMINEE RESPONSES TO A THREE-FACTOR TEST WITH USER-SUPPLIED PARAMETERS SIMULATING EXAMINEE RESPONSES IN THE PRESENCE OF GUESSING AND NON-ZERO FACTOR MEANS THREE-FACTOR ANALYSIS WITH PROMAX ROTATION: 32ITEMS FROM THE SCIENCE ASSESSMENT TEST PRINCIPAL FACTOR SOLUTION OF A FACTOR ANALYSIS ON SIMULATED DATA: NO GUESSING NON-ADAPTIVE FACTOR ANALYSIS OF SIMULATED DATA: PRINCIPAL FACTOR SOLUTION, NO GUESSING ADAPTIVE ITEM FACTOR ANALYSIS OF 25 SPELLING ITEMS FROM THE 100-ITEM SPELLING TEST CLASSICAL ITEM FACTOR ANALYSIS OF SPELLING DATA FROM A TETRACHORIC CORRELATION MATRIX

15 14 APPENDIX A: A BRIEF HISTORY OF ITEM RESPONSE THEORY ANTECEDENTS CONNECTIONS IRT TEST SCORING IRT ITEM ANALYSIS CURRENT TRENDS REFERENCES