The necessity and importance of cognitive diagnosis is being realized by more and

Transcription

1 WANG, YING-CHEN, Ph.D. Factor Analytc Models and Cogntve Dagnostc Models: How Comparable Are They? A Comparson of R-RUM and Compensatory MIRT Model wth Respect to Cogntve Feedback. (2009) Drected by Dr. Terry Ackerman. 99pp The necessty and mportance of cogntve dagnoss s beng realzed by more and more researchers. As a result, a number of models have been defned for cogntve dagnoss the IRT-based dscrete cogntve dagnoss models (ICDMs) and the tradtonal contnuous latent trat models. However, there s a lack of lterature that compares the newly defned ICDMs based on constraned latent class models to more tradtonal approaches such as a multdmensonal factor analytc model. The purpose of ths study s to compare the feedback provded to examnees usng a multdmensonal tem response model (MIRT) versus feedback provded usng an ICDM. Specfcally, a Monte Carlo study was used to compare the dagnostc results from the R-RUM, a noncompensatory model wth dchotomous abltes, to dagnoses made based on the 2PL CMIRT model, a compensatory model wth contnuous abltes. A fully crossed desgn was used to consder the effects of test qualty, Q-matrx structure and nter-attrbute correlaton on the agreement rates of the dagnostc feedback for examnees between these two models. Gven that one of the factors of ths study s test qualty, an ntal study was performed to explore the possble relatonshp between test qualty (ncludng estmated model parameters) based on the models used to characterze examnee responses. In addton, because these models provde examnee nformaton n dfferent ways (one dscrete and one contnuous), a method usng logstc regresson, whch s used to dscretze the contnuous estmates provded by the 2PL CMIRT, s dscussed as a way to maxmze

2 dagnostc agreement between these two models. The sgnfcance of ths study s that, f the two models agree consstently across the expermental condtons, model selecton for cogntve purposes can be based largely on the preference of the researcher, whch s nformed by an underlyng theory and assessment purposes. However, f the two models do not agree consstently, ths study wll help (1) to dentfy stuatons where the two models agree or dsagree consstently and (2) to explore the feasblty of usng the MIRT model for classfyng examnees cogntvely. The results from the frst study demonstrate that the two models defne test qualty n dfferent ways and that tem parameters of the two models are weakly assocated. Therefore, subsequent comparsons are made wthn each model after estmatng the R-RUM and the 2PL CMIRT, usng common datasets. The results from the fnal study ndcate that (1) the two models agree more consstently under the R-RUM generaton, (2) there s a hgher agreement rate between the two models under most scenaros of smple structure, (3) there s more error for both models under the MIRT generaton, and (4) the MIRT model does not appear to be as successful at classfcaton decsons as the R-RUM. Possble future drectons are dscussed.

3 FACTOR ANALYTIC MODELS AND COGNITIVE DIAGNOSTIC MODELS: HOW COMPARABLE ARE THEY? A COMPARISON OF R-RUM AND COMPENSATORY MIRT MODEL WITH RESPECT TO COGNITIVE FEEDBACK by Yng-chen Wang A Dssertaton Submtted to the Faculty of The Graduate School at The Unversty of North Carolna at Greensboro n Partal Fulfllment of the Requrements for the Degree Doctor of Phlosophy Greensboro 2009 Approved by Commttee Co-Char Commttee Co-Char

4 To my husband, my son and my ssters In memory of my parents

5 APPROVAL PAGE Ths dssertaton has been approved by the followng commttee of the Faculty of Graduate School at The Unversty of North Carolna at Greensboro. Commttee Co-Char Commttee Co-Char Commttee Members Aprl 27 th, 2009 Date of Acceptance by Commttee February 3 rd, 2009 Date of Fnal Oral Examnaton

6 ACKNOWLEDGEMENTS I cannot be where I am wthout professonal help from faculty of Educatonal Research Methodology (ERM) Department. It would be mpossble for me to enjoy measurement and learn so much f I had not transferred to ths program. I am very grateful to ther help. The ERM professors are: Dr. Terry Ackerman, Dr. Rchard Luecht, Dr. Robert Henson, Dr. Rck Morgan, Dr. John Wllse and Deborah Bartz. v

7 TABLE OF CONTENTS Page LIST OF TABLES.v LIST OF FIGURES...v CHAPTER I. INTRODUCTION...1 II. LITERATURE REVIEW IRT-based Cogntve Dagnostc Models Tradtonal Factor Analytc Models..19 Lterature on Compensaton and Noncompensaton.26 Comparson of the R-RUM and the 2PL CMIRT III. METHODOLOGY.33 Expermental Condtons Smulaton Study 1: A Comparson of Test Qualty and Item Parameters between the R-RUM and the CMIRT Smulaton Study 2: How Comparable Are the Two Models wth Respect to Cogntve Feedback? Estmaton Method IV. RESULTS Intal Descrptve Statstcs Symmetry of the Two Models How Comparable Are the Two Models wth Cogntve Feedback? V. CONCLUSIONS AND FUTURE DIRECTIONS Conclusons Future Drectons REFERENCES v

8 LIST OF TABLES Page Table 1. Test Qualty Table...37 Table 2. Expermental Condtons for Smulaton Study...39 Table 3. Descrptve Statstcs for the R-RUM Table 4. Descrptve Statstcs for the 2PL CMIRT Model Table 5. Descrptve Statstcs for Test Qualty Defnton for Hgh-qualty Test When r= Table 6. Descrptve Statstcs for Test Qualty Defnton for Hgh-qualty Test When r= Table 7. Descrptve Statstcs for Test Qualty Defnton for Hgh-qualty Test When r= Table 8. Descrptve Statstcs for Test Qualty Defnton for Medum-qualty Test When r=.2 Table 9. Descrptve Statstcs for Test Qualty Defnton for Medum-qualty Test When r=.5 Table 10. Descrptve Statstcs for Test Qualty Defnton for Medum-qualty Test When r= Table 11. Descrptve Statstcs for Test Qualty Defnton for Low-qualty Test When r= Table 12. Descrptve Statstcs for Test Qualty Defnton for Low-qualty Test When r= Table 13. Descrptve Statstcs for Test Qualty Defnton for Low-qualty Test When r= Table 14. Descrptve Statstcs for the Relaton between Item Parameters of the Two Models n Case of Hgh-qualty Test, Complex Structure Table 15. Descrptve Statstcs for the Relaton between Item Parameters of the Two Models n Case of Medum-qualty Test, Complex Structure 72 v

9 Table 16. Descrptve Statstcs for the Relaton between Item Parameters of the Two Models n Case of Low-qualty Test, Complex Structure Table 17. Descrptve Statstcs for the Relaton between Item Parameters of the Two Models n Case of Hgh-qualty Test, Smple Structure 73 Table 18. Descrptve Statstcs for the Relaton between Item Parameters of the Two Models n Case of Medum-qualty Test, Smple Structure Table 19. Descrptve Statstcs for the Relaton between Item Parameters of the Two Models n Case of Low-qualty Test, Smple Structure Table 20. Descrptve Statstcs for Recoverablty of Item Parameters of the Two Models n Case of Hgh-qualty Test, Complex Structure Table 21. Descrptve Statstcs for Recoverablty of Item Parameters of the Two Models n Case of Medum-qualty Test, Complex Structure Table 22. Descrptve Statstcs for Recoverablty of Item Parameters of the Two Models n Case of Low-qualty Test, Complex Structure Table 23. Descrptve Statstcs for Recoverablty of Item Parameters of the Two Models n Case of Hgh-qualty Test, Smple Structure Table 24. Descrptve Statstcs for Recoverablty of Item Parameters of the Two Models n Case of Medum-qualty Test, Smple Structure Table 25. Descrptve Statstcs for Recoverablty of Item Parameters of the Two Models n Case of Low-qualty Test, Smple Structure Table 26. Percentage of Raw Agreement between the Two Models Table 27. Kappa between the Two Models Table 28. Percentage of Agreement wth the True Attrbute Patterns Table 29. Kappa-based Agreement wth the True Attrbute Patterns v

10 LIST OF FIGURES Page Fgure 1. Smple Structure. 20 Fgure 2. Factorally Complex Structure Fgure 3. Flow Chart for Smulaton Study Fgure 4. Flow Chart for Smulaton Study v

11 CHAPTER I INTRODUCTION Tradtonally, testng ndustres have focused on constructng measures to assess a sngle dmenson. The test s assumed to measure only one latent or unobserved ablty or skll va the measured varables or tems. Each examnee s rank ordered based on the total tem scores or a sngle contnuous latent ablty and therefore only a sngle score s reported. Such reports have been wdely used for hgh-stake decsons such as college admssons, scholarshp awards and even graduaton. As a result, researchers and practtoners have appled varous statstcal tools to verfy that only one latent ablty s present n the data structure. Despte ts parsmonous nature, tradtonal scalng of examnees has some lmtatons. Most psychologcal and educatonal tests measure multple sklls and the undmensonalty assumpton cannot be met under these crcumstances (Hambleton & Swamnathan, 1985). In addton, t falls short of cogntve psychology n the twenteth century. Cogntve psychometrcs nvolves measurement models assessng hgh-order thnkng, whch s related to a set of sklls. It s commonly agreed that research n hgh-order thnkng s fundamental to the testng ndustry, as many tests are based on cogntve problem-solvng sklls (Gerl, Leghton, & Hunka, 2000). As a summatve assessment model, tradtonal modelng, such as undmensonal tem response theory (IRT) models, mght be approprate. However, tradtonal assessment s lmted n ts ablty to provde any formatve feedback for mprovng nstructon, 1

12 learnng and currculum development. Prncpals, teachers and educators need more nformatve reports for classroom nstructons and nterventon programs. Ths urgent publc demand s culmnated n the No Chld Left Behnd Act (2001), whch explctly calls for nterpretve, descrptve and dagnostc reports and the use of assessment results for mprovng students academc achevements. Whereas both forms of assessment are necessary, one durng the learnng and teachng process and the other at the end of the nstructon, formatve assessments are more useful dagnostcally at the classroom level throughout the course of nstructon. In the smplest case, formatve assessments should determne mastery or non-mastery for a set of K sklls. Recently, a varety of probablstc latent class models have been developed for cogntve dagnostc purposes. These models assume that classes are defned by a set of dscrete latent abltes, ether bnary or multcategorcal. Each of these IRT-based cogntve dagnostc models (ICDMs) has an tem response functon (IRF) that predcts the probablty of the correct response for each tem, gven the attrbute status of each examnee on each skll. As n IRT, the use of an IRF enables researchers to evaluate the qualty of test tems through the evaluaton of the tem parameters. Once an approprate model s selected, each examnee s profle s produced. As an alternatve for cogntve dagnoss, some researchers have ponted out that other IRT-based contnuous latent models parallel the above dscrete ICDMs. Contrary to the dscrete ICDMs, these models place each of the underlyng ablty dstrbutons on a contnuum. DBello, Roussos and Stout (2007) and Stout (2007) dscussed these contnuous models as possble psychometrc models for cogntve dagnoss. Among these models, the applcaton of multdmensonal tem response 2

13 theory (MIRT) models s common n research. For nstance, Appled Psychologcal Measurement devoted the wnter ssue of 1996 to research n MIRT models. Instead of provdng an estmate of a profle defnng whch attrbutes (or sklls) have been mastered (.e., a mastery profle), MIRT models produce factor scores. Therefore, f one were nterested n determnng whch sklls should be mproved, further research must be performed to choose some factor score for each skll, at and above whch the examnees are classfed as masters and below whch the examnees are classfed as nonmasters. Consequently, f research or assessment s based on the factor scores from MIRT models, t s mportant to research how these conclusons about cogntve status of examnees compare to those from the ICDMs. Both types of models, MIRT models or ICDMS, can be classfed accordng to skll nteractons nto compensatory models and noncompensatory or conjunctve models. Compensaton means that hgher values on one skll can offset the lower values on other sklls when calculatng the probablty of the correct response to an tem. The extreme case of a compensatory model s the dsjunctve model, whch means a certan mnmum on ONLY one of the relevant attrbutes s necessary to compensate for the lack of ablty on all other sklls for the correct response of the tem. Noncompensaton or conjuncton means certan mnmums on all sklls are necessary for a hgh chance of a correct answer of the tem. Anyone not havng a mnmum ablty for at least one attrbute wll lack the ablty to answer the tem correctly. Havng a hgher ablty n one attrbute s NOT suffcent to compensate for the lower ablty n other attrbute(s) and to answer the tem correctly (see Chapter II for more detals). 3

14 The vast arrays of the psychometrc models for cogntve dagnoss and ther dfferent ways to express cogntve complexty (e.g, underlyng latent dstrbutons, sklls nteracton, etc) make model selecton dffcult for accurate formatve assessments. If the selecton s to be made among models dfferng only n scale assumptons, ths mght only pose the challenge of selectng some set of some factor scores from MIRT models to evaluate the examnees cogntvely. If the selecton s made among models dfferng only n skll nteractons, ths mght only pose the challenge of determnng the type of skll nteractons to provde cogntve feedback. If the selecton s to be made among models dfferng n both scale assumptons and skll nteracton (compensatory or noncompensatory), ths would pose the challenge of determnng the type of skll nteractons for cogntve evaluaton of examnees n addton to the challenge of determnng a reasonable set of cut ponts. In the latter case, t s expected that the cogntve evaluaton of examnees wll be dfferent wth a noncompensatory ICDM versus a compensatory MIRT or a compensatory ICDM versus a noncompensatory MIRT. It s always dffcult to select a reasonable psychometrc model because of the challenge of dentfyng how the sklls nteract wth each other across tems, ndvduals, groups and forms. In addton, t s not always clear whether the true underlyng dstrbutons of abltes are dscrete or contnuous. However, f n applcaton, fnal decsons based on cogntve feedback are smlar even when usng dfferent models, then model selecton may be based on an underlyng theory wthout a focus on how these decsons wll mpact ultmate decsons for examnees. Due to the recency of the cogntve dagnoss, there has been lmted research concernng the 4

15 comparson of the ICDMs and MIRT models for cogntve dagnostc purpose. Therefore, t s the research goal of ths study to compare the two types of models and nvestgate f model selecton can nfluence fnal decsons that may be made for an examnee. For the purpose of the current study, two models wth dfferent scale assumptons and dfferent skll nteractons one compensatory MIRT model and one noncompensatory ICDM model were chosen (see Chapter II). The purpose of the current study s to determne how comparable the two models are wth respect to the cogntve evaluaton of the examnees. The two models have dfferent assumptons about attrbute scale and skll nteractons. Therefore, t s necessary to dentfy what technque s most approprate to compare the two dfferent models. In chapter III, a technque s descrbed such that the two models yeld the most consstent evaluaton of the examnees. Next, based on ths technque, the models are compared wth respect to how much the two models agree for cogntve dagnostc purposes. To address these goals, a smulaton study was performed. Three factors test qualty, the Q-matrx (Tatsuoka, 1983) structure and the correlaton between the attrbutes were chosen n the smulaton study. However, as the ICDMs are recently developed, ts relatonshp wth MIRT models s stll unclear. Therefore, a prelmnary smulaton study must be performed to nvestgate the relatonshp between the two models. The relatonshp between the two models means (1) f they defne test qualty n the same way and (2) what the relatonshp between the tem parameters of the two models s. It s possble that the two models dffer n ther defntons of test qualty, but the tem parameters of the two models mght be assocated wth each other. 5

16 Chapter III descrbes n detal the questons and methodologes about the ntal smulaton study used to establsh a defnton of test qualty of the ICDMs and MIRT models so that these two methodologes can be farly compared on the fnal research goals. Two flowcharts (Fgure 3 and Fgure 4) are provded to llustrate the smulaton procedures. Chapter IV dscusses the ntal study and chapter V addresses the fnal research goals. The answers to the ntal study wll facltate the understandng of the relatonshp between the ICDMs and MIRT models, whch wll be used to ensure a far comparson between the models based on test qualty. The answers to the fnal research goal wll provde nformaton about the mportance of model selecton for cogntve feedback. As the demand and the need for cogntve assessment are ncreasng rapdly, model selecton s becomng more and more crucal for formatve assessment to be popular (DBello & Stout, 2007; Bolt, 2007). If model selecton does not mpact the outcome related to examnees cogntve status, t s possble for popular models to be used wthout affectng the results. If model selecton does mpact the outcome, the study s helpful to dentfy stuatons where the two models agree or dsagree consstently. The results from the fnal research goal wll also provde nsght nto the feasblty of usng MIRT models for cogntve classfcaton of examnees. Chapter II provdes a dscusson of the ICDMs and tradtonal analytc models ncludng the MIRT models. The revew on dfferent skll nteracton s dscussed and the comparson of the two selected models s provded. Chapter III dscusses the questons, methodologes and statstcs of each smulaton study. Chapter IV deals 6

17 wth the prelmnary study and the fnal research goal of the study. Chapter V ends the study wth conclusons and future drectons. 7

18 CHAPTER II LITERATURE REVIEW Cogntve dagnoss, skll assessment or skll proflng refers to the parttonng the latent multdmensonalty nto dscrete latent attrbutes and evaluatng the examnees wth respect to ther status of mastery of each attrbute (Hartz, Roussos & Stout, 2002). In the lterature on cognton, attrbute s used nterchangeably wth dmenson, factor, skll, subskll and latent ablty. In ths study, the ICDMs refer only to the stochastc models recently developed. All of these models assume that attrbutes are dscrete and are dscussed n detal n Secton 2.1. The tradtonal contnuous latent varable models, referred as tradtonal factor analytc models, are presented n Secton 2.2. In both sectons, conjunctve models and compensatory models are dscussed. Secton 2.3 ncludes the defntons and lterature revew of compensaton and noncompensaton. The last secton presents the comparson of the selected models. 2.1 IRT-based Cogntve Dagnostc Models IRT-based cogntve dagnostc models (ICDMs) recently developed all defne the probablty of correctly answerng an tem as a functon of a set of dscrete attrbutes measured by the tem. In addton, the models requre that a Q-matrx has been defned wth elements q k, where 1 ndcates that the k th attrbute s requred by the th tem and 0 otherwse. In most cases, the Q-matrx s assumed as fxed and s determned by content experts. In addton, most ICDMs assume that only mastery of 8

19 those attrbutes specfed by the Q-matrx s necessary for the correct responses. These ICDMs can be classfed accordng to skll nteracton nto noncompensatory or conjunctve and compensatory models. The conjunctve models are presented frst and the compensatory models are presented next. Conjunctve Models Reparameterzed Unfed Model (RUM, Hartz et al, 2002, also referred to as the Fuson model) was defned based on the Unfed Model (DBello, Stout & Roussos, 1995). The Unfed Model s among the frst cogntve models to acknowledge that the Q-matrx s an ncomplete representaton of all the cogntve requrements for the test, thus dfferentatng the Unfed Model from most early cogntve dagnoss models. Specfcally, the Unfed model ncludes P θ ), where θ j s a sngle Cl ( j contnuous ablty parameter as a undmensonal projecton of examnee j s relevant attrbutes outsde those defned n the Q matrx (usng a Rasch model wth dfferent parameters c ). The problem wth the Unfed Model s that t s not estmable because there are 2k +3 parameters (k = the number of attrbutes requred by the tem) for each tem and thus, the parameters are not dentfable. Hartz (2002) developed the RUM (Fuson Model) out of the Unfed Model. She reparameterzed the Unfed model so that t was estmable and she retaned the nterpretablty of the parameters. The reparameterzed model has 2+K parameters per tem, where K represents the total number of requred attrbutes for an tem. The R-RUM defnes the probablty of a correct response P = 1/ α, θ ) as: ( X j j j 9

20 P ( X j K (1 α k ) q jk k = 1 / α, θ ) = [ r ] P ( θ ) (2.1) j j k= 1 c j where K = k= 1 q k k =P(correctly applyng all tem requred attrbutes gven q k =1 for all tem requred attrbutes), whch s the probablty of gvng a correct answer to all the attrbutes gven that an examnee j s a master of all the trats (k=1, K) related to tem. r k = P( Y P( Y jk jk = 1/ α = 1/ α jk jk = 0) = 1) whch s nterpretable as tem dscrmnaton parameter for attrbute k or the penalty for not masterng attrbute k c = the amount that correct tem performance requres θ j, n addton to the requred Q attrbutes; referred to as the completeness ndex for tem. The ranges of the parameters are 0 1, 0 rk 1, 0< c <3. For the dscrmnaton parameter, r k s 1 when the tem does not requre the k th attrbute and 0 when the dscrmnaton s maxmum. The addtonal ablty, θ j, s assumed to be contnuous, rangng from - to +. As the value of θ j approaches nfnty, P Cl ( θ j ) approaches to 1 for all values of c. When the value of c s approxmately 0, the dfferent values of P θ ) wll nfluence the tem response functon. The Cl ( j estmaton of the RUM was solved usng a Markov Chan Monte Carlo (MCMC) algorthm and a stepwse parameter selecton procedure. 10

21 The RUM s among the most common ICDMs studed (e.g, Jang, 2005). Hartz (2002) appled the model to PSAT/NMQT for the purpose of mprovng students performance on SAT. Jang (2005) also appled the RUM comprehensvely to ETS-TOEFL standardzed testng. Jang constructed the Q-matrx by combnng the characterstcs of the tems wth the results from DIMTEST and DETECT. The nsgnfcant tem parameters were elmnated and the program for the RUM was rerun on the data, usng the modfed Q-matrx. The follow-up study, surveys and ntervews, was conducted on a sample of 28 students and two teachers, to cross-valdate the dagnostc reports. Roussos, Hartz and Stout (2003) appled the RUM to the math secton of Amercan College Testng s assessment. The Reduced RUM (R-RUM, Hartz et al, 2002, Henson & Douglas, 2005; Fu, 2005) The R-RUM s a smplfed verson of the RUM wth the addtonal ablty, θ, removed. Wth the non-q attrbutes ( P θ ) ) removed, t s mplctly j Cl ( j acknowledged that the Q-matrx s a complete representaton of the sklls requred for the test or the non-q attrbutes are nsgnfcant. The nterpretatons of the remanng parameters are the same as n the RUM and thus the probablty of a correct response s defned as: P ( X j 1 / α ) r K (1 α k jk q = = k (2.2) j k= 1 ) Henson & Douglas (2005) appled ths model n the study on the ICDM test dscrmnaton ndces. The NIDA Model (nosy nputs, determnstc and gate, Junker and Sjstma, 2001; Mars, 1999) In the NIDA model, the probablty of a correct response s: 11

22 P ( X j j = 1/ α, s, g) = K α jk 1 α jk qk [(1 s k ) g k ] (2.3) k= 1 Where s P( η = 0 / α = 1, q = 1), a slppng parameter k = jk jk k g k P( η = 1/ α = 0, q = 1), a guessng parameter = jk jk k η jk, a latent varable defned at attrbute level, wth 1 ndcatng the examnee j has correctly appled attrbute k on tem and 0 otherwse. The NIDA model predcts the probablty of gvng a correct response as the product of slppng and guessng parameters. In the model, s k s an error probablty that an examnee ncorrectly apples attrbute k when n fact, he or she s a master of that attrbute and g k s the probablty that an examnee correctly apples attrbute k when he or she s a non-master of that attrbute. Because the slppng and guessng parameters are defned at the attrbute level, only the Q-matrx dstngushes dfference among tems and no tem specfc parameters are defned. Mars (1999) gves another verson of the NIDA model wth the parameters estmated for each tem and so the probablty of a correct response s defned as: P = 1/ α, s, g) = ( X j j K α jk 1 α jk qk [(1 s k ) g k ] (2.4) k= 1 However, lke the Unfed Model, ths model s not dentfed. de la Torre and Douglas (2004) appled the NIDA model for assessng the sklls used n mxed number subtracton. Based on the content and the problem-solvng characterstcs of the 20-tem test, they dentfed an eght-skll Q matrx for fracton subtracton. 12

23 The DINA Model (determnstc nputs, nosy and gate, Junker & Sjstma, 2001; Macready & Dayton, 1977; Haertel, 1989). The DINA model defnes the probablty of a correct response as a functon of two probabltes based on whether the examnee has mastered the requred attrbutes for the th tem. Specfcally, (1 ξ ) j j P( X = 1/ ξ, s, g ) = (1 s ) g (2.5) j j j j ξ Where K ξ j α q = k, whch s an ndcator of whether examnee j has mastered all the k=1 jk requred attrbutes for tem, wth 1 ndcatng the mastery of all of the tem s requred attrbutes and 0 nonmastery of at least one attrbute; s P( = 0 / ξ = 1), a slppng parameter; defnng the probablty that = X j j the examnee j, a master of all trats, ncorrectly responds to the tem. g P( = 1/ ξ = 0), a guessng parameter, meanng that a nonmaster of = X j j at least one attrbute, guesses and correctly responds to the tem. The DINA model constrans 1 s ) to be greater than g. The model smplfes ( examnees nto two groups masters and non-masters. In the non-master group, the examnees mssng one attrbute are equvalent to those mssng all the attrbutes. Zhang (2006) appled the DINA model for dfferental tem functonng (DIF) study. In the study, Zhang manpulated the tem parameters for the dfferent groups and completed a DIF analyss on smulated data and usng real data. In addton to the NIDA model, de la Torre and Douglas (2004) also appled the DINA model for the cogntve dagnoss of the sklls used n mxed number subtracton. Recently, based on 13

24 real data, de la Torre and Lee (2007) used the DINA model to explore the relatonshp between the ICDMs, classcal testng theory and IRT ndces. Compensatory Models In the followng secton, examples of compensatory models are ntroduced. They nclude the compensatory RUM (Hartz, 2002), NIDO (Templn, Henson, Douglas, 2006) and a dsjunctve model DINO model (Templn & Henson, 2006). As defned n the prevous chapter, a dsjunctve model s an extreme case of the compensatory model n the sense that the competency on ONLY one skll s enough for the correct answer of the tem. Last are the LCDM (Henson, Templn, & Wllse, 2008) and the GDM (von Daver, 2005), the two general versons of compensatory and noncompensatory model as was shown by Henson, Templn and Wllse (2008) through ther ntroducton of the log-lnear cogntve dagnostc model (LCDM). Compensatory RUM (Hartz, 2002). The compensatory RUM s a compensatory verson of the R-RUM, where the probablty of a correct response s defned as: P( X K k= + exp[ β + γ k qk a jk ] 1 1/ β, a, q, γ ) = (2.6) 1+ exp[ β γ q a ] = K k= 1 k k jk whereβ = the ntercept parameter nterpreted as the baselne log-odds of gettng the tem correct for examnees not masterng the skll. γ k =the ncreased log-odds of gettng the tem correct for each mastered Q-matrx ndcated skll Therefore, for those who are nonmasters of all the Q-matrx specfed attrbutes, the probablty of the correct response s a functon of the ntercept parameter. Ths 14

25 model was later defned as a specal case of the generalzed dagnostc model (GDM, to be dscussed, von Daver, 2005) and was appled to TOEFL test (von Daver, 2005). The NIDO Model (nose nput determnstc or gate, Templn, Henson &Douglas, 2006) Based on NIDA model, Templn, Henson and Douglas (2006) developed a compensatory model so that the probablty of a correct response: P( X j exp[ ( β k + γ kα jk ) qk ] α (2.7) 1+ exp[ ( β + γ α ) q ] k= 1 = 1/ j, qk ) = K K k= 1 k k jk k whereβ = the threshold of gettng the skll correct for examnees not masterng the k skll; γ k = the skll level dscrmnaton parameter Notce that the NIDO model defnes the probablty of a correct response usng only two parameters per skll. Lke the NIDA model, ths model does not have parameters at the tem level and so the tem parameters wll have dentcal values wthn the same skll. As a result, the probablty of gettng the tem correct wll be dentcal for tems wth an dentcal Q-matrx entry. The DINO Model (determnstc nput nose or gate, Templn & Henson, 2006) Based on the DINA model, Templn and Henson (2006) developed a dsjunctve model. Smlar to the notatonξ j n the DINA model, the notaton ω j s used to dvde examnees nto two groups: those who have mastered at least one attrbute of the Q-matrx ( ω =1) and those who have not mastered any Q-matrx specfed entres j ( ω j =0) for the th tem. Specfcally: 15

26 K qk ω = 1 (1 α ) (2.8) j k= 1 Incorporatng ths notaton nto the DINA model, the conjunctve model now becomes a dsjunctve model, predctng the probablty of a correct response as a functon of the slp and guessng parameters: jk P( X j (1 ωj ) j = 1/ ω ) = (1 s ) g (2.9) j ω where ( 1 s ) > g. Templn and Henson (2006) appled the DINO model to evaluate and dagnose the pathologcal gamblers. The Log-lnear Cogntve Dagnostc Model (LCDM, Henson, Templn & Wllse, 2008) The LCDM s a flexble log-lnear model that can ft many of the noncompensatory or compensatory models dscussed above. Frst, gve a general model when the number of attrbutes s 2 (K=2). The LCDM predcts the probablty of correct response as: P( X j exp( γ 1α 1+ γ 2α 2 + γ 12αα 1 2 β ) = 1/ α) = (2.10) 1+ exp( γ α + γ α + γ αα β ) where γ 12 represents skll nteractons wth a value greater than 0 ndcatng the noncompensaton and 0 or less ndcatng compensaton. γ k s the dscrmnaton parameter for each attrbute related to tem. β s the ntercept parameter nterpreted as the probablty of a correct response for those who are nonmasters of the requred sklls. Notce ths s a model for dchotomous data. Usng examples, Henson, Templn and Wllse (2008) demonstrated how the LCDM could ft compensatory RUM, DINA, 16

27 DINO and reduced RUM. Perhaps more mportantly, the LCDM provdes a parameterzaton for assessng the dfferences between each model and thus can be used to dentfy a reduced model such as the models prevously descrbed. The authors also performed MCMC estmatons on a real dataset. The results from the LCDM estmaton ndcated that some tems were consstent wth the DINA, one tem was consstent wth the DINO and some tems were consstent wth compensatory RUM. The Generalzed Dagnoss Model (GDM, von Daver, 2005) The GDM s a general and flexble verson of the ICDMs. The GDM can provde parameter estmates for multple tem types (dchotomous and ordered responses) wth multple latent ablty types (ether dchotomous or approxmately contnuous). Wth the GDM, the Q-matrx entres can be ether dchotomous or polytomous sklls. Wthn the class of the GDM, both compensatory and noncompensatory ICDMs may be specfed (Henson et al, 2008). The GDM predcts the probablty of correct responses by: P( X T exp[ β x + γ x. h( qk, a jk )] = x / β, a, q, γ ) = (2.11) m T 1 exp[ β γ h( q, a )] + = + y 1 y y. k jk where h( q, a) = ( h1 ( q, a),..., hk ( q, a)) s a vector of functons γ x = ( γ x1,..., γ xk ), (2 k -1) dmensonal slope parameters to determne the contrbuton of each non-zero Q-matrx entry. β x, the real-valued dffculty parameters When h ( q, a) =α q, the compensatory RUM s a specal case of the GDM. k jk Wth the excepton of the RUM, all the above ICDMs can be modeled wth the GDM 17

28 (Henson et al, 2008). However, the GDM can approxmate the RUM (Henson et al, 2008). Notce when k n equaton 2.11 s 1 and α j s defned as a contnuous latent varable wth normal dstrbuton, the GDM s an expresson for the two-parameter logstc IRT model. The GDM was appled to both the smulated data and the real data (von Daver, 2005). For the smulated data, the classfcaton accuracy across four sklls usng Cohen s kappa was above.85 across fve dfferent replcatons. The applcaton was done on TOEFL Internet-based testng plot data wth two forms (Form A and B) and two sectons (Readng and Lstenng). The Q-matrces were suppled by the experts. Seven out of eght sklls were strongly related to the overall ablty obtaned usng the tradtonal 2PL IRT model. The skll profle ndcated four hghly correlated skll classfcatons for the Lstenng secton and the three hghly correlated skll classfcatons for the Readng secton. The popular ICDMs n the lterature have been commonly conjunctve models, such as the RUM and DINA. These ICDMs are IRT-based n the sense that they share some smlartes wth the IRT models n ther assumptons. The ICDMs assume local ndependence condtonal on the latent ablty (.e., α ). Specfcally, they assume that after condtonng on an examnee s abltes, the responses of an examnee to dfferent tems wll not nfluence each other and that examnees from the same group (.e., the same α j ) should have the same expected response pattern. In the ICDMs, monotoncty means that the probablty of correctly respondng to an tem s non-decreasng n each coordnate of the attrbutes wth all other coordnates held j 18

29 fxed (Junker & Sjtsma, 2001). 2.2 Tradtonal Factor Analytc Models Lnear Factor Model Factor analyss started wth Charles Spearman (1904). He proposed the one-factor theory, whch assumed the test measured one general factor n common, g, general ntellgence. He suggested that all human ntellectual actvtes have ths general factor n common. In addton, the more two tests have n common wth the general factor, the hgher ther correlaton would be. He also proposed a second factor, the specfc factor. Ths factor was only specfc to a sngle actvty or varable and not correlated wth the general factor. Its presence could reduce the correlaton between the tests. Therefore, wthn a test, t s the general factor, a factor unversal to a person s ablty, that accounts for the correlaton among the tems. Some researchers dd not agree wth the one-factor model. Thurstone (1938) s one of the famous proponents of the multple factors. Analyzng the responses from 240 volunteer students on ffty-sx tests, he dentfed nne ndependent factors. Later, Thurstone (1941) completed a second study and found the same factors present. It was Thurstone who put forward the concept of smple structure, a very mportant concept n factor analyss. Smple structure descrbes a test where each tem loads on only one dmenson. Graphcally, smple structure can be represented as follows: 19

30 Fgure 1. Smple Structure X 1 X 2 F 1 X 3 X 4 X 5 F 2 X 6 As opposed to smple structure, a test s factorally complex when a measured varable s related to more than one factor or an tem s measured by more than one factor (refer to Fgure 2). 20

31 Fgure 2. Factorally Complex Structure X 1 X 2 F 1 X 3 X 4 X 5 F 2 X 6 Generally, for each person, the factor model may be expressed: x = µ +Λ f + ε (2.12) k k In ths model, x s a column vector of the measured varable, or responses to tems. The constant µ represents the th tem s dffculty. Λ s a ( k) matrx of factor loadngs, representng the amount of nformaton that each tem contans about each factor k related to tem. Factor loadng descrbes dscrmnatng power of the tem. For standardzed data, factor loadngs range from 0 to 1 wth 1 ndcatng maxmum dscrmnaton and 0 ndcatng no relaton wth the factor. f k s a column vector of latent varables and є s a column vector of unque factors. When K>1, t s a mult-factor model. When K=1, Λ s a column vector and the equaton (2.12) s the 21

32 expresson for classcal testng theory (CTT) (f corresponds to T, unobservable true score n CTT). Item Response Theory Models In the above lnear factor models (equaton 2.12), the observed varable s predcted based on a lnear combnaton of a set of latent varables. However, equaton 2.12 s not approprate for dchotomous tem responses. When equaton 2.12 s a one-factor model, the model has the followng lmtatons. Frst, the assumpton of lnearty between the tem and the latent factor cannot be met (McDonald, 1999). It s possble that equaton (2.12) yelds a probablty less than 0 f the factor score s too small, and a probablty greater than one f the factor score s large enough. Second, t assumes that error and factor are ndependent of each other and that the error varance s constant across all values of factors. When K n equaton 2.12 s greater than 1, the lnear factor model s a multple-factor model. When applyng the lnear multple-factor model to educatonal measurement, the same lmtatons assocated wth the lnear one-factor model stll exst except that each factor has ts constant error varance across the values of the latent ablty. In educatonal measurement, one method to overcome these lmtatons s by usng a nonlnear transformaton such as s commonly used the popular IRT models. IRT models have some favorable features such as the nvarance of both tem parameter estmates and ablty estmates and the ablty to predct the probablty of the correct response for an examnee to an tem gven the tem parameter(s). In addton, the standard error of measurement, that s the nverse of square root of nformaton, vares across ablty. The relatonshp between the probablty of a correct 22

33 response and the latent ablty s monotonc, that s, as ablty ncreases, the probablty of the correct response ncreases. In IRT models, the common models are ether logstc models or the normal orgve models (Lord, 1952) and they dffer approxmately by a constant, but the logstc IRT models are more popular due to ther smplcty n computaton. IRT models can be classfed nto three-parameter (3PL) model (Brnbaum, 1968), two-parameter (2PL) model (Brnbaum, 1968) and one-parameter (1PL) model (Rasch, 1961). Because the focus of the current study s about cogntve dagnoss, only the multdmensonal tem response theory (MIRT) models are dscussed. Mult-dmensonal IRT models The multdmensonal IRT (MIRT) models predct the probablty of the correct response for an tem as a functon of a set of tem parameters as well as a vector of the gven ablty levels. In MIRT, there are two classes of popular models the compensatory MIRT models (CMIRT, Reckase & McKnley, 1991) and the noncompensatory MIRT models (NCMIRT, Sympson, 1978). Noncompensatory Multdmensonal IRT (NCMIRT) Model (Sympson, 1978) Each dmenson n the NCMIRT has ts own dffculty parameter ( d k ) and ts own dscrmnaton parameter, a k, for the k th trat related to tem. Hgher values of the dffculty parameters ndcate more dffcult tems and lower values ndcate easy tems. The multplcatve nature of the noncompensatory models prohbts an examnee from compensatng for a low ablty on one dmenson by havng a hgh ablty on another or the other dmenson(s). The most complex model of ths famly of NCMIRT s the 3PL NCMIRT, where the probablty of a correct response s: 23

34 P ( x j K ( a kθ jk d k ) e = 1 / Θ j, a k, d k ) = c + (1 c ) ( a kθ jk d k = e k ) (2.13) The 2PL NCMIRT model s a smpler verson of ths 3PL model wth c constraned to zero for =1,,I. The 1PL NCMIRT model s the smplest verson of equaton (2.13) wth the dscrmnaton parameters constraned to unty and guessng fxed at zero. Compensatory Multdmensonal IRT (CMIRT) Model (Reckase & McKnley, 1991). Unlke the noncompensatory model, the CMIRT model has a vector of dscrmnaton parameters, one dffculty parameter and one guessng parameter per tem. The negatve values of the dffculty parameter ( d ) ndcate the more dffcult tems whle the postve values suggest the easer tems. Regardless of the number of dmensons, there s only one tem dffculty parameter and one tem guessng parameter. The 3PL CMIRT model, as s ndcated, ncludes the dscrmnaton parameter a for each skll k related to tem, a guessng parameter (c ), and a dffculty parameter ( d ) for all dmensons. Specfcally, the 3PL multdmensonal logstc model s: e P ( x = 1 / Θ, a, d ) = c + (1 c ) (2.14) j j k 1+ K a k k = 1 e θ jk + d K a k θ jk + d k = 1 The dscrmnaton parameters n (2.14) are constraned to be postve and the length of the tem vector s equal to the amount of multdmensonal dscrmnaton (Ackerman, 1994; Reckase & McKnley, 1991). Due to the addtve nature of the elements n the exponent, the examnees havng a low ablty on one dmenson can 24

35 beneft from havng a hgh ablty on another or other dmenson(s). As to the 2PL CMIRT (Reckase, 1985), the guessng parameter s set to zero. Thus the model becomes: K a k k = 1 e jk + d e P ( x = 1 / Θ ) = (2.15) j j 1+ θ K a k θ jk + d k = 1 Note that ths s equvalent to the nonlnear factor analyss wth a logt lnk as prevously descrbed (Chrstoffersson, 1975; McDonald, 1967). Wth the 1PL or the Rasch CMIRT model, the guessng parameters are set to zero and the dscrmnaton parameters are constraned to unty. These two types of the models can be rewrtten as a generalzed multdmensonal tem response theory (GMIRT) model (Ackerman & Bolt, 1995): P( x j / Θ j, a k, b k jk k= 1 e, µ ) = (2.16) [1+ e K f jk k= 1 K f K ] + µ [ e where f jk = a θ d ). In equaton (2.16), u s a weght wth 0 representng k ( jk k fully compensatory model and 1 fully noncompensatory model, but any value between 0 and 1 ndcates the varyng degree of compensaton requred by the attrbutes. Ths model may be vewed as a general expresson of the MIRT models and the undmensonal IRT models. In addton, a guessng parameter could be ncluded to defne a three-parameter model. In educatonal measurement, the nonlnear factor model and the MIRT models, are more popular. The 1996 wnter ssue of Appled Psychologcal Measurement was k= 1 f jk ] 25

36 devoted to research of MIRT models. As shown n the next secton, a large amount of research has been completed usng MIRT models n educatonal measurement. As members of the IRT famly, the relatonshp between MIRT models and the lnear factor analyss has been establshed (Chrstoffersson, 1975). Due to ts popularty, there may be some crcumstances where the MIRT model would be selected to provde dagnostc nformaton as to the ICDMs. Therefore, t s the goal of the current study to compare these two types of models to nvestgate how consstent the two models are wth respect to cogntve dagnostc and to dentfy the stuatons where they are comparable. 2.3 Lterature on Compensaton and Noncompensaton The concepts of compensaton and the noncompensaton or conjuncton was frst ntroduced by Coombs (1964), Coombs and Kao (1955) and Johnson (1935). Under conjunctve model, the jont abltes of all attrbutes are necessary for answerng the tem correctly. Anyone lackng the ablty n one attrbute wll lack suffcent knowledge to answer the tem correctly and so wll most lkely mss the tem. That s, havng a hgher ablty on one attrbute s NOT suffcent for compensatng for the lower ablty n other attrbute(s) and answerng the tem correctly. In contrast, compensatory models allow for a hgher ablty on one attrbute to compensate for the lower ablty on other attrbute(s), thus ncreasng the probablty of gettng the tem correct. Popular compensatory models nclude the lnear factor models and some MIRT models wth addtve propertes. Unlke equaton 2.13, whch s multplcatve across dmensons, equaton 2.14 to equaton 2.15 are addtve across 26

37 the dmensons. Although addtve models n the lterature assume a compensatory relatonshp between the latent abltes and the response holds, other models, such as a dsjunctve model, can also be consdered compensatory. Dsjunctve model requre that a mnmum competency on ONLY one attrbute s enough for the correct answer. Apart from dsjunctve model, dsjunctve processng may also be represented by the negatve nteracton term (Henson, Templn, & Wllse, 2008). The compensatory and noncompensatory models are dfferent from each other n the nature of cognton. The mpled cogntve assumpton of compensaton s that the complete mastery of the Q-matrx sklls s not necessary for the correct answer of the tem. Instead, an ablty at or above a mnmum level on any of the relevant sklls plays a domnant role n answerng the tem correctly (n the dsjunctve case, t s enough to have a mnmum on one skll for the correct response of the tem). The cogntve assumpton of noncompensaton s that all the sklls relevant to the tem are necessary for the correct response of the tem. Emprcal evdence supports both types of models. Some research found compensaton outperformed noncompensaton whle other research found compensaton and noncompensaton were comparable or noncompensaton was superor. For example, Smpson (2005) used the GMIRT model to nvestgate the relatonshp between noncompensatory processng and the task of matrx completon. She found u, an ndcator of the degree of compensaton, n the GMIRT model, was greater than 0, supportng the compensatory processng n the cogntve soluton of matrx completon. Mslevy et al. (2002) found that compared wth the conjunctve model, the compensatory model produced relatvely hgh 27

38 reducton n posteror varance, ndcatng the compensatory model s a better ft. Comparng the compensatory model wth the noncompensatory model, Van Leeuwe & Roskam (1991) found that a compensatory MIRT model provded better ft to LSAT data than a noncompensatory MIRT model. Hambleton and Slater (1997) compared a compensatory polcy wth a polcy combnng compensatory and conjunctve components wth respect to standard settng. Ther results demonstrated that the compensatory polcy ncreased the levels of decson consstency and the levels of decson accuracy whereas the polcy combnng both compensatory and conjunctve components lowered the levels of decson consstency and the levels of decson accuracy. Under the polcy wth the conjunctve components, the canddates faled at a very hgh rate. Consstent wth Hambleton and Slater s results, Haladyna and Hess (1999) found compensatory strateges outperformed conjunctve strateges decsvely n terms of relablty and rater consstency. Rchter and Späth (2006), n ther study of decson-makng, found that people ntegrated nformaton wth other types of task-relevant knowledge n judgment and decson makng, whch was an ndcaton of compensatory decson-makng. On the other hand, some research does fnd both models are comparable or support the noncompensatory model. Way, Ansley and Forsyth (1988) smulated data usng both compensatory and noncompensatory models. Ther ndependent varable was the correlaton between the dmensons and the dependent varable was the ablty estmates. Ther results showed that the observed score dstrbutons for each model were comparable and the θ estmates were most hghly related to the average of the 28

39 two θ parameters. In a study of the success of the graduate students (Nelson, Nelson & Malone, 2000), both the compensatory term and the conjunctve term were found to be sgnfcant predctors. Investgatng geometrc analogy soluton as a functon of systematc varatons n nformaton structure of the tem, Mulholland, Pellgegrno and Glaser (1980) found that the best-fttng functon was a nonaddtve model (a conjunctve model) nstead of a smple addtve model (a compensatory model). In the study of teacher lcensure, Mehrens and Phllps (1989) found that the conjunctve model was more approprate when the purpose was to set a cut-off value for the mnmal competence nstead of predctng the degree of success. To study Korean hgh school students decson-makng process, Hong & Chang (2004) conducted ther study usng thnk-aloud, tape-recordng and observatons and concluded that students preferred the non-compensatory rules nstead of the compensatory rules whch allowed the trade-off among alternatve strateges. Wth the complexty of cognton, t s mpossble for one model to be the best for all scenaros. Apart from cognton, many factors mght nfluence whch type of skll nteracton mght occur. These factors nclude assessment purposes, content areas, test desgns, attrbute structures, or dfferent target populatons. Skll nteractons mght vary across tems, sklls, test structures, ndvduals, groups and populatons. It s qute possble that some data mght be a mxture of compensaton and conjuncton. 2.4 Comparson of the R-RUM and the 2PL CMIRT A common sayng may depct the dlemma of psychometrcans very precsely: A person wth one watch knows what tme t s; a person wth two watches s never qute sure. The challenge becomes greater when there are many models avalable. 29

40 That s, models wll have to be selected based on a compromse of model ft, the purpose of the models and some addtonal factors such as the assessment purpose and the way of reportng the cogntve status. However, when the measurement from two dfferent models yelds a smlar nterpretaton, then one can make a selecton based on personal preference, software avalablty or/and the assessment purposes. Thus, the goal of the current study s to nvestgate the effect of two dfferent models on the fnal cogntve dagnoss of the examnees. To make such a comparson, two models were selected R-RUM and 2PL CMIRT model. When choosng the models, four factors were taken nto consderaton model popularty, the substantve tem parameter nterpretatons, skll nteractons and attrbute scales. Among the ICDMs, the conjunctve models are more commonly used such as the RUM, the R-RUM and the DINA (e.g. Hartz et al, 2002; Jang, 2005; Henson and Douglas, 2005). Among the tradtonal MIRT models, the CMIRT models are more often found to outperform the NCMIRT models (e.g., Bolt &.Lall, 2003; Mslevy et al, 2002). The R-RUM shares smlar tem parameter nterpretatons as the 2PL MIRT model. n the R-RUM, rangng from 0 to 1, can be nterpreted as the condtonal tem dffculty parameter based on Q-matrx. It s closer to d, tem dffculty parameter n the 2PL MIRT models. In the R-RUM, r k s nterpretable as tem dscrmnaton parameter for attrbute k, wth 0 ndcatng the maxmum dscrmnaton and 1 ndcatng no dscrmnaton. Ths s somewhat smlar to a k, dscrmnaton parameter n the 2PL MIRT models. The rest of the ICDMs do not share the smlar tem parameter nterpretatons wth MIRT models as the R-RUM. 30