Estimation of composite score classification accuracy using compound probability distributions

Transcription

1 Psychologcal Test and Assessment Modelng, Volume 55, 2013 (2), Estmaton of composte score classfcaton accuracy usng compound probablty dstrbutons Chrs Wheadon 1 & Ian Stockford 2 Abstract Presented s a demonstraton of an ntutvely smple, flexble and computatonally nexpensve approach to estmatng classfcaton accuracy ndces for composte score scales formed from the aggregaton of performance on two or more assessments. Ths approach uses a two stage applcaton of the polytomous extenson of the Lord-Wngersky recursve algorthm and can be drven by any IRT model wth desred smplcty or requred complexty to best represent the propertes of the tests. The approach s demonstrated usng operatonal data from a hgh stakes mathematcs qualfcaton whch s formed from two tests admnstered on dstnct occasons. To provde the smplest representaton of a test contanng both dchotomous and polytomous tems, the partal credt model s appled to model behavour on the two tests. As an extenson to ths, a testlet model s appled to allow jont calbraton of parameters from both tests. Ths model provdes more nformaton to the calbraton process at the expense of some added computatonal complexty. Further to ths, the potental applcaton of ths approach n the absence of operatonal data s nvestgated usng a comparson of smulated data to the observed data. Key words: Classfcaton accuracy, IRT, composte scores 1 Correspondence concernng ths artcle should be addressed to: Chrs Wheadon, PhD, Centre for Educaton Research and Polcy, AQA, Stag Hll House, Guldford, Surrey, GU2 7XJ, UK; emal: cwheadon@aqa.org.uk 2 Centre for Educaton Research and Polcy, AQA, Guldford, Surrey, UK

2 Estmaton of composte score classfcaton accuracy Introducton The purpose of ths paper s to present a new method for calculatng the classfcaton accuracy of composte scores. Wherever scores are reported as classfcatons such as pass / fal or grade A to grade E users of those scores have an nterest n understandng how accurate those classfcaton decsons are. Classfcaton accuracy approaches provde an estmate of the accuracy of the gradng through a comparson of the degree to whch observed classfcatons agree wth those based on examnees true scores (Lee, Hanson, & Brennan, 2002; Lvngston & Lews, 1995). Composte score classfcaton accuracy refers to the accuracy of classfcaton when scores have been scaled or aggregated across multple assessments (Lvngston & Lews, 1995). There are a number of benefts to understandng the extent of msclassfcaton and the factors that nfluence t. These nclude beng aware of the potental consequences when desgnng assessments (or combnatons of assessments) used for a qualfcaton, as part of assessment qualty control montorng processes, and also n educatng users of qualfcaton results n areas such as the over-nterpretaton of grades. Many prevous studes have consdered classfcaton accuracy for sngle assessments. Wheadon and Stockford (2011) presented an emprcal evaluaton of the classfcaton accuracy and consstency of sngle assessments formng hgh stakes qualfcatons n England. Ths adopted both a Classcal Test Theory (CTT) and Item Response Theory (IRT) approach as prevously mplemented n other assessment contexts by Lvngston and Lews (1995) (CTT) and Lee (2008) (IRT). For applcaton of CTT approaches to classfcaton accuracy see also Breyer and Lews (1994), Hanson and Brennan (1990), Woodruff and Sawyer (1989), and Peng and Subkovak (1980). In greater depth, Verstralen and Verhelst (1991) have nvestgated the consequences of applyng dfferent IRT based measurement models for tem calbraton and accuracy calculaton n an tem bankng scheme, wth Lee, Hanson, and Brennan (2002), Wang, Kolen, and Harrs (2000), and Bramley and Dhawan (2010), consderng further IRT based approaches at the test level. Regardng artculatons of classfcaton accuracy at the composte score level, He (2009) consders the extensons avalable for more conventonal relablty ndcators; however, composte score classfcaton accuracy has only been consdered n a lmted number of studes. Van Rjn, Verstralen, and Bégun (2009), Douglas and Mslevy (2010) and Chester (2003) looked at the consequences of dfferent decson rules appled to classfy canddates based on composte scores ncludng consderaton of the valdty of the rules dependent on the content and ams of the assessment. Issues to be addressed when consderng composte score data sets are the herarchcal and multdmensonal nature of the tems across separate assessments. Multdmensonal IRT models (Reckase, 1997) offer a potental soluton to the management of multple assessment multple trat scenaros; however, the addtonal model complexty ntroduced renders operatonalzaton of these approaches challengng. Multdmensonal IRT models allow the effcency of assessment to be mproved as estmatons of performance on related constructs can draw strength from each other, as shown, for example, by Frey and Setz (2011), but where an assessment s requred, due to valdty constrants, to sample from a gven number of

3 164 C. Wheadon & I. Stockford dmensons there s less to be ganed from a complex modellng soluton appled posthoc. Ths study seeks to demonstrate a smple, robust and ntutve analytcal soluton to estmatng composte score classfcaton accuracy. Ths approach apples no constrants on the smplcty or complexty of the model used to represent the consttuent tests. The proposed approach s demonstrated on an operatonal data set and ts applcaton usng smulated data s dscussed to provde a prelmnary nsght nto the approprateness for use n nstances of (partally) absent data. 2 Method 2.1 The Data The operatonal data selected for consderaton s that arsng from an examnaton undertaken n England, specfcally a GCSE Mathematcs examnaton sat n summer Ths partcular GCSE Mathematcs qualfcaton s composed of two tests sat on dfferent occasons wthn a relatvely short perod of tme (eght days n ths nstance). Both tests have a maxmum mark of 100 and are composed of a mxture of dchotomous and polytomous tems, as outlned n table 2. Table 1: Frequency of tems wth the quoted number of response categores Number of Items wth Score Category K K = 2 K = 3 K = 4 K = 5 K = 6 Total Number of Items Maxmu m Mark Test Test When multple tests are aggregated to qualfcaton level, varous methods can be appled to scale the test scores. For smplcty the approach used n ths study was to sum canddates scores on each test and consder the accuracy of gradng aganst these qualfcaton level cut-scores. The approach can be easly extended, however, to complex non-lnear scalng and aggregaton approaches. The number of canddates wth vald marks entered for both tests was 17,957, however, for the purposes of practcalty when fttng the models descrbed below, a sub-set of 1,000 canddates was drawn at random from ths populaton.

4 Estmaton of composte score classfcaton accuracy Measurement models In order to estmate msclassfcaton rates for test data t s necessary to select a theoretcal model to represent canddate behavour. To provde a probablstc representaton of canddate performance at the tem level, IRT models have been appled here as descrbed n the followng sectons Partal Credt Model The partal credt model (PCM) (Masters, 1982) s an extenson of the dchotomous Rasch model (Rasch, 1960) and allows representaton of the probablty of a canddate achevng a certan score category on a gven tem. For canddate, wth ablty, respondng to tem j, whch has K j avalable score categores, ths model can be expressed as: exp Pr X x, x 0 j 1 exp k l 1 j k k K m Pr X 0 j 1 exp m1 l1 j K m 1 m1 l1 jl where X j s the canddate s acheved score on the tem, x k s the tem level score avalable n category k, and jk s the k th threshold locaton of tem j. To provde the tem parameters, jk, and the person parameters,, condtonal maxmum lkelhood (CML) estmaton can be used (Mar, Hatznger, & Maer, 2010) resultng n a sngle most lkely value of for each canddate and for each tem category. As ths model only specfes a sngle ablty parameter for each canddate (as opposed to multdmensonal IRT models where canddate ablty s represented by a vector of abltes) ths contans the mplct assumpton that the tems composng the test are measurng a sngle dmenson. Ths represents the smplest IRT model to descrbe polytomous tems and s used as the base model n ths study wth separate parameter estmaton beng performed for the two tests. To smulate data usng ths model a smplfed approach s taken. Rather than applyng the full PCM, all tems are modelled as beng dchotomous therefore mrrorng the Rasch model wth both tem and person parameters are drawn from a normal dstrbuton wth a mean of 0 and a standard devaton of The Testlet Model Hgh stakes qualfcatons are frequently composed of assessments n dfferent modes or assessng dverse sklls or content areas. Therefore, dfferent assessments are measurng (or attemptng to measure) a number of trats, hence, canddates true scores are also lkely to dffer between assessments (but are lkely to be postvely correlated for any qualfcaton that can make reasonable clams of valdty). jl jl (1)

5 166 C. Wheadon & I. Stockford A model whch accommodates these dfferences n lnked abltes s the testlet model. Ths model facltates the analyss of a populaton of tems whch can be grouped nto sub-populatons due to some common property. Each sub-populaton of tems whch share ths common property forms a testlet. Ths groupng of tems was ntally proposed n the context of computer adaptve testng by Waner and Kely (1987) to nvestgate whether the assumpton of local ndependence of tems was beng compromsed. Wthn the testlet model, canddates are estmated ablty parameters,, for the combned populaton of tems along wth a modfer of ths ablty for each testlet, termed a testlet propensty. For polytomous tems, ths model s provded by the re-expresson of the equaton provded by L, L, and Wang (2010) and untsaton of testlet and tem dscrmnaton parameters as: where k expl 1 j k K exp 1 j m m l1 Pr X x jl d j jl d j d j s canddate s testlet propensty for testlet d contanng tem j. To accommodate the ncreased model complexty compared wth the PCM, the tem, person and testlet parameters are estmated n a Bayesan framework va a Markov Chan Monte Carlo (MCMC) approach usng Gbbs samplng. In contrast to provdng a sngle set of most lkely tem and person parameters as s the case for CML, ths numercal approach s executed a number of tmes resultng n a populaton of possble tem, person and testlet parameters (Fox, 2010). Multple runs of the Bayesan parameter estmaton wll, therefore, provde an ndcaton of stablty related to model ft. (2) 2.3 Estmaton of classfcaton accuracy Rates of grade msclassfcaton are usually expressed as the nverse measure, classfcaton accuracy. The classfcaton accuracy for an ndvdual canddate s defned as the probablty that ther observed score falls n the same grade classfcaton as hs or her true ablty. In an IRT framework a canddate s test level true score (reflectng true ablty) s defned as the sum of hs or her expected tem level scores, such that on a gven test composed of J tems, canddate has a test true score,, defned as: K j J J Pr X x x E X (3) j k k j j1 k 1 j1 Two approaches can be used to estmate the canddate level classfcaton accuracy, as descrbed n the followng sub-sectons Numercal estmaton of classfcaton accuracy Snce IRT models provde a probablstc representaton of canddate behavour t s trval, although computatonally expensve, to estmate classfcaton accuracy statstcs

6 Estmaton of composte score classfcaton accuracy numercally. Ths can be acheved at the canddate level by performng a Monte Carlo smulaton to generate canddates observed scores (at the test level, composte level, or both) based upon the combnaton of tem and person parameters. The frequency of canddates observed scores occurrng n the same grade classfcaton as hs or her true score can then be summed and expressed as a proporton of the smulatons run to provde an estmate of the canddate level classfcaton accuracy. Whlst ths numercal approach s computatonally expensve t s congruent wth estmaton n a Bayesan framework such as that appled for parameter estmaton under the testlet model. Indeed, such an approach s proposed by Waner, Bradlow, and Wang (2007) for estmaton of classfcaton accuracy at the aggregated testlet level whch s equvalent to composte score when defnng testlets n the manner descrbed here. The numercal approach s dffcult to use, however, when the scores from testlets are scaled before they are aggregated Analytcal estmaton of classfcaton accuracy As an alternatve to the numercal approach, the probablstc models can be extended to determne analytcally the probablty that a canddate wth gven wll acheve each score on the test. Lord and Wngersky (1984) propose a recursve approach to calculatng the probablty that a canddate wll acheve each score. Ths can be extended to manage polytomous tems summarsed as: x KJ Pr Y Pr Y Y x Pr X x, J > 1 (4) J J 1 J J x0 Pr X Y, J = 1 (5) J J where Y J s the canddate s test level score on a test of length J, and the probablty of a J test score of zero s gven by PrYJ 0 Pr X j 0 method reles on knowledge of the probablty of achevng a test score of j1 therefore effcent for determnng the value of Pr. Snce ths recursve Y x t s Y for all scores. Collecton of these values for all possble values of Y provdes a condtonal sum score probablty dstrbuton. As proposed by Lee (2008), for a gven test, the probablty that a canddate s correctly classfed can then be determned by ntegratng ths condtonal sum score probablty dstrbuton between the cut-scores that surround the canddate s true score. The canddate level classfcaton accuracy, I CA, s therefore defned as: Y 1 U, Pr I Pr C Y C Y y (6) CA yyl, Ca s the grade classfcaton based on a test score of a, and Y U, and Y L, are where the cut-scores above and below the canddate s true score, respectvely.

7 168 C. Wheadon & I. Stockford Proposed here s the extenson of ths approach to combne test level condtonal sum score probablty dstrbutons, usng the Lord-Wngersky recursve algorthm, to provde a condtonal sum score probablty dstrbuton based on composte scores whch can be appled to equaton 6 usng the qualfcaton level cut-scores. Ths provdes an estmate of a canddate s qualfcaton level classfcaton accuracy. To acheve ths, equatons 4 and 5 can be re-expressed as: where Yˆ NT ZN 1 T ZN T ZN y T YN y T Pr Θ Pr Θ Pr Θ, N T > 1 (7) y0 Pr Y ' y Θ, N T = 1 (8) J Z N T s the canddate level composte score arsng from aggregaton of N T tests, Y ˆa s the maxmum scaled test score on test a, Θ s the vector of test level ablty pa- T rameters for canddate and PrZN 0 Pr 0 T Yn N Θ Θ. Usng ths analytcal approach t s relatvely easy, once probabltes have been derved at the test level, to ncorporate lnear and non-lnear scalng nto the process of estmatng classfcaton accuracy at the composte score level. n1 For clarty, the steps of the proposed procedure are: 1. Ft an approprate IRT model to the tests (be that separate estmaton at the test level usng the PCM, jont parameter estmaton usng the testlet model, or otherwse). 2. Apply the Lord-Wngersky recursve algorthm at the test level (equatons 4 and 5) for each canddate based on hs or her probablty dstrbutons of scorng each category on each tem. Ths results n a test level condtonal sum score probablty dstrbuton for each canddate. 3. If of nterest, apply equaton 6 to these condtonal sum score probablty dstrbutons, to determne I CA at the test level. 4. Scale the test level condtonal sum score probablty dstrbuton usng the requred transformaton (be that lnear or non-lnear). 5. Reapply the Lord-Wngersky recursve algorthm at the qualfcaton level (equatons 7 and 8) for each canddate based on ther scaled test level condtonal sum score probablty dstrbutons. Ths provdes a qualfcaton level condtonal sum score probablty dstrbuton for each canddate. 6. Apply equaton 6 to ths condtonal sum score probablty dstrbuton usng the qualfcaton level cut-scores to determne I CA at the qualfcaton level. It should be noted use of the Lord-Wngersky algorthm depends on the assumpton that the condtonal dstrbutons of the tem scores are ndependent of each other. Ths assumpton wll not hold for composte scores when models have been ftted separately to each test: the condtonal probablty of achevng a score on one test gven a score on the

8 Estmaton of composte score classfcaton accuracy other test would gve a more accurate estmaton of the lkelhood of classfcaton accuracy than the margnal probabltes used n the process descrbed here Models ftted Three models are ftted: 1. The partal credt model s ftted to each test separately, the Wngersky-Lord algorthm s appled to the model parameters derved from each test separately, and then combned once agan usng the Wngersky-Lord algorthm. Ths s the smplest applcaton of the procedure suggested here, but s subject to the lmtatons concernng condtonal ndependence as descrbed above resultng n degradaton of the classfcaton accuracy estmaton. 2. The partal credt model s ftted to the combned sets of tems from both tests and the Wngersky-Lord algorthm appled to the sngle set of model parameters. Whle ths approach would not be generally recommended t allows the degradaton of the estmaton from loss of condtonal ndependence n the separate estmaton procedure to be evaluated and s vald here due to the hghly correlated nature of the test scores composng the composte score. 3. The testlet model s ftted across both tests, wth each test representng one testlet. The person, tem and testlet parameters are then used to defne separate condtonal sum score dstrbutons for each test. The two sets of probabltes are then combned usng the Wngersky-Lord algorthm. Ths approach allows the estmaton of the model parameters to beneft from jont estmaton across both tests. Ths approach should yeld a more accurate estmaton than the separate estmaton of partal credt models and does not suffer the loss of condtonal ndependence to whch model 1 s subjected Classfcaton accuracy summary statstcs Regardless of both the approach used to estmate the classfcaton accuracy and level of the herarchy at whch t s expressed, I CA s avalable at the ndvdual canddate level. Collectvely, ths provdes a rch representaton of how classfcaton accuracy vares wth dfferent canddate propertes (usually plotted aganst canddate true score). However, for many applcatons such as routne qualty montorng, the defnton of a sngle summary statstc s potentally benefcal for manageablty and nterpretablty. The summary statstc appled here s that proposed by Lee (2008) whch takes the mean of the canddate level classfcaton accuraces. Ths statstc can be nterpreted as the probablty that a canddate selected at random from the cohort wll be accurately classfed. Whlst ths provdes an ntutve measure t should be borne n mnd f usng ths measure for qualty montorng purposes that ths measure s heavly dependent on the dstrbuton of canddates across the mark range. Ths measure reflects as much about the propertes of the cohort as t does about the underlyng assessment (Wheadon & Stockford, 2011).

9 170 C. Wheadon & I. Stockford 2.4 Software The analyses descrbed throughout ths work have been mplemented n R (R Development Core Team, 2011). The PCM model and accompanyng CML s mplemented usng the erm package (Mar, Hatznger, & Maer, 2010). The MCMC and Gbbs samplng procedure appled when estmatng parameters under the testlet model was performed usng JAGS (Plummer, 2012) accessed va the R2jags (Su & Yajma, 2011) R package when analysng the operatonal data. Due to ts mproved handlng of mssng data, ths estmaton used WnBUGS (Lunn, Thomas, Best, & Spegelhalter, 2000) for the producton and estmaton of smulated data due to ts robustness n the presence of mssng data at the expense of a degree of computatonal speed. The testlet model specfcaton was taken from Curts (2010). In order to support further research n ths area the authors have developed the R package classfy (Wheadon & Stockford, 2012). 3 Results 3.1 Descrptve statstcs Before consderng the classfcaton accuracy estmates t s mportant to examne the descrptve statstcs to provde some context for the later analyses. The test level descrptve statstcs for the operatonal data are presented n Table 2. The Cronbach s alpha and average tem to test correlaton are hgh suggestng each test comprses a coherent scale. Both tests show only a slght postve skew wth mean marks around 50% suggestng that the tests are approprately targeted at the cohort. The correlaton between canddates scores on the tests s hgh, whch would suggest that a sngle trat s beng measured across both tests, and that the herarchcal structure wthn the data has mnmal effect. Indeed, the dsattenuated correlaton, whch approaches a value of 1, s hghly suggestve of a sngle dmenson beng assessed. In spte of ths apparent undmensonalty, there s stll, at least, a strong theoretcal case for the use of a testlet model n ths specfc case as the tests are sets of tems that are desgned to be admnstered separately and are lkely to be taught as coherent courses n separaton. The consequences of fttng the testlet model to ths hghly coherent data set are evaluated n the next secton. Mean SD Max Cronbach s Alpha Table 2: Descrptve Statstcs Skew Kurtoss Average Item to Test Correlaton Inter-Unt Correlaton Test Test

10 Mean gamma Estmaton of composte score classfcaton accuracy The testlet effect To evaluate the magntude of the testlet effect for the cohort as a whole, the rato of the varaton n abltes can be compared to the varaton ntroduced by the testlet effect defned as: d( j) 2 where d( j) s the varance assocated wth the testlet parameter. For the operatonal data set presented, ths value s 0.11 suggestng a low level of local dependence wthn the testlets. Further, around 11% of canddates have a gamma value that does not ntersect wth zero wthn one standard devaton (Fgure 1). Whle there does therefore appear to be a measurable testlet effect t would seem unlkely that the testlet model would perform consderably better than a model whch neglects ths herarchy for the purposes of estmatng classfcaton accuracy n ths nstance. Addtonal value s, however, added under the testlet model snce nformaton s combned from both tests durng parameter estmaton. The analogous non-herarchcal approach s to apply the PCM model to all tems combned across both tests (as prevously specfed as model 2). Ths approach s appled n ths case to examne the degradaton due to loss of condtonal dependence; but t s only potentally vable wth ths hghly coherent data set as t would volate assumptons of the model n the majorty of cases. (9) Mean theta Fgure 1: Mean and standard devaton of gamma values

11 count 172 C. Wheadon & I. Stockford 3.3 Model ft of observed data To further evaluate the approprateness of the appled models t s necessary to establsh how well the models ft the data. As measures of classfcaton accuracy are based on the cumulatve nformaton yelded by all tem nformaton, and the dstrbuton of score probabltes compared to the grade boundares, the most mportant measure of ft appears to be the comparson of the observed and expected score dstrbuton. Ths predcted observed score dstrbuton s defned as the composte condtonal sum score dstrbutons, provded by equatons 7 and 8, summed across all canddates (Hanson & Bégun, 2002). Therefore, the IRT models were ftted to the observed data and the estmated frequency dstrbutons compared wth the observed dstrbuton. As can be seen from Fgure 2, the estmated dstrbuton from the PCM ntersected the multple models produced by the Bayesan ft performed under the testlet model. Crtcally, the expected score dstrbutons under both models appears to follow the observed dstrbuton suggestng good model ft at the test level n both cases model TRT 1 TRT 2 TRT 3 TRT 4 TRT 5 TRT 6 TRT 7 TRT 8 TRT 9 TRT 10 PCM score Fgure 2: Observed and expected score dstrbutons wth grade boundares super-mposed

12 Estmaton of composte score classfcaton accuracy Classfcaton accuracy The canddate level composte score classfcaton accuracy values are llustrated n Fgure 3 (ncludng data sets usng smulated tem parameters for later reference). All of the models follow the same typcal shape, wth the lowest values at the grade boundares whch are fundamentally lmted to a maxmum of 0.5. As can be seen from Fgure 4, the dfferences between the testlet model and the PCM models are small apart from score ponts drectly around the grade boundares, wth the largest dfferences occurrng around the narrowest boundares. Ths s due to any dfferences between the models beng accentuated by the narrow boundares where senstvty to msclassfcaton s greatest. Snce the testlet propenstes are small, the dfferences between the models are lkely to be due to the combnaton of nformaton across tests due to jont estmaton under the testlet model and the constrant of tem parameters to a common scale resultng n dfferng tem level ft. The summary classfcaton accuracy ndces for the two ndvdual tests are around 0.79 under both models. Whlst the accuracy wth whch canddates are classfed at the test level s not of prmary concern here t s worth notng that these values are hgher than any measured n Wheadon and Stockford (2011). From consderaton of the descrptve statstcs, ths s unsurprsng gven that both tests have consderably hgher mean grade boundary separatons (13.8 marks and 13.6 marks, respectvely) than any consdered as part of the prevous study. The composte score classfcaton ndces for the dfferent models are presented n Table 3, ncludng the values for the smulated data sets for later reference. Due to a combnaton of the ncrease n measurement nformaton provded by multple tests and the ncreased separaton of subject level grade boundares (27.4 marks) over those found at test level, the composte score classfcaton ndex values ncrease to around These tests beneft from long raw mark scales whch allow clear dfferentaton of ablty and wde spacng of grade boundares. Vrtually no dfference s apparent between the jont estmaton of the partal credt model and the separate estmaton of the partal credt model. Dfferences would be due to the loss of condtonal ndependence between the test scores on the two separate tests when the model parameters are estmated separately. As the tests are hghly correlated, the degradaton represents a worst case scenaro. Ths shows that the consequences of volatng ths assumpton may be mnmal when estmatng the classfcaton accuracy summary statstc. Table 3: Classfcaton accuracy under dfferent models PCM (Model 1) Operatonal Data PCM wth JE (Model 2) Testlet (Model 3) Smulated Parameters Rasch Testlet Mean Classfcaton Accuracy SD of Classfcaton Accuracy

13 Fgure 3: Classfcaton accuracy under dfferent models Classfcaton Accuracy 174 C. Wheadon & I. Stockford model PCM (model 1) PCM Jont Estmaton (model 2) Rasch Smulated Betas Testlet (model 3) Testlet Smulated Betas E D C B A A* Total Score

14 Fgure 4: Dfference n classfcaton accuracy from PCM estmaton Classfcaton Accuracy dfference from PCM estmaton Estmaton of composte score classfcaton accuracy model Testlet (model 3) Testlet Smulated Betas Rasch Smulated Betas PCM Jont Estmaton (model 2) E D C B A A* score

15 count 176 C. Wheadon & I. Stockford 3.5 Smulated data sets Model ft of smulated data Whle models of the operatonal data showed good ft to the observed score dstrbutons, the entrely smulated models showed a poor ft to both the observed dstrbuton and to each other, as shown n Fgure 5. However, snce these person and tem parameters were drawn from arbtrary (yet reasonable) dstrbutons of parameters, ths s not altogether surprsng and these dfferences are lkely to be due to the dstrbutons of smulated parameters beng poorly matched to the operatonal data. To nvestgate mprovements to the accuracy of the smulaton, the effects of fxng ether the person or tem parameters was consdered. Snce t s more lkely that the dstrbuton of person parameters can be estmated from performance elsewhere, the values of were constraned to match those arsng from the operatonal data. Ths gves rse to the estmated composte score dstrbutons gven n Fgure 6. As expected, ths yelded more satsfactory ft to the observed dstrbuton for both models, although the data smulated under the testlet model seems to provde a better ft than the data smulated under the Rasch model Classfcaton accuracy for smulated data In addton to the operatonal data sets the classfcaton accuracy plots for the smulated data wth constraned person parameters are shown n Fgures 3 and 4. It should be noted model TRT 1 TRT 2 TRT 3 TRT 4 TRT 5 TRT 6 TRT 7 TRT 8 TRT 9 TRT 10 Rasch score Fgure 5: Observed and smulated expected score dstrbutons

16 count Estmaton of composte score classfcaton accuracy model TRT 1 TRT 2 TRT 3 TRT 4 TRT 5 TRT 6 TRT 7 TRT 8 TRT 9 TRT 10 Rasch score Fgure 6: Observed and smulated expected score dstrbutons wth constraned and values that, for the testlet model, the shape of the relatonshp between classfcaton accuracy and composte score s largely the same for the smulated and the observed data. Ths suggests that the dfference n summary statstc (presented n Table 3) largely occurs due to the dfferng dstrbutons of canddates across the composte mark scale rather than dfferences n the underlyng models. Fgure 3 shows the estmates of classfcaton accuracy are hgher for the smulated data set usng the Rasch model than the PCM model for all composte score values other than those on the A* boundary. Ths s because the tem parameters are drawn from a normal dstrbuton wth a mean of 0 and a standard devaton of 1 and do not reflect the tem structure or the parameter values of the observed data. The dfferences are mnmal, however, suggestng that t may be possble to generate reasonable estmatons of classfcaton accuracy at ndvdual ponts on the composte score scale. The accuracy of estmates of summary statstcs, however, remans dependent on knowledge of unknown, but not wholly unpredctable, populaton denstes. 4 Dscusson Ths paper has shown how the work of Lee (2008) on the estmaton of classfcaton accuracy measures for sngle tests can be extended to estmate classfcaton accuracy for scores comprsed of dscrete tests by usng a two stage applcaton of Lord and Wngersky s (1987) recurson formula. The approach s appealng through ts ntutve use of

17 178 C. Wheadon & I. Stockford the probablty dstrbutons yelded by potentally smple IRT models whch replaces the need to model any multdmensonalty or herarchcal structure whch may exst between tests. Provded the model used to represent each test s locally vald, the test level probablty dstrbutons to whch the model nformaton s dstlled can be appled to the process outlned here. Furthermore, t s relatvely easy to apply lnear or non-lnear scalng to the scores once the probabltes have been derved; the scalng process s problematc for other composte score classfcaton procedures. The smplcty of the proposed model comes at the expense of some reducton n test nformaton suppled to the fttng process f parameter estmaton s undertaken ndependently at the test level. As the probabltes derved from each test are not ndependent, there s also lkely to be some further degradaton of the classfcaton accuracy estmates. However, the results presented here suggest the degradaton may be mnmal. Where data s not avalable, the paper has also shown how smulated tem parameters can yeld reasonably accurate values of classfcaton accuracy along a score scale. How well tem and person nformaton can be predcted more generally, however, s an emprcal queston that could be worth further nvestgaton n partcular contexts. The present study was lmted to a consderaton of two hghly correlated tests. The more closely correlated the tests the more the estmates of composte score classfcaton accuracy wll be degraded due to loss of local ndependence between the test scores. Further work s requred to demonstrate the robustness of the proposed approach wth data sets wth varyng degrees of multdmensonal herarchcal effects and comparson to classfcaton accuracy estmates derved from other modellng solutons. The dervaton of a smple, mathematcally appealng approach to the calculaton of classfcaton accuracy for scores derved from multple tests opens up a range of further research opportuntes. The approach could be used, for example, to determne the relatve strengths and weaknesses of dfferent scalng and aggregaton schemes. Most mportantly, however, the smplcty of the approach means that the estmaton of classfcaton accuracy at the composte score level could become a routne part of qualfcaton qualty measures as opposed to a research actvty n tself. References AQA. (2011). Unform marks n A-level and GCSE exams and ponts n the Dploma. Manchester: Assessment and Qualfcatons Allance. Bramley, T., & Dhawan, V. (2010). Estmates of relablty of qualfcatons. Coventry, UK: Offce of Qualfcatons and Examnatons Regulaton. Breyer, F., & Lews, C. (1994). Pass-fal relablty for test wth grade boundares: a smplfed method (ETS Research Rep. No ). Prnceton, NJ: Educatonal Testng Servce. Chester, M. D. (2003). Multple measures and hgh-stakes decsons: a framework for combnng measures. Educatonal Measurement: Issues and Practce, 22, Curts, S. M. (2010). BUGS Code for Item Response Theory. Journal of Statstcal Software, Code Snppets, 36(1),1-34.

18 Estmaton of composte score classfcaton accuracy Douglas, K. M., & Mslevy, R. J. (2010). Estmatng classfcaton accuracy for complex decson rules based on multple scores. Journal of Educatonal and Behavoural Statstcs, 35(3), Fox, J.-P. (2010). Bayesan Item Response Modelng. New York: Sprnger. Frey, A., & Setz, N.-N. (2011). Hypothetcal Use of Multdmensonal Adaptve Testng for the Assessment of Student Achevement n the Programme for Internatonal Student Assessment. Educatonal and Psychologcal Measurement, 71(3), Hanson, B. A., & Bégun, A. A. (2002). Obtanng a common scale for tem response theory tem parameters usng separate versus concurrent estmaton n the common-tem equatng desgn. Appled Psychologcal Measurement, 26, Hanson, B. A., & Brennan, R. L. (1990). An nvestgaton of classfcaton consstency ndexes estmated under alternatve strong true score models. Journal of Educatonal Measurement, 27, He, Q. (2009). Estmatng the relablty of composte scores. Coventry, UK: Offce of Qualfcatons and Examnatons Regulaton. Lee, W. (2008). Classfcaton consstency and accuracy for complex assessments usng tem response theory. (No. 27) CASMA Research Report. Iowa Cty, IA: Center for Advanced Studes n Measurement and Assessment, Unversty of Iowa. Lee, W., Hanson, B. A., & Brennan, R. L. (2002). Estmatng consstence and accuracy ndces for multple classfcatons. Appled Psychologcal Measurement, 26(4), L, Y., L, S., & Wang, L. (2010). Applcaton of a general polytomous testlet model to the readng secton of a large-scale Englsh language assessment (Research Report). Prnceton, New Jersey: Educatonal Testng Servce. Lvngston, S. A., & Lews, C. (1995). Estmatng the consstency and accuracy of classfcatons based on true scores. Journal of Educatonal Measurement, 32(2), Lord, F., & Wngersky, M. (1984). Comparson of IRT true-score and equpercentle observed-score equatngs. Appled Psychologcal Measurement, 8, Lunn, D. J., Thomas, A., Best, N., & Spegelhalter, D. (2000). WnBUGS a Bayesan modellng framework: concepts, structure, and extensblty. Statstcs and Computng, 10, Mar, P., Hatznger, R. & Maer, M. (2010). erm: Extended Rasch Modelng. R package verson , Masters, G. (1982). A Rasch model for partal credt scorng. Psychometrka, 47(2), Offce of Qualfcatons and Examnatons Regulaton. (2012). Ofqual s Relablty Compendum. Coventry, UK: Offce of Qualfcatons and Examnatons Regulaton. Opposs, D., & He, Q. (2011). The relablty programme: fnal report. Coventry, UK: Offce of Qualfcatons and Examnatons Regulaton. Peng, C., & Subkovak, M. J. (1980). A note on Suynh s normal approxmaton procedure for estmatng crteron-referenced relablty. Journal of Educatonal Measurement, 17,

19 180 C. Wheadon & I. Stockford Plummer, M. (2012). Just Another Gbbs Sampler, verson sourceforge.net/ R Development Core Team. (2012). R: A Language and Envronment for Statstcal Computng, Rasch, G. (1960). Probablstc Models for Some Intellgence and Attanment Tests (Reprnt, wth Foreword and Afterword by B. D. Wrght, Chcago: Unversty of Chcago Press, 1980). Copenhagen, Denmark: Danmarks Paedogogske Insttut. Reckase, M. D. (1997). The past and future of multdmensonal tem response theory. Appled Psychologcal Measurement, 21(1), Su, Y-S., & Yajma, M. (2011). R2jags: A Package for Runnng jags from R, R package verson , van Rjn, P., Verstralen, H., & Bégun, A. A. (2009). Classfcaton accuracy of multple-test based decsons usng tem response theory. Paper presented at the annual meetng of the Natonal Councl on Measurement n Educaton, San Dego, CA. Verstralen, H. H. F. M., & Verhelst, N. D. (1991). Decson accuracy n IRT models (Measurement and Research Department Report 91-7). Arnhem: CITO. Waner, H., & Kely, G. L. (1987). Item clusters and computerzed adaptve testng: a case for testlets. Journal of Educatonal Measurement, 24(3), Waner, H., Bradlow, E. T., & Wang, X. (2007). Testlet Response Theory and Its Applcatons. Cambrdge: Cambrdge Unversty Press. Wang, T., Kolen, M. J., & Harrs, D. J. (2000). Psychometrc propertes of scale scores and performance levels for performance assessments usng polytomous IRT. Journal of Educatonal Measurement, 37(2), Wheadon, C., & Stockford, I. (2012). classfy: A package for generatng IRT-based classfcaton accuracy and consstency statstcs, R package verson 0.1, http: CRAN.Rproject.org/package=classfy Wheadon, C., & Stockford, I. (2011). Classfcaton accuracy and consstency n GCSE and A level examnatons offered by the Assessment and Qualfcatons Allance (AQA) November 2008 to June Coventry, UK: Offce of Qualfcaton and Examnatons Regulaton. Woodruff, D. J., & Sawyer, R. L. (1989). Estmatng measures of pass-fal half-tests. Appled Psychologcal Measurement, 13,