Using SAS/OR for Automated Test Assembly from IRT-Based Item Banks

Transcription

1 Usng SAS/OR for Automated Test Assembly from IRT-Based Item Bans Yung-chen Hsu, GED Testng Servce, LLC, Washngton, DC Tsung-hsun Tsa, Research League, LLC, Matawan, J ABSTRACT In recent years, advanced development n psychometrc theory and computer technology has led to dramatc changes n test constructon practces. In testng organzatons, the tas of assemblng test forms can be consdered as an assgnment to solve a mathematcal programmng problem, n whch a certan objectve functon needs to be met subject to specfc practcal constrants, such as test length and content coverage. The SAS/OR s a powerful modelng envronment for solvng mathematcal optmzaton problems. Large or mdszed testng organzatons can greatly beneft from ts avalable solvers to handle real test assembly problems and needs. We used procedure OPTMODEL to present the programmng formulaton of a test assembly problem. The mert of the PROC OPTMODEL s that the models n the SAS statements are declared n a form whch mmcs the symbolc formulaton of an optmzaton model. A generc test assembly model wth specfc statstcal constrants based on tem response theory and content related test specfcatons was shown as an eample. The model can be easly modfed to nclude addtonal practcal constrants to obtan an optmal soluton. Testng organzatons can use ths approach to automatcally create test forms f relatvely well developed tems bans are avalable. ITRODUCTIO A test assembly problem s to select a set of tems from a large pool of pre-calbrated tems, nown as an tem ban, based on the test specfcatons. An tem ban s a repostory of test tems, essentally a database, whch stores all nformaton pertanng to the tems such as tem format, tem characterstcs and content domans. Optmzaton technques see optmal solutons under specfc constrants of psychometrc and test composton specfcatons wthn a gven tem ban. The technques have been a major advancement for testng organzatons to acheve automated test assembly. The early automated test assembly method usng a mathematcal programmng approach was not based on the psychometrc test theory (Feuerman and Wess, 1973). Due to the advanced nnovatons n psychometrc theory and computer technology, current test constructon practces use mathematcal programmng technques are generally based on tem response theory (IRT). IRT s a modern psychometrc test theory that descrbes the relatonshp between tem characterstcs and test taer abltes. The three-parameter logstc model (3PL) s one of the wdely used undmensonal IRT models for dchotomous responses n varous large-scale testng programs. The model can be epressed as P (, a, b, c ) c 1 c 1 ep[ Da ( b )] representng the probablty of answerng a partcular dchotomously scored tem correctly gven the profcency level. The parameters a, b, and c are the characterstcs of tem and the common choce of the scalng constant D s 1.7. Generally, the tem parameters can be estmated by usng PROC LMIXED (Sheu, Chen, Su, & Wang, ). The tem nformaton functon, whch s derved from Fsher nformaton (Load, 198, Suen 199, van der Lnden & Boeoo-Tmmnga, 1989), s defned as I ( ) ' [ P ( )] P ( ) Q ( ) where Q ' 1 P and P ( ) ( ) /. The test nformaton of a test ncludng n tems s defned as P n ) I ( ). 1 I ( 1

2 MATHEMATICAL APPROCHES OF COSTRUCTIG TEST FORMS Thenussen (198) frst presented a bnary nteger programmng approach to construct a test wth a target nformaton functon. The objectve of the model s to mnmze the number of tems n the test subject to the constrants that the nformaton n the test s above the pre-specfed target at a number of ablty levels. Several practcal constrants were consdered to ncorporate nto the modelng approach (Theunssen, 1986; Baer, Cohen, & Barmsh, 1988; and de Grujter, 199). A dfferent perspectve, the so-called Mamn model, that consders the specfcaton of a relatve target nformaton functon, was formulated by van der Lnden & Boeoo- Tmmnga (1989) n selectng tems from an tem ban. The model can be nterpreted as specfyng the relatve shape of the target nformaton functon at certan ablty ponts. The automated test assembly problem can be treated as an optmzaton of matchng a target test nformaton functon subject to content coverage. Specfyng an absolute target test nformaton functon may not be easy n practce f there s no avalable reference. The mplementaton of Mamn model for automated test generaton targets a gven shape of the IRT test nformaton functon s especally useful for new testng programs. The nature of the test can be more easly specfed than assgnng eact target nformaton values. Actually, the formulaton of Mamn model can be treated as a specal case of the goal programmng model approach (Hsu, 1993), a branch of mult-objectve optmzaton. Most varatons (e.g., Boeoo-Tmmnga, 199; Hsu, 1993; van der Lnden & Adema, 1998) developed later for solvng practcal test assembly ssues are generally based on Mamn model for optmal test constructon. Let r be the relatve nformaton values at the ablty pont and assume that the tems n the tem ban are represented by decson varable, = 1,,, denotng whether the tems are to be ncluded nto the test form ( = 1) or not ( = ). The model s formulated as subject to 1 mamze y I ( ) r y, 1,, K 1 n {,1}, 1,, y I ( ) denotes the nformaton functon of tem I at the ablty pont and n s the test length. As an eample, the second constrant represents the number of tems n the test. The model allows addtonal practcal constrants, such as test composton (e.g., cogntve levels, mutually eclusve tems) and admnstraton tme, to be taen nto account and specfed nto the model. A EXAMPLE TEST ASSEMBLY PROBLEM We smulated a set of tem parameters to create an tem ban that has four content domans and each doman contans tems. For smplcty, we set the parameter c to zero. In practce, the c parameter has very small varaton because tems wth large value of c wll not be ncluded n the tem ban. Generally, parameter c has lttle mpact n test assembly. ITEM IFORMATIO Mamn model s used as an llustratve eample. We computed the tem nformaton at 13 ablty ponts ( = (-3, -., -,-1.,-1, -.,,., 1, 1.,,., 3)). lbname nesugdat 'c:\nesug1\dat'; %let D=1.7; %let D=%sysevalf(&D*&D); data nf; set nesugdat.temban; array r{13} r1-r13; a=a*a;

3 do =1 to 13; p=1/(1+ep(-&d*a*(((-7)/.)-b))); q=1-p; r{}=&d*a*p*q; end; eep r1-r13; run; proc transpose data=nf out=nfc pref=p; run; data nfc; set nfc (drop=_name_); run; PROBLEM FORMULATIO I PROC OPTMODEL The tas s to compose a classfcaton test of 4 tems and the test s dvded nto four equal sectons, wth tems sequenced n poston 1-1, 11-, 1-3, and 31-4 for the four domans, respectvely. We assumed that the classfcaton test has multple cut ponts, whch means the test nformaton curve would have two peas. The test assembly problem s formulated as subject to 1 I ( ) mamze y r y, 1,, {,1}, 1,, y The followng code shows the use of the procedure OPTMODEL for the test assembly tass: (1) a selectve test wth a sngle cut-off pont; () a classfcaton test multple wth cut-off ponts; and (3) a dagnostc test wth no cutoff pont. The relatve nformaton values at dfferent ablty ponts are n Table 1. We specfed two cut-off ponts for the classfcaton test. TABLE 1 Relatve nformaton values Selectve Classfcaton Dagnostc SAS/OR provdes a convenent modelng language wthn PROC OPTMODEL for formulatng, solvng, and mantanng optmzaton models. We start the problem statement wth PROC OPTMODEL. Because we are dealng a huge amount of varables, we use set statement to group the numbers for ndeng the varables. Then we declare the decson varables n the model. The code uses the selectve test as an eample. Once the problem s solved, the value represents whether an tem s selected or not. The objectve functon s smple. The constrants relate the decson varables wth the four domans. The solve statement nvoes an approprate solver to solve ths med nteger lnear programmng problem. The results are stored for further processng. 3

4 proc optmodel; set theta=1..13; set nban=1..; num Inf{theta,nBan}; read data nfc nto [j= ] { n nban} <Inf[j,]=col("p" )>; num r1{theta}=[ ]; var {nban} BIARY, y; ma obj=y; con tinf{j n theta}: sum{ n nban}inf[j,]*[]>=r1[j]*y; con ca1: sum{ n 1..}[]=1; con ca: sum{ n 1..1}[]=1; con ca3: sum{ n 11..1}[]=1; con ca4: sum{ n 11..}[]=1; solve wth mlp; create data nesugdat.testset3 from [d]={nban} sel=; qut; Fgure 1 shows the dstrbuton of the values for the log(a ) and b of the tems n the tem ban. Fgures, 3, and 4 are the test nformaton for the three tests, respectvely. The test nformaton has one pea for the selectve test and two peas for the classfcaton. The dagnostc test has flat test nformaton. 6 3 log(a) Test nformaton b FIGURE 1. Scatter plot of log(a) and b. FIGURE. Selectve test Test nformaton 3 3 Test nformaton FIGURE 3. Classfcaton test. FIGURE4. Dagnostc test. 4

5 COCLUSIO For testng organzatons, the test assembly tas s to solve a constraned combnatoral optmzaton problem. If relatvely well developed tems bans have been developed, the problem nvolves a large amount of varables. PROC OPTMODEL has a succnct way to read and create data sets. It provdes a powerful modelng language to formulate and solve the optmzaton model. Ths procedure can nterface to varous optmzaton solvers to compute solutons to the formulated problems. We showed how to formulate a smple test assembly problem. The model can be easly nspected and modfed to address a wde varety of test specfcatons. REFERECES Theunssen, T. J. J. M. (1986). Some applcatons of optmzaton algorthms n test desgn and adaptve testng. Appled Psychologcal Measurement, 1 (4), De Grutjer, D.. M. (199). Test constructon by means of lnear programmng. Appled psychologcal measurement, 14 (), Baer, F. B., Cohen, A. S. & Barmsh B. R. (1988). Item characterstcs of tests constructed by lnear programmng. Appled psychologcal measurement 1 (), Boeoo-Tmmnga, E. (199). A cluster-based method for test constructon. Appled psychologcal measurement, 14 (4), Feuerman, M., & Wess, H. (1973). A mathematcal programmng model for test constructon and scorng. Management scence, 19 (8), Hsu, Y.-C. (1993). The goal programmng approach for test constructon (Master thess). The Unversty of Arzona, Tucson, AZ. Lord, F. M. (198). Applcatons of tem response theory to practcal testng problem. Hllsdale, J: Erlbaum. Sheu, C.-F., Chen, C.-T., Su, Y.-H., & Wang, W.-C. (). Usng SAS PROC LMIXED to ft tem response theory models. Behavor Research Methods, 37 (), -18. Suen, H. K. (199). Prncples of test theores. Hllsadle, J: Erlbaum. Van der Lnden, W. J., & Boeoo-Tmmnga, E. (1989). A mamn model for test desgn wth practcal constrants. Psychometra, 4 (), Van der Lnden, W. J., & Adema, J, J. (1998). Smultaneous assembly of multple test forms. Journal of Educatonal Measurement, 3 (3), ACKOWLEDGMETS SAS and all other SAS Insttute Inc. product or servce names are regstered trademars or trademars of SAS Insttute Inc. n the USA and other countres. ndcates USA regstraton. COTACT IFORMATIO Your comments and questons are valued and encouraged. Contact the author at: Yung-chen Hsu GED Testng Servce, LLC 11 Connectcut Ave., W 4th Floor Washngton, DC 36 Wor Phone: Emal: yung-chen.hsu@gedtestngservce.com Web: