Multi-Group Confirmatory Factor Analysis for Testing Measurement Invariance in Mixed Item Format Data

Transcription

1 Journal of Modern Appled Statstcal Methods Copyrght 2008 JMASM, Inc. November, 2008, Vol. 7, No. 2, /08/$95.00 Mult-Group Confrmatory Factor Analyss for Testng Measurement Invarance n Mxed Item Format Data Km H. Koh Nanyang Technologcal Unversty Sngapore Bruno D. Zumbo Unversty of Brtsh Columba Canada Ths smulaton study nvestgated the emprcal Type I error rates of usng the maxmum lkelhood estmaton method and Pearson covarance matrx for mult-group confrmatory factor analyss (MGCFA) of full and strong measurement nvarance hypotheses wth mxed tem format data that are ordnal n nature. The results ndcate that mxed tem formats and sample sze combnatons do not result n nflated emprcal Type I error rates for rejectng the true measurement nvarance hypotheses. Therefore, although the common methods are n a sense sub-optmal, they don t lead to researchers clamng that measures are functonng dfferently across groups.e., a lack of measurement nvarance. Key words: Mult-Group Confrmatory Factor Analyss, Measurement Invarance, Bnary and Ordnal Items. Introducton Mult-group confrmatory maxmum lkelhood factor analyss has become the most commonly used scale-level technque to evaluate measurement nvarance/ equvalence of a test across dfferent groups (e.g., gender, language), over dfferent medums of admnstraton (e.g., web-based versus paper-and-pencl testng), or across accommodated and non-accommodated condtons. Measurement nvarance s tenable when the relatons between observed varables and latent construct(s) are dentcal across relevant groups. In partcular, ndvduals wth the same standng on a latent varable but sampled from dfferent subpopulatons should Km H. Koh s Assstant Professor, Centre for Research n Pedagogy and Practce, Natonal Insttute of Educaton. Emal: khkoh@ne.edu.sg. Bruno D. Zumbo s Professor of Measurement, Evaluaton and Research Methodology, as well as member of the Department of Statstcs and the Insttute of Appled Mathematcs. Emal hm at: bruno.zumbo@ubc.ca. An earler verson of ths artcle was presented at the 2007 Amercan Educatonal Research Assocaton (AERA) conference. have the same expected observed score on a test of that varable (Horn and McArdle, 1992). The common understandng n the research lterature s that wthout measurement nvarance, observed means (or latent means) are not drectly comparable (Drasgow & Kanfer, 1985). Mxed tem format data are often found n educatonal measurement wheren many classroom and large-scale assessments n use today are blended nstruments that nclude a mxture of multple-choce and constructedresponse tems. Typcally, multple-choce tems are dchotomously scored and constructedresponse tems are polytomously (partal-credt) scored. These two types of scores are on an ordnal scale. Two commonly encountered, and nterrelated, problems assocated wth ordnal scale are measurement scale coarseness and multvarate nonnormalty. Measurement scale coarseness s caused by a crude classfcaton of the latent varables to ordnal scales wth small numbers of response categores. Because of the dscrete nature of ordnal scales, the dstrbutons of the response data obtaned from dchotomous and polytomous tems are not conducve to multvarate normalty. Ideally, data derved from an ordnal scale should be analyzed usng estmaton methods that are desgned for use wth such data. Weghted Least Squares (WLS, Jöreskog 471

2 MEASUREMENT INVARIANCE IN MIXED ITEM FORMAT DATA & Sörbom, 1996), Asymptotc Dstrbuton Free (ADF, Browne, 1984), or Robust Maxmum Lkelhood estmaton of model parameters usng the polychorc correlaton and asymptotc covarance matrx s theoretcally sound for MGCFA wth ordnal and mxed tem format data. Practtoners, however, seldom use these methods. The mplct reasonng appears to be two-fold: (a) there s lack of awareness of these relatvely new methods, and (b) these new methods are understood to requre large sample szes; larger than ones found n many research settngs, and are, generally, not computatonally vable wth tests or measures nvolvng more than 25 tems 1. Consequently, the ordnal-scaled data are often treated as f they were contnuous and analyzed wth the normal theory Maxmum Lkelhood (ML) estmaton method and Pearson covarance matrx. The purpose, therefore, of ths study was to nvestgate the statstcal propertes of the maxmum lkelhood factor analyss of a Pearson covarance matrx for 1 The WLS/ADF estmaton method requres relatvely large sample szes (.e., at least 2,000-5,000 observatons per group, Browne, 1984) to allevate problems due to convergence or mproper solutons and s not a vable method for models wth a large number of tems. Also, dagonally weghted least squares wth the correspondng asymptotc covarance matrx and the polychorc (or tetrachorc) covarance matrx s lmted due to the fact that no more than 25 tems can be used due to the excessve computer memory demands wth the so-called weght matrx,.e., asymptotc covarance matrx of the vectorzed elements of the observed covarance matrx. Wth p varables there are L elements n the same covarance matrx, and the weght matrx s of order LxL, where L=(p(p+1))/2. Therefore, as an example, for a model that has 20 tems, the weght matrx would contan 22,155 dstnct elements and for 25 tems the weght matrx would contan 52,975 dstnct elements. Lkewse, the Satorra-Bentler corrected ch-square n LISREL and Muthen s estmaton method for ordered categorcal data n the software Mplus are also lmted by the large number of tems that are found n large-scale educatonal measurement. Therefore, most appled research n MGCFA has ordnal or mxed tem format data wth small sample szes and large numbers of tems, therefore these computatonal and statstcal restrctons prevent many appled researchers from usng the WLS/ADF estmaton method. testng measurement nvarance hypotheses n MGCFA wth mxed tem format data. Specfcally, the study examned the effects of mxed tem formats and sample sze combnatons on the Type I error rates of MLbased ch-square dfference tests for two commonly nvestgated measurement nvarance hypotheses, namely strong and full nvarance. To be clear, we are not advocatng usng a Pearson covarance matrx for testng measurement nvarance wth mxed tem formats, but rather we are nterested n nvestgatng: (a) what happens to the Type I error rates for those researchers who contnue to choose to use these sub-optmal methods, and (b) the emprcal Type I error rate of the extant research lterature that used these sub-optmal methods (before the more optmal ones were wdely avalable) for measurement nvarance. We are also not advocatng for the exclusve use of hypothess testng n ths context. Our am s to reflect common research and appled measurement practce (both n terms of the methods used and the type of data) and hence to document the Type I error rates that one would fnd n these appled settngs. Ths matter of keepng an eye on everyday research practce wll come up agan n the Methods Secton when we descrbe the varous hypothess tests we are nvestgatng. Theoretcal Framework The fundamental dea underlyng the measurement models n MGCFA s the use of a set of observable varables (.e., tems) to represent the latent varable(s). When the ordnal-scaled tems are used as proxes for the latent contnuous varable(s), the assumptons of nterval measurement scale and multvarate normalty are volated. Measurement errors nduced by a crude categorzaton of the latent contnuous varables can lead to the volatons of the covarance structure. Because the Pearson covarance s attenuated n the ordnal varables, the covarance structure model may not hold for the observed varables. Therefore, ML estmaton based on the dstorted sample covarance matrx s lkely to be based. When ordnal data are used wth the ML estmaton method and Pearson covarance matrx n sngle-group confrmatory factor 472

3 KOH & ZUMBO analyss, the ch-square goodness of ft statstc s nflated due to departures from multvarate normalty n the observed varables, albet neglgble bas s found n the model parameter estmates (e.g., Hutchnson & Olmos, 1998; Muthén & Kaplan, 1992; Potthast 1993; Rgdon & Ferguson, 1991). Hence, usng the ML chsquare statstc as a formal test statstc of model-data ft under the condtons of multvarate nonnormalty leads to an nflated Type I error rate for rejectng a true model. Methods Smulaton data focused on the stuaton wheren one has a test wth a mxture of dchotomously and polytomously scored tems. The desgn varables were three condtons of mxed tem formats and sx sample sze combnatons, resultng n a 3 6 factoral desgn wth 18 cells n our smulaton expermental desgn. Wthn each cell, 100 replcatons were generated. A 30 tem test was smulated wth mxed tem formats that were vared accordng to the proportons of dchotomous and polytomous tems as follows: A. 67% (20) dchotomous tems and 33% (10) polytomous tems (3 scale ponts), B. 50% (15) dchotomous tems and 50% (15) polytomous tems (3 scale ponts), and C. 33% (10) dchotomous tems and 67% (20) polytomous tems (3 scale ponts). These tem format proportons reflect the real achevement assessment data found n educatonal testng contexts such as the Trends n Internatonal Mathematcs and Scence Study (TIMSS) and the Natonal Assessment of Educatonal Progress (NAEP). Gven that most of the achevement data, when partal scores are allotted, use 3-category polytomous tems, the polytomous tems n the smulaton were lmted to tem responses wth 3 scale ponts. The sample sze combnatons conssted of equal and unequal sample szes for the two groups: 200 vs. 200; 500 vs. 500; 800 vs. 800; 200 vs. 500; 200 vs. 800; and 500 vs These were the typcal sample szes across two groups used wth the ML estmaton method and Pearson covarance matrx n MGCFA appled research. Smulaton Procedure For undmensonal dchotomous tems, the tem responses were generated from the three-parameter logstc (3PL) tem response theory model (Brnbaum, 1968), (1 c ) P ( θ ) = c + 1+ exp[ 1.7a ( θ b )], where a, b and c are the tem dscrmnaton, dffculty, and guessng parameters, respectvely. The P ( θ ) denotes the probablty of answerng correctly to tem by a randomly selected examnee wth ablty. The 3PL tem parameters a, b, and c of each of the 20 dchotomous tems were real tem parameter estmates taken from the 1999 TIMSS Mathematcs Achevement Test. Usng a random number generator to produce numbers unformly dstrbuted on the nterval [0,1], the probabltes were converted to ether 0s or 1s to reflect examnee tem scores. When the random number selected was less than or equal to P ( ), a 1 was assgned to an examnee for tem, and a 0 otherwse (Hambleton & Rovnell, 1986). For the polytomously scored tems, the generalzed partal credt model (GPCM)(Murak, 1992) was used to generate undmensonal polytomous tem responses, whch were categorzed nto r +1 ordered score categores (0, 1,, r ) for -th tem. The model states that the probablty of gettng tem score U j =q for a randomly sampled examnee wth ablty to the -th tem s gven by Σ P ( θ ) = Pr ob( U = q ) =, q θ exp[ Σ r j= 0 exp[ Σ 1.7a ( θ b q v= 0 j v= 0 1.7a ( θ b q = 0,1,, r, + d v )] + d v )], where a s the slope parameter of tem ; b s the locaton parameter of tem ; and d v are a set of threshold parameters of tem wth assocated constrans d 0 = 0 and 473

4 MEASUREMENT INVARIANCE IN MIXED ITEM FORMAT DATA Σ v r =1 d v = 0 (Murak, 1992). A total of 20 polytomous tem parameters (as, bs, ds) were obtaned from the TIMSS data. The approach descrbed by González- Romá, Hernández & Gómez-Bento (2002) was used to generate ordered polytomous tems. For each examnee, a latent trat estmate was generated from a standard normal dstrbuton, N(0,1). The GPCM probabltes were summed across categores to create a cumulatve probablty for each score level, and then the probablty of respondng above category k [ ] was computed. For each smulated tem and examnee a sngle random number (u) was randomly sampled from a unform dstrbuton over the nterval [0,1], and the tem scores were assgned as follows: * 2 θ * * 2 ( θ ) < u P1 ( θ * k = 3 f ( ) u P P k = 2 f ) k = 1 f P ( ) < u 1 θ. Two populaton data were smulated wth equvalent parameters to represent measurement nvarance. The populaton data conssted of 20 dchotomous and 20 polytomous tems. Data sets wth dfferent proportons of dchotomous and polytomous tems were then created by a random selecton of the tems from the frst two populaton data. As can be seen n Table 1, the tem response dstrbutons across groups for each of the mxed tem format condtons were only slghtly negatvely skewed. Testng for Measurement Invarance Hypotheses Three MGCFA nested models were used for the testng of the strong and full measurement nvarance hypotheses. Model 1 served as a baselne model where no parameters were constraned between groups. The baselne model was properly specfed and hence model msspecfcaton was not a condton n the study. The frst ch-square value was obtaned from the baselne model for comparson wth more constraned models. In Model 2 (.e., strong measurement nvarance model), the number of factors and factor loadngs were Table 1: Mean Skewness of the Mxed Item Format Populaton Data Mxtures of Mean Item Formats Skewness 67% Dchotomous and % Polytomous Items 50% Dchotomous and % Polytomous Items 33% Dchotomous and % Polytomous Items constraned to be equal across groups. The number of factors, factor loadngs, and error varances were constraned to equalty across groups n Model 3 (.e., full measurement nvarance model). The tenablty of an nvarance hypothess s determned by the statstcal sgnfcance of the ch-square dfference test between two nested models. A non-sgnfcant ch-square dfference test statstc (e.g., baselne model versus full measurement nvarance model) ndcates that the full measurement nvarance hypothess s tenable. It should be noted that, wth an eye toward reflectng what goes on n research practce, we dd not test for the equalty of ntercepts -- and hence we dd not use a mean and covarance structure (MACS) model (Wu, L, & Zumbo, 2007). That s, even though there has been perodc advocacy for testng for equalty of ntercepts t has been largely neglected n appled measurement practce. A thorough revew of emprcal tests of measurement nvarance n appled psychology by Vandenberg and Lance (2000) revealed that although 99% of the studes that they had revewed nvestgated loadng nvarance, only 12% nvestgated ntercept equalty and 49% nvestgated resdual varance equalty. Therefore by not usng the MACS model and not testng ntercepts we are not advocatng that one gnore ntercept equalty but rather we are amng to reflect common research practce. In short, we want our emprcal Type I error rates from our smulaton study to reflect those error rates n the research lterature and n practce. 474

5 KOH & ZUMBO Estmaton Method The MGCFA was conducted by usng the Pearson product moment covarance matrces along wth the normal theory ML estmaton method n the LISREL Dependent Varables For each combnaton of the condtons, MGCFA was conducted for testng the two hypotheses of measurement nvarance. Effects of mxed tem formats and sample sze combnatons on the tests of hypotheses of measurement nvarance were analyzed through the mean rejecton rates of the true models (Type I error rates). Results A qualty check on the smulated data was conducted by testng the full and strong measurement nvarance hypotheses at the populaton level for each mxed tem format combnaton. As can be seen n Table 2, the dfferences n ch-squares between models, that s, baselne vs. full nvarance, and baselne vs. strong nvarance are not statstcally sgnfcant at the alpha level of.05. The results ndcate that the factor structure of the artfcal achevement test s nvarant across groups. Thus, any sample data drawn from the populaton data are expected to yeld equvalent factor structures for the two groups n the MGCFA framework. The results n Table 3 show that the emprcal rejecton rates of the ML ch-square dfference test have the nomnal alpha (.05) that fall wthn ther two-taled confdence nterval (at a Bonferron corrected confdence nterval of 99%) for the full and strong measurement nvarance hypotheses across mxed tem formats and sample sze combnatons. Ths ndcates that mxed tem formats and sample sze combnatons do not affect the emprcal Type I error rates of the ML ch-square dfference tests n the hypotheses testng of full and strong measurement nvarance. Keep n mnd that the tem response dstrbutons across groups are not very skewed. Concluson The fndngs of the current study suggest that the practce of usng mult-group confrmatory maxmum lkelhood factor analyss of a Pearson covarance matrx to test measurement nvarance hypotheses wth mxed tem format data does not lead to nflated ch-square dfference test statstcs. These fndngs are certanly welcome news for someone readng and revewng the extant research lterature and research reports. However, although these are postve fndngs, we encourage researchers to use methods that treat the data as ordnal (e.g., polychorc matrces or perhaps full-nformaton methods) and to test for the equalty of ntercepts. Our results lead us to conclude that although common practce s, n a sense, suboptmal t at least s not leadng to a tendency to over-clam dfferences n measurement scales across groups.e., an nflated Type I error rate. [The reference lst can be found after the subsequent tables.] 475

6 MEASUREMENT INVARIANCE IN MIXED ITEM FORMAT DATA Table 2: Maxmum Lkelhood Ch-square Goodness-of-Ft Statstcs between Models Ch-square Dfference Mxed Item Format Model P Statstc 67% Dchotomous Items 33% Polytomous Items (20:10) Baselne vs. Full Invarance Baselne vs. Strong Invarance ² = 32, df = 60 ² = 21, df = % Dchotomous Items 50% Polytomous Items (15:15) Baselne vs. Full Invarance Baselne vs. Strong Invarance ² = 38, df = 60 ² = 23, df = % Dchotomous Items 67% Polytomous Items (10:20) Baselne vs. Full Invarance Baselne vs. Strong Invarance ² = 39, df = 60 ² = 23, df = Note: Numbers of dchomotous and polytomous tems are n parentheses. Table 3: Emprcal Type I Error Rates of ML Ch-square Dfference Test for the Full and Strong Measurement Invarance Hypotheses Across Mxed Item Formats and Sample Sze Combnatons Sample Szes (n1: n2) Hypothess 67% Dchotomous 33% Polytomous Mxed Item Formats 50% Dchotomous 50% Polytomous 33% Dchotomous 67% Polytomous 200 : 200 FI SI : 500 FI SI : 800 FI SI : 500 FI SI : 800 FI SI : 800 FI SI Note: Those emprcal Type I error rates that have the nomnal alpha (.05) outsde of ther twotaled confdence nterval (at a Bonferron corrected confdence nterval of 99%) would be n bold font. FI and SI denote Full and Strong Measurement Invarance Hypotheses, respectvely. 476

7 KOH & ZUMBO References Brnbaum, A. (1968). Some latent trat models and ther use n nferrng an examnee s ablty. In F. M. Lord and M. R. Novck (Eds.), Statstcal theores of mental test scores. Readng, MA: Addson-Wesley. Browne, M.W. (1984). Asymptotcally dstrbuton-free methods for the analyss of covarance structures. Brtsh Journal of Mathematcal and Statstcal Psychology, 37, Drasgow, F., & Kanfer, R. (1985). Equvalence of psychologcal measurement n heterogeneous populatons. Journal of Appled Psychology, 70, González-Romá, V., Hernández, A., & Gómez-Bento, J. (2002). An evaluaton of the multple-group mean and covarance structure analyss model for detectng dfferental tem functonng n graded response tems. Paper presented at the Internatonal Test Commsson (ITC) Conference on Computer-Based Testng and the Internet. Wnchester, UK. Hambleton, R. K., & Rovnell, R. J. (1986). Assessng the dmensonalty of a set of test tems. Appled Psychologcal Measurement, 10, Horn, J. L., & McArdle, J. J. (1992). A practcal and theoretcal gude to measurement nvarance n agng research. Expermental Agng Research, 18, Hutchnson, S. R., & Olmos, A. (1998). Behavor of descrptve ft ndexes n confrmatory factor analyses usng ordered categorcal data. Structural Equaton Modelng, 5, Jöreskog, K. & Sörbom, D. (1996). LISREL 8: User s reference gude. Chcago, IL: Scentfc Software Internatonal. Murak, E. (1992). A generalzed partal credt model: Applcaton of an EM algorthm. Appled Psychologcal Measurement, 16, Muthén, B., & Kaplan, D. (1992). A comparson of some methodologes for the factor analyss for non-normal Lkert varables: A note on the sze of the model. Brtsh Journal of Mathematcal and Statstcal Psychology, 45, Potthast, M. J. (1993). Confrmatory factor analyss of ordered categorcal varables wth large models. Brtsh Journal of Mathematcal and Statstcal Psychology, 46, Rgdon, E. E., & Ferguson, Jr. (1991). The performance of the polychorc correlaton coeffcent and selected fttng functons n confrmatory factor analyss wth ordnal data. Journal of Marketng Research, Vol. XXVIII, Vandenberg, R. J., & Lance, C. E. (2000). A revew and synthess of the MI lterature: Suggestons, practces, and recommendatons for organzatonal research. Organzatonal Research Methods, 3, Wu. A. D., L, Z., & Zumbo, B. D. (2007). Decodng the meanng of factoral nvarance and updatng the practce of multgroup confrmatory factor analyss: A demonstraton wth TIMSS data. Practcal Assessment, Research and Evaluaton, 12(3),

8 Ths document was created wth Wn2PDF avalable at The unregstered verson of Wn2PDF s for evaluaton or non-commercal use only. Ths page wll not be added after purchasng Wn2PDF.