NeuroImage 60 (2012) Contents lists available at SciVerse ScienceDirect. NeuroImage. journal homepage:

Transcription

1 NeuroImage 60 (0) Contents lsts avalable at ScVerse ScenceDrect NeuroImage journal homepage: FMRI group analyss combnng effect estmates and ther varances Gang Chen a,, Zad S. Saad a, Audrey R. Nath b, Mchael S. Beauchamp b, Robert W. Cox a a Scentfc and Statstcal Computng Core, NIMH/NIH/DHHS, 9000 Rockvlle Pke, Bethesda, MD 089, USA b Department of Neurobology and Anatomy Unversty of Texas Medcal School at Houston, 643 Fannn Street, Sute G.550G Houston, TX 77030, USA artcle nfo abstract Artcle hstory: Receved 6 October 0 Revsed 5 December 0 Accepted December 0 Avalable onlne 30 December 0 Keywords: FMRI group analyss Effect estmate precson or relablty Mxed-effects multlevel analyss (MEMA) Weghted least squares (WLS) Restrcted maxmum lkelhood (REML) Outlers AFNI Conventonal functonal magnetc resonance magng (FMRI) group analyss makes two key assumptons that are not always justfed. Frst, the data from each subject s condensed nto a sngle number per voxel, under the assumpton that wthn-subject varance for the effect of nterest s the same across all subjects or s neglgble relatve to the cross-subject varance. Second, t s assumed that all data values are drawn from the same Gaussan dstrbuton wth no outlers. We propose an approach that does not make such strong assumptons, and present a computatonally effcent frequentst approach to FMRI group analyss, whch we term mxed-effects multlevel analyss (MEMA), that ncorporates both the varablty across subjects and the precson estmate of each effect of nterest from ndvdual subject analyses. On average, the more accurate tests result n hgher statstcal power, especally when conventonal varance assumptons do not hold, or n the presence of outlers. In addton, varous heterogenety measures are avalable wth MEMA that may assst the nvestgator n further mprovng the modelng. Our method allows group effect t-tests and comparsons among condtons and among groups. In addton, t has the capablty to ncorporate subjectspecfc covarates such as age, IQ, or behavoral data. Smulatons were performed to llustrate power comparsons and the capablty of controllng type I errors among varous sgnfcance testng methods, and the results ndcated that the testng statstc we adopted struck a good balance between power gan and type I error control. Our approach s nstantated n an open-source, freely dstrbuted program that may be used on any dataset stored n the unversal neuromagng fle transfer (NIfTI) format. To date, the man mpedment for more accurate testng that ncorporates both wthn- and cross-subject varablty has been the hgh computatonal cost. Our effcent mplementaton makes ths approach practcal. We recommend ts use n leu of the less accurate approach n the conventonal group analyss. Publshed by Elsever Inc. Introducton Group analyss of fmri datasets s typcally carred out n two levels. In the frst level, each ndvdual subject's dataset s analyzed n a tme seres regresson model to provde a measure of the effect of nterest (lnear combnaton of regresson coeffcents) at each voxel. In the second level, the effect estmates of nterest at each voxel n standard space are combned across subjects usng Student t-test, ANOVA, ANCOVA, multple regresson, or lnear mxed-effects (LME) models. Then, group nferences are made wth a general clam about a hypotheszed populaton from whch the sampled subjects were recruted. Ths two-level approach, by far the most common n publshed neuromagng studes (Mumford and Nchols, 009), rests on two assumptons. Frst, wthn- or ntra-subject varance of the effect estmates s unform n the group (Penny and Holmes, 007), or alternatvely, the betweensubjects varance s much larger than wthn-subject varance. Second, Correspondng author. E-mal address: gangchen@mal.nh.gov (G. Chen). effect estmates are assumed to follow a Gaussan dstrbuton.e., no outlers. The conventonal group analyss strategy works reasonably well f the requred assumptons hold to some extent. Gven the small effect szes and hgh nose levels n FMRI data, t s questonable to assume neglgble or equal standard error of the ndvdual subject effect estmates, or to gnore outlers n group analyss. Irregulartes from the scanner or outlyng BOLD responses can lead to the volaton of the assumptons of small or homoscedastc samplng errors n the standard summary statstcs approach (Penny and Holmes, 007). Dfferences n attenton to tasks and n habtuaton effects across subjects may also ntroduce dfferent precson of effect estmates. Moreover, as sophstcated experment desgns evolve, t s very typcal to have unequal numbers of subjects across groups, dfferent numbers of data ponts (tme seres lengths), or dfferent numbers of samples of a stmulus/ condton/task type across subjects. For example, due to experment constrants or subjects mssng trals, the data mght have unequal number of correct versus ncorrect responses, and such a scenaro nevtably results n heterogeneous effect estmate precson (wthn-subject varablty), potentally volatng the assumptons of conventonal group analyss methodologes /$ see front matter. Publshed by Elsever Inc. do:0.06/j.neuromage

2 748 G. Chen et al. / NeuroImage 60 (0) Another potental concern n FMRI group analyss s that the group sample sze s often farly small; thus, one or two outlers can dramatcally alter the effect estmate. Even though cross-subject varablty s typcally consdered n practce to account for such nhomogenety, outlers can nflate ts estmate, leadng to underpowered statstcal testng. Another example s the emergence of aggregated or federated datasets that come from dfferent scanners or laboratores, or wth slghtly dfferent task/condton varants. The resultng relablty dfferences n effect estmaton from multple sources necesstate an approach that crucally ncorporates the relablty heterogenety nto the model and controls for confoundng effects (e.g., personalty or phenotypc features) when amalgamatng the datasets (Bjork et al., n press). Intutvely, a summarzng approach at the group level should consder dfferentatng each subject's effect estmate based on ts precson; that s, we assgn a hgher weght to a subject f the effect estmate has a narrower confdence nterval (e.g., more relable), and vce versa. Such weghtng strategy can even be found n nature; for example, a hgh-level behavoral task s performed as an ntegraton of multple smple operatons smultaneously executed by many neurons that wegh each sensory cue proportonal to ts relablty (Ohshro et al., 0). Recent FMRI group analyss approaches have explctly consdered both effect sze and ts varance at group level. Worsley et al. (00) combned effect estmates wth ther standard devatons, and solved the resultant model wth an expectaton maxmzaton (EM) algorthm, asssted wth spatal regularzaton. Beckmann et al. (003) also dscussed the ncorporaton of relablty nformaton from the frst level to second level analyss. Woolrch et al. (004, 008) adopted a Bayesan approach through Markov chan Monte Carlo (MCMC) samplng and multvarate non-central t-dstrbuton fttng n group nference. Our contrbutons here are three-fold. Frst, we present a computatonally effcent frequentst approach that ncorporates both wthnand cross-subject varabltes at the group level, and model outlers wth a Laplace dstrbuton for the cross-subject random effects. We adopt a sgnfcance testng statstc that acheves power ncrease wth type I errors stll close to the nomnal level. Our algorthms nvolve teratve schemes at the voxel level, and we acheve executon tme on the order of mnutes for the whole bran wth a standard desktop computer. The performance of our approach wll be compared wth a Bayesan counterpart n actvaton nference wth real data and n power gan and type I error control wth smulated data. Whle the fnal whole bran statstcal nferences may not change sgnfcantly from the standard approach n cases wth szeable or homogeneous groups, we make the case for the new approach because t s more accurate, s computatonally effcent, and provdes a more detaled descrpton of the sources of varance, thereby enablng better nsght nto the data. Second, a few overall heterogenety measures across subjects are provded. A statstc s avalable for sgnfcance testng of overall heterogenety of the group. In addton, outler testng s suggested at the ndvdual level that may assst the nvestgator n dentfyng outler subjects or n ncorporatng potental covarates that could account for across-subject varablty. Thrd, we performed smulatons n varous scenaros to compare dfferent sgnfcance testng methods n cross-subject varance estmate, type I error controllablty, and power. These smulaton results are compared wth prevous work by Woolrch et al. (004) and Mumford and Nchols (009). Modelng strategy actvty n audtory and vsual cortex n each subject, provdng a good test bed for group analyss methods. Usng fve voxels as examples Fg. shows effect sze and varablty estmates n fve voxels selected from the 0-subject dataset, and llustrates the naccuracy of the two assumptons made by tradtonal group analyss methods (same wthn-subject varance and no outlers). These fve voxels were not randomly selected as representatves f such voxels exst of the entre bran; nstead they were used to showcase varous scenaros of nhomogenety n effect estmate precson. Voxels and were extracted from rght and left vsual cortex (mddle occptal gyrus) respectvely, Voxels 3 and 4 were from a left audtory regon, superor temporal gyrus (STS), and Voxel 5 was n left caudate. At least one of two assumptons n the conventonal group analyss approach s volated at each of these fve voxels. At all fve voxels, the wthn-subject varablty s sgnfcantly larger than the crosssubject varablty, and dffers markedly between subjects. At Voxels and, only half of the ten subjects had relable estmates that were sgnfcant at 0.05 level (two-sded, uncorrected), whle Voxels 3, 4, and 5 had only three or less such subjects. Subject 0 s an outler at Voxels and 3, but n dfferent ways: Voxel s sgnfcantly actvated wth the same drecton of the effect sze (outler wth a relable estmate wth the same sgn as the mean effect), whle the effect at Voxel 3 s not statstcally sgnfcant and has a dfferent sgn (outler wth an unrelable estmate wth the opposte sgn). The normal probablty plots n Fg. further ndcate the exstence of outlers at all fve voxels. More subtly, n Voxel, Subjects 5, 6, 7, and 9 have roughly the same effect estmate but wth markedly dfferent varabltes. Presentng the MEMA model The standard second-level analyss assumes that the wthnsubject varablty for the effect of nterest s relatvely small or roughly the same across subjects (Penny and Holmes, 007). The correspondng model wth n subjects can be formulated nto a regresson equaton wth p+ fxed effects, β ¼ Xp j¼0 α j x j þ δ ¼ x T a þ δ ; ¼ ; ; n; where x T =(x 0,, x p ) are known ndependent varables, a =(α 0,, α p ) T areparameters to be estmated, β s the effect of nterest from the th subject, and n partcular, α 0 s assocated wth the ntercept x 0 =. (A one-sample Student t-test can be performed usng a model that corresponds to p=0).ifp, x j can be an ndcator (dummy) varable showng, for example, the group to whch the th subject belongs, or a contnuous varable such as a subject-specfc covarate lke age, IQ or behavoral data (j=,, p), or an nteracton between fxed effects. δ s the subject-specfc error, the amount the th subject's data devates from the fxed effects at the populaton level, and s ntally assumed to follow a normal dstrbuton N(0, τ ). Of course, we don't really know the true effect β from the th subject. Instead, what we have s ts estmate ^β n the form of a lnear combnaton of regresson coeffcents from ndvdual analyss of the th subject's tme seres data. Naturally, such an estmate carres some precson nformaton, where precson s defned as the recprocal of the estmate varance. Thus, more accurately, we have ðþ Mxed-effects multlevel (or meta) analyss (MEMA) ^β ¼ β þ ε ðþ To llustrate the utlty of MEMA mplemented n the AFNI (Cox, 996) program sute as 3dMEMA, we consder a test dataset n whch 0 subjects vewed audovsual recordngs of natural speech (detals n Applcatons and results). These stmul evoked robust where ε represents the samplng error of β n the th subject, and s assumed to follow N(0, σ ), where σ s the ntra-/wthn-subject varance, whch s also unknown but can be estmated wth ^σ from the ndvdual subject analyss.

3 G. Chen et al. / NeuroImage 60 (0) Effect estmate (% sgnal change) Voxel Voxel Voxel Voxel Voxel Subject Effect estmate quantles Theoretcal quantles Fg.. (Upper panel) Indvdual subject effect estmates and ther accuracy at fve voxels are shown wth ampltudes of FMRI response to an audovsual speech stmulus n left and rght vsual cortex (Voxels and ), left and rght audtory cortex (Voxels 3 and 4), and left caudate (Voxel 5). Effect estmates from ndvdual subject analyses are ndcated wth flled crcles ( ). The varablty of each estmate s shown wth an error bar of two standard devatons, and the estmate precson s defned as the recprocal of varance. The relatve sze of the flled crcle reflects the weght of the estmate from each ndvdual subject, recprocal of the sum of wthn- and between-subject varances. The dotted horzontal lne ndcates the null hypothess of group effect beng 0. The gray horzontal lne s the group effect estmated from the conventonal approach, equal weghtng across subjects wth Student t-test. The black horzontal lne s the group effect wth the MEMA approach descrbed n the manuscrpt. The gray and black lnes overlap for Voxels 3 and 5. (Lower panel) Quantle Quantle plots of the ten subjects' effect estmates wth crcles ( ) at the fve voxels are shown aganst standard normal dstrbuton (horzontal axs). The sgnfcant devaton of the end ponts from the sold lne y=x at all fve voxels ndcates the exstence of outlers among the subjects. Combnng Eqs. () and (), we have a mxed-effects multlevel (herarchcal, or meta) analyss (MEMA) model for data from n subjects ^β ¼ Pp α j x j þ δ þ ε ¼ X T a þ δ þ ε ; or ^β N x T a; ^σ þ τ ; ¼ ; ; n j¼0 or n a concse matrx format, ^b ¼ X T a þ d þ e; or ^b N X T a; τ I n þ Φ where ^b n ¼ ^β ; ; ^β T; n XnðpþÞ ¼ðx ; ; x n Þ T ; d n ¼ ðδ ; ; δ n Þ T ; e n ¼ ðε ; ; ε n Þ T ; Φ nn ¼ dag ^σ ; ; ^σ n,andi n s an n n dentty matrx. The assumptons underlyng model (3) are: (a) ε ~N(0, ^σ ); (b) the δ 's are ndependent and dentcally dstrbuted wth N(0, τ ), where τ s the cross-/nter-/between-subjects varablty, sometmes called heterogenety; (c) Cov(ε, ε j )=0, for j, meanng the data from any two subjects are ndependent; and (d) Cov(ε, δ j )=0 for all and j, ndcatng that cross- and wthn-subject varabltes are ndependent of each other. The varance of the effect of nterest V ^b ¼ τ I n þ Φ reflects the fact that the total varablty n the data comes from two sources (or a two-stage samplng process), wthn-subject varablty Φ and cross-subject varablty τ. We can also nterpret the total varablty n a Bayesan sense as two components of the nvestgator's uncertanty (Raudenbush, 009). ð3þ Solvng MEMA If we make the (unjustfed) assumpton that both the cross-subject and wthn-subject varances, τ and σ, are known, the model (3) can be easly solved through weghted least squares (WLS) by mnmzng the weghted sum of squared resduals (Kutner et al., 004), and the X soluton s ^a ¼ X T WX T W ^b, where the weghts n W ¼ dag ; ; are the recprocals of the sum of wthn-subject τ þσ τ þσ n and cross-subject varances. The varance for ^a s a concave functon, Vð^a Þ ¼ X T ; WX ð4þ and ^a e N a; X T WX. The dervaton n (4) reles on the fact that W ½ X s of full rank because W ½ and X are of full column rank and rank (W ½ X)=rank(X). In practce both τ and σ are estmated, and so are the WLS soluton for ^a and ts varance Vð^a Þ, X ^a ¼ X T T T ^WX ^W ^b; ^V ð^a Þ ¼ X ^WX ð5þ where ^W ¼ dag ; ; ^τ þ ^σ ^τ þ ^σ n :

4 750 G. Chen et al. / NeuroImage 60 (0) Estmatng the cross-subject varablty τ Despte the suggeston that no frequentst soluton exsts for the model (3) (Woolrch, 008; Woolrch et al., 004), there have been mportant developments n the context of meta-analyss or metaregresson (e.g., combnng the results of ndependent clncal trals) durng the past 0 years (Cooper et al., 009; Hartung et al., 008). Specfcally, several methods of estmatng τ have been proposed (Vechtbauer, 005), such as the method of moments (MOM) (DerSmonan and Lard, 986), maxmum lkelhood (ML), restrcted maxmum lkelhood (REML), emprcal Bayesan (EB), among others (Hedges, 983, 989; Hunter and Schmdt, 990; Sdk and Jonkman, 005a; Sdk and Jonkman, 005b). Here we wll focus on three methods, MOM, REML, and ML usng a Laplace dstrbuton assumpton of the wthn-subject varablty (to allow for outlers). All the three methods are part of our mplementaton n 3dMEMA, and the choce of method s made partly dependng on the data at voxel level. Method of moments (MOM) We start wth a fxed-effects model by assumng no cross-subject varablty (τ =0) n Eq. (3), ^b ¼ Xa 0 þ e: An ordnary least squares (OLS) or WLS soluton for Eq. (6) provdes a prmary or provsonal estmate of a 0 n the mxed-effects model (3). Whle the OLS estmate tends to perform well when τ s relatvely large, the WLS estmate s better when τ s moderate or small. Here we adopt the WLS estmate, X ^a 0 ¼ X T W 0 X T W 0^b; ð7þ and defne the weghted resdual sum of squares (WRSS) of the WLS estmate (7) as Q ¼ ^b X^a TW0 0 ^b X^a 0 ¼ ^b T P 0^b ð8þ where W 0 ¼ dag ; ;, and P σ σ 0 =W 0 W 0 X(X T W 0 X) X T W 0. Q n s often called the homogenety statstc snce we pretend that the cross-subject varance τ =0 n calculatng Q, but ths pretense allows us to use Q to measure how much cross-subject varablty the data contan. In other words, f τ =0, we expect Q to be small; on the other hand, f τ >0, Q wll most lkely be bg. The role of Q as an ndcator of cross-subject varablty s also reflected n ts expected value, EQ ð Þ ¼ E ^b T P 0^b ¼ τ trðp 0 Þ þ n p. Equatng Q to ts expected value (Hartung et al., 008), we obtan the MOM estmate of τ, ^τ Q n p ¼ ð Þ trðp 0 Þ. To avod a negatve estmate n computaton a truncated verson s usually employed, ^τ Q n p ¼ max 0; ð Þ : ð9þ trðp 0 Þ The MOM estmate, nvolvng no teratve algorthms and thus computatonally economcal, s consstent but not necessarly effcent (Raudenbush, 009; Vechtbauer, 005), whch leads us to a more effcent method, REML, for estmatng τ. When the conventonal group analyss assumpton holds (all subjects have the same wthn-subject varance, σ = =σ n =σ ), t s nstructve T to note that the MOM estmate reduces to ^τ ¼ n p ^b X^a 0 ^b X^a 0 σ as n ths case tr(p 0 )=(n p )/σ. Furthermore, due to the truncaton nvolved n (9), smulatons (Vechtbauer, 005) showedthatmoms ð6þ slghtly postvely based when the wthn-subject varance s very large or the number of degrees of freedom at ndvdual level s too small, but the bas s neglgble when the number of degrees of freedom at the ndvdual level s above 40 and there are 0 or more subjects at group level, condtons typcally satsfed n FMRI studes. REML method The profle resdual log-lkelhood for REML s the logarthm of the densty of the observed effect treated as a functon of the crosssubject varablty τ, gven the data ^b (Raudenbush, 009; Vechtbauer, 005), l a; τ ; ^b ¼ n ln π ð Þþ ln det W ½ ð ÞŠ h ln det X T WX TW ^b X T a ^b X T a ¼ n ln π ð Þþ ln det W ½ ð ÞŠ h ln det XT WX ^b T P^b, whch leads to a Fsher scorng (FS) algorthm that s robust even for poor startng values and usually con- verges quckly (Appendx A), τ ^bt PP^b trðpþ kþ ¼ τ k þ ; ð0þ trðppþ where τ k s the kth teratve approxmaton of τ, and P=W WX (X T WX) X T W. It s worth notng that, when all subjects have the same wthn-subject varance, the REML estmate has a closed and ntutve form (Appendx A), ^τ ¼ n p ^b X T ^a actly the same as the respectve MOM estmate. T ^b XT ^a ML method wth a Laplace dstrbuton of subject-specfc error σ, ex- It s not rare to see extremely bg or small effect estmates ^b relatve to the group effect at a voxel/regon level (cf. Fg. ). Such outlers mght come from rregulartes from the scanner, outlyng BOLD responses, or pure chance. If these outlyng effect estmates are unrelable (e.g., have large varances), the mpact on the group result s mnmal, regardless of the heterogenety estmate for τ, MOM or REML, thanks to the weghtng nvolved n WLS (5). However, f the outlyng effect estmates are relable (e.g., have small varances), weghtng mght not be effectve enough and we need a more robust strategy to deal wth such outlers. For nstance, a subject mght have been gnorng the stmulus durng ts presentaton, leadng to lttle or no response to the sensory nput; ths response would be relable (wth small varance), but should obvously not be combned wth effect estmates from other subjects who were alert. The REML estmate of τ va (0) assumes a Gaussan dstrbuton of ndvdual subject's sample error, ε ~N(0, σ ), =,, n, at each voxel. The default Gaussan assumpton s omnpresent, because of ts convenent statstcal propertes and the central lmt theorem. Appealng to ths assumpton works well f the sample sze s reasonably bg, whch s not always the case n FMRI studes. When the assumpton s volated (e.g., outler voxels/regons/subjects), the crosssubject varablty τ tends to be over-estmated, and one or two outlers could dramatcally dstort the analyss, leadng to naccurate group effect estmates and/or deflated statstcal power. The conventonal approach of throwng away outlers s not only mpractcable at the voxel level, but also subjectve, arbtrary, and controversal n terms of outler dentfcaton. Here we propose a tractable alternatve model of cross-subject varablty, the Laplace (or double exponental) dstrbuton. Wager et al. (005) proposed an teratvely reweghted least squares method to handle outlers by teratvely standardzng the resduals by the medan absolute devaton, but ther model dd not dfferentate the resduals between wthn-subject and cross-subject varablty. Woolrch (008) assumed the mxtures of two Gaussan dstrbutons n the framework of Bayesan approach, one for the normal and the

5 G. Chen et al. / NeuroImage 60 (0) other for the outler subjects. Baker and Jackson (008) consdered three canddates of long-taled dstrbutons, Student t, arcsnh, and Subbotn (of whch the Laplace dstrbuton s a specal case). By extendng a method adopted for a case wth p=0bydemdenko (004) to our model (3) n the frequentst context, we assume, nstead of N(0, τ ), the followng Laplace dstrbuton for the subject-specfc error term n Eq. (3), δ ~L(0, ν), =,, n, where L(m, ν) hasdensty px; ð m; vþ ¼ v exp ½ j x m j=v Š wth locaton parameter (mean/mode/medan) m and scale parameter ν (wth a varance of ν ). The Laplace dstrbuton has heaver tals than the normal dstrbuton, allowng us to better handle outlers than REML, when one or two subjects have exceptonally unrelable effect estmates at a voxel or regon. Ths approach reduces the dsturbng effects from outlers wthout requrng arbtrary outler decsons or thresholds from the nvestgator. We adopt the Emprcal Fsher Scorng (EFS) algorthm (Demdenko, 004) n the followng format, a ¼ a þ λ ν ν k H k g k ðþ kþ k where k s the teraton ndex; H k and g k are derved n Appendx B. In descrpton we refer to the Gaussan and Laplace approaches as the ntenton of adoptng REML wth Gaussan and ML wth Laplace assumpton. However, as explaned n the Dscusson, at voxel level the real mplementaton of REML wth Gaussan and ML wth Laplace assumpton proceeds wth MOM. Only f the MOM result reaches near sgnfcance or more would t be followed and materalzed by REML or ML. Statstcal nferences wth MEMA Hypothess testng For the null hypothess of a group effect H 0 : α j ¼ 0; a testng statstc can be constructed from (5), ^α j T s ¼ s ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff X T ^WX jj ðþ ð3þ where A jj denotes the jth dagonal component of matrx A. When the number of subjects, n, s relatvely large, T S can be taken, wth a Gaussan dstrbuton approxmaton, as a Wald test (Hartung et al., 008). However, the Wald test tends to be overly lberal when appled to cases wth a moderate number of subjects (Hartung et al., 008; Raudenbush, 009), such as FMRI group analyss; thereby, t may be better approxmated wth a Studentzed t-dstrbuton. The Gauss Markov theorem guarantees that, f the cross- and wthn-subject varance τ and σ were known, the WLS estmate ^a n (5) would be unbased wth the lowest varance (X T WX) among all lnear unbased estmates, the best lnear unbased estmator (BLUE). Furthermore, f the effect estmates ^b from ndvdual subject analyses follow a Gaussan dstrbuton, the BLUE property can be extended to both lnear and nonlnear unbased estmates, based on the Cramér Rao nequalty. Such property gves the mpresson that the Studentzed t-statstc T S n (3) would lead to a statstcal power from MEMA hgher than or at least equal to the conventonal approach of gnorng the wthn-subject varablty. In practce, the true values of τ and σ are never known; thus, for each specfc test, T S may yeld a hgher or lower value than ts counterpart wth the conventonal approach wth Student t-test. However, the BLUE Also known as OLS estmate based t-statstc, e.g., n Mumford and Nchols (009) and Lndqust et al. (0). property ndcates that, on average, T S may provde a more powerful nference to an extent that depends on the combned mpact of wthn- and cross-subject varablty (Beckmann et al., 003) and on the presumed dstrbutons under whch the model fts the data. Another complcaton about T S s the determnaton of ts degrees of freedom, due to the uncertanty resultng from estmatng the wthnsubject varance σ. Varous approaches have been proposed for approxmatng the degrees of freedom, ncludng smply assgnng n-p- (Vechtbauer, 00), the Satterthwate correcton (Kebel et al., 003), estmaton through spatally smoothed rato of cross-subject varance and average wthn-subject varance (Worsley et al., 00), or posteror fttng wth a multvarate noncentral t-dstrbuton from MCMC smulatons (Woolrch et al., 004). Mumford and Nchols (009) showed that the estmate for effectve degrees of freedom based on Satterthwate approxmaton dd not perform well wth real and smulated data. Also, as a shortcut for MCMC samplng, the fast posteror approxmaton approach adopted n FLAME of FSL (Woolrch et al., 004), although presented under the Bayesan framework, s essentally equvalent to our REML soluton (0) because of the non-nformatve pror wth a unform dstrbuton. In addton, the sgnfcance-testng statstc mplemented n FLAME of FSL s bascally T S wth the same fxed degrees of freedom across the bran, n-p-. An approxmaton method proposed by Kenward and Roger (997) suggests nflatng the estmated varance and then adjustng the degrees of freedom through Satterthwate (946) correcton. Here we focus on provdng a more accurate estmate of varance for the effect estmate ^a than ^V ð^a Þ n (5). There are three sources of uncertanty that may contrbute to based estmate of ^V ð^a Þ:(a) unknown but estmated wthn-subject varance σ,(b) unknown but estmated crosssubject varance τ,and(c) truncaton practce n estmatng crosssubject varance τ, as shown n MOM (9), REML(0), and outler modelng wth ML (). The mpact of the frst two sources s unknown, but the thrd one would defntely lead to a postve bas. If an estmator s unbased, the possblty of resultng n a negatve estmate when the true τ =0 s 50% (Vechtbauer, 005). Thus the truncaton practce s expected to cause a postve bas n estmatng τ. The amount of bas decreases as the number of subjects, n, ncreases, or when the crosssubject varance becomes domnant. In other words, the bas s prevalent wth small number of subjects or wth a hgh rato of wthnsubject relatve to total varance. Usng a smple case of one-sample test, we obtan ^V ð^a Þ n Eq. (5) as P n ¼ ^τ þ ^σ!, a monotoncally ncreasng functon of ^τ, ndcatng that postve bas n estmatng τ would result n T S beng over-conservatve n controllng type I errors and under-powered n dentfyng actvated regons n the bran. Denote the mean sum of weghted least squares resduals as S ^W ¼ ^b T n p P^b, where ^b T P^b s the weghted resdual sum of squares (WRSS) for the WLS soluton (5), and P ¼ ^W = P W = ¼ ^W ^WX X X T T WX ^W. Relatve to (5), Knapp and Hartung (003) suggested an mproved estmator, ^V ð^a Þ ¼ S ^W X T ^WX ¼ ^b T, n p P^b X T ^WX wth the ntenton of usng the scale factor S ^W to counteract based estmate of ^V ð^a Þ n (5). Followng Vechtbauer (00), we generalze a t-statstc, proposed by Knapp and Hartung (003) wth the above mproved varance estmator ^V ð^a Þ nstead of the one n (5), to a new testng statstc for the null hypothess (), ^α T KH ¼ s ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff ^b T P^b X T ^WX : ð4þ n p Assumng a t-dstrbuton wth n-p- degrees of freedom, ths Studentzed statstc T KH n Eq. (4) has been shown to be more accurate than the jj

6 75 G. Chen et al. / NeuroImage 60 (0) Wald test and T S wth n-p- degrees of freedom (Knapp and Hartung, 003; Sdk and Jonkman, 005a). As ^b T P^b follows a χ (n-p-)-dstrbuton wth both mean and varance beng n-p- (Hartung et al., 008), the scalng factor S ^W n the denomnator of T KH canbesmallerorgreaterthan. As a result T KH could yeld values ether larger or smaller than T S n (3) wth n-p- degrees of freedom. Hartung et al. (008) recommended T KH for the followng two reasons: (a)a specfc choce of degrees of freedom for T S s controversal, and may render conservatve testng results (see Voxel 5 n Applcatons and results); and (b) ther smulatons showed that T KH was superor to T S n holdng the nomnal sgnfcance level. We wll also explore these two ssues later wth our own smulatons. Consder the two specal cases of wthn-subject varablty underlyng the summary statstcs approach to group analyss, n a onesample test n the model (3) wth only one explanatory varable (p=0 and X=(,,) T ): assumng neglgble wthn-subject varablty (σ τ, or σ 0, =,, n), or assumng the same wthn-subject varablty across all subjects,.e., σ = =σ n =σ (Penny and Holmes, 007). Snce the soluton (5) reduces to equal weghtng among the ndvdual effects, both T S and T KH reduce to the conventonal one-sample Student t-test (Appendx C). An extra statstcal nference capablty wth the MEMA model (3) s that we can test the null hypothess of homogenety across subjects, H 0 : τ ¼ 0 ð5þ under whch the model (3) reduces to the fxed-effects model (6). Null hypothess (5) can be tested by the homogenety statstc Q defned n (8) wth a quadratc χ (n-p-)-dstrbuton, often descrbed as Cochran's χ test (Vechtbauer, 00). If null hypothess (5) holds (the cross-subject varablty s neglgble), all the varance n the data comes from the wthn-subject varances, and the WLS soluton (5) corresponds to the fxed-effects model n Eq. (6). A regon n the bran where τ s sgnfcantly nonzero ndcates that there exsts some varablty or heterogenety across subjects, and warrants further exploraton when τ s very large (.e., much of the cross-subject heterogenety s left mproperly dentfed). Ideally, one would am to explan as much of the cross-subject varablty as possble wth subject groupng and/or covarates such as age, IQ, etc., untl the cross-subject random-effect component d can be dropped from the model (3) so that the fxed-effects model (6) would be approprate. However, dentfyng all the possble explanatory varables for the model (6) s rarely achevable n real practce, especally wth the massvely unvarate approach common n FMRI data analyss. On the other hand, Q-statstc provdes a vald approach to defnng a regon of nterest (ROI) that could be used to assocate ndvdual subject BOLD response wth some behavoral measure (Lndqust et al., 0), avodng the problematc practce of ROI defnton based on actvaton sgnfcance. One caveat about the Q-statstc s that t may become non-central n χ dstrbuton when the heterogenety s noteworthy,.e., some amount of cross-subject varablty s unaccounted for n the model (3). The non-centralty mpact on sgnfcance testng mght be relatvely small, but one potental mprovement s to use a mxture of χ dstrbutons as shown n Lndqust et al. (0). In addton to the homogenety Q-test (8), there are alternatve statstcs for null hypothess (5) such as lkelhood rato (LR) tests (Lndqust et al., 0), Wald test and Rao's score tests. Lndqust et al. (0) explored LR tests under three numercal solutons of crosssubject varance usng a mxture of χ dstrbutons, and elaborated on the challenge of approxmatng the asymptotc property of the LR tests. Vechtbauer (007) showed wth smulatons that the Q-test (8) has the best overall balance between type I error rate and power compared to the alternatves. For example, all the methods have comparable power n detectng heterogenety, but the Q-test keeps type I error rate close to the nomnal α-value (e.g., 0.05) when the number of wthnsubject data ponts s greater than 00, whle the LR tests tend to be over-conservatve n type I error control. Asde note heres thatthe fxed-effects model (5) can be appled to group analyss when there are only a few subjects or when summarzng the results from multple runs or sessons at ndvdual level. In the latter stuaton, the WLS soluton (5) s consdered better than the smple unweghted average that s wdely used (Lazar et al., 00) because the WLS method wth each weght equal to the recprocal of each run/sesson's or each subject's varance gves the BLUE for the group effect (Plackett, 950). For sngle-subject analyss methods that cannot combne multple magng runs, ths s the proper way to merge ntra-subject results pror to the group level, whch s better than smple averagng across runs or sessons that s currently practced n the FMRI communty. Quantfyng cross-subject varablty As a measure of cross-subject heterogenety, τ n the MEMA model (3) shows the extent to whch the subjects dffer from each other, but ts value and nterpretaton are not drectly comparable across studes because the effect magntude s ted up wth the factors n each specfc experment desgn such as task/condton, stmulus duraton, bran regons, etc. Smlarly, the Cochran's χ test, the Q-statstc defned n (8), s another measure of cross-subject heterogenety, but t depends on the number of subjects, as shown by ts expected value E(Q)=τ tr (P 0 )+n p. Due to these dependences, Hggns and Thompson (00) proposed two measures of heterogenety that, n addton to reflectng the amount of varablty across subjects, are ndependent of n and effect magntude (scale-free). Extendng the orgnal defnton for smple meta-analyss n Hggns and Thompson (00), weadopt the frst measure of heterogenety for our MEMA model (3), qffffffffffffffffffffffff Q H 0 ¼ n p. Alternatvely, we replace Q wth ts estmated expectaton value, ^τ trðp 0 Þþn p, and obtan a slghtly dfferent defnton, sffffffffffffffffffffffffffffffffffffffffffffffffffff H ¼ ^τ trðp 0 Þ n p þ : ð6þ The factor (n p )/tr(p 0 )n(6) measures the weghted average wthn-subject varablty, whch s self-evdent when no covarates exst (p=0) n the MEMA model (3). Because H= under the null hypothess (5), H can be nterpreted as the rato of standard devaton at group level and the weghted average standard devaton at ndvdual level; that s, H s an approxmate rato of confdence nterval wdths between the group and ndvdual subject levels, or between the MEMA model (3) and ts correspondng fxed-effects model (6). In other words, the varaton across the ndvdual effect estmates s H tmes what would be expected f cross-subject varablty dd not exst (Hggns and Thompson, 00). The second measure of heterogenety s defned as, I ¼ H ¼ H ^τ : ð7þ ^τ þ n p trðp 0 Þ Lke the popular concept of ntra-class correlaton (ICC), I accounts for the proporton of total varablty n the effect estmates that orgnates from the cross-subject rather than wthn-subject varablty. Accordng to Hggns and Thompson (00), anh value above.5 (I greater than 0.56) can be consdered to show sgnfcant heterogenety across subjects whle Hb. (I b0.3) should be of lttle concern. Identfyng outlers at regonal level Wth heterogeneous samplng varances ncorporated n the MEMA model (3), we not only obtan a more accurate statstcal testng, but also are able to estmate the heterogenety measure τ and test for the homogenety of subjects wth the Q-statstc (8).Furthermore,fwedefne ^σ λ ¼ ; ð8þ ^τ þ ^σ

7 G. Chen et al. / NeuroImage 60 (0) R L R L R L R L R L 5 (A) Student t (B) T KH +Gaussan (C) T S +Gaussan (D) T KH +Laplacan (E) FLAME + -5 Fg.. Sgnfcance maps of fve group analyss methods. The upper panel (Z=59) shows the vsual cortex actvatons n axal vew wth warm colors of z-score whle the lower panel (Z=74) ndcates the audtory actvatons n STS wth cold colors. One-taled sgnfcance level was set at 0.05 wthout cluster thresholdng. FLAME result (not shown here) s vrtually dentcal to 3dMEMA wth TS (3) and Gaussan assumpton (column C). λ can be nterpreted as the proporton of total varablty that comes from the th subject, and may be used to dentfy voxels or regons where a subject has exceptonally low relablty. Conversely, smlar to the heterogenety measures H and I, and lke the concept of ICC, λ ¼ ^τ ^τ þ ^σ provdes a thrd heterogenety measure that shows the proporton of total varablty that occurs across subjects. In addton, the followng Wald statstc h W ^b X^a P^b P^b O ¼ rffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff n h Var W ^b X^a o ¼ rffffffffffffffffffffffffffffffffffffffffffffffffffff h ¼ qffffffffffffffffffffffffffffffffffffffffffffffffff Var P^b P T W P ð9þ gves a sgnfcancetestforthenullhypothessabouttheresdualsofthe th subject (Vechtbauer, 00), H 0 : ^β x T ^a ¼ 0, or, ^δ þ ^ε ¼ 0, servng as another ndcator for voxels or regons where a subject has exceptonally hgh or low effect sze. Combnng the heterogenety measure ^τ,the homogenety Q-test (8),λ,andtheWaldtestO (9), one can detect outler regons or subjects, and further nvestgate the possblty of ncludng covarates or groupng subjects, potentally fne-tunng the orgnal model and ncreasng the statstcal power. Applcatons and results MEMA: Model performance wth real data Descrpton of the audovsual experment and the analyses Our group analyss modelng strategy was appled to the data from a block-desgn experment wth 0 subjects, descrbed at length as Experment n Nath and Beauchamp (0). A bref account of the data follows. Whole bran BOLD data were acqured on a 3.0 T scanner wth voxel sze of mm 3 and repetton tme (TR) of 05 ms. Three 5-mn scan runs were acqured for each subject, totalng 450 bran volumes. Two types of audovsual speech stmul were presented to the subjects. In the frst type, the vdeo mage was degraded, but the audtory content was not degraded, and vce versa for the second type. Each scan seres contaned fve blocks of audtory-relable and fve blocks of vsual-relable congruent words. Each 0- second block contaned ten trals, wth one dfferent word per tral lastng. to.8 s. Preprocessng steps ncluded slce tmng correcton, moton regstraton, voxel-wse mean scalng, and algnment to the Talarach standard space n mm 3 resoluton. Spatal smoothng was appled wth a kernel sze of 4 mm full wdth at half maxmum. The pre-processed data from each subject were concatenated across the three runs, and were analyzed wth an ARMA(, ) model for the resdual tme seres usng 3dREMLft. There are three approaches to handlng multple runs of data at ndvdual subject level: a) analyze each run separately; b) concatenate all runs but analyze the data wth separate regressors for an event type across runs; or c) concatenate all runs but analyze the data wth the same regressor for an event type across runs. Unlke other FMRI data analyss packages that adopts ether strategy a) or b), the nserton of a tme dscontnuty between runs/sessons n 3dREMLft also allows the nvestgator to analyze all the data from one subject n a sngle regresson wth all runs/sessons ncluded, whle stll modelng temporal Table Runtme (n mnutes) comparson a between MEMA and FLAME n FSL. Program 3dMEMA b FLAME FLAME + Outler modelng processor 4 processors Wthout Wth a Group analyss on a Mac OS X 0.6. wth.66 GHz dual-core Intel Xeon: 0 subjects, 8,379 voxels n mm 3 resoluton nsde the bran n Talarach standard space. b Runtme dfference between MEMA t-tests TS and TKH s neglgble.

8 754 G. Chen et al. / NeuroImage 60 (0) (A) T KH +Gaussan vs. Student t (A) T KH +Gaussan - Student t (B) T KH +Gaussan vs. T s +Gaussan (C) T KH +Gaussan vs. T KH +Laplacan (D) T KH +Gaussan vs. FLAME + (E) T S +Gaussan vs. FLAME (B) T KH +Gaussan - T S +Gaussan (C) T KH +Gaussan - T KH +Laplacan (D) T KH +Gaussan - FLAME (E) T S +Gaussan - FLAME Fg. 3. Scatterplots (left) and hstograms (rght) that compare the z-scores of sx group analyss methods. The shaded areas n scatterplots ndcate that both z-scores are below.645 (correspondng to one-taled sgnfcance level of 0.05). The data ponts on the y-axs n (D) and (E) are due to the fact that 3dMEMA allows mssng data whle FLAME n FSL does not. The hstograms show the correspondng z-score dfference to the scatterplots among the voxels not shaded (voxels wth mssng data were also excluded).

9 G. Chen et al. / NeuroImage 60 (0) correlatons (Appendx D). Opton c) could be mportant when the sample sze of an event type s relatvely small n a sngle run. Two regressors of nterest, audtory-relable and vsual-relable stmul, were created through convoluton between stmulus tmng wth a shapepresumed HDR functon (e.g., Cohen, 997). Sx head moton parameters were added n the model as regressors of no nterest. In addton, thrd order Legendre polynomals were ncluded to account for slow drfts n the data. The effect of nterest n the analyss was the contrast between audtory-relable and vsual-relable stmul. Group analyss was performed on ths contrast wth four dfferent methods: (a) Student t-test, (b) T S wth the assumpton of Gaussan dstrbuton for the cross-subject random effects, (c) T KH wth the assumpton of Gaussan dstrbuton for the cross-subject random effects, and (d) T KH n (4) wth the assumpton of Laplace dstrbuton of the cross-subject random effects. Trackng fve voxels Data at fve voxels (Fg. ) were extracted for demonstraton purposes. The results of Student t-test and several MEMA analyses are lsted n Appendx E. In summary, the cross-subject varablty s very small relatve to the wthn-subject varablty at all fve voxels. The conventonal approach mght render a lower or hgher group effect estmate (lower: Voxels, 3; hgher: Voxels, 4) as well as ts statstc value (lower: Voxels,, 3; hgher: Voxel 4) than the MEMA methods, dependng on the specfc nterplay of three factors, varyng precson, cross-subject varablty and the presence of outlers, as shown n the mpacts on the results at all fve voxels. The adjustment va the scalng factor n T KH does not nvolve the estmate of cross-subject varablty τ, whch remans the same between the two tests T S and T KH, but mght ncrease (Voxels, 4) or decrease (Voxels, 3) the t-statstc relatve to T S under the Gaussan assumpton, and the same holds under the Laplace assumpton (ncrease: Voxel 4; decrease: Voxels,, 3). The Laplace assumpton tends to estmate a smaller cross-subject varablty, especally when outlers are present (Voxels,, 3) than the Gaussan assumpton and the conventonal method, and mght provde hgher (Voxels, ) or lower (Voxel 3) statstcal values. The Q-statstc, defned n (8) for testng crosssubject varablty (null hypothess τ =0), depends on wthnsubject varances only; thus, ts value remans the same between the Gaussan and Laplace assumptons and between the two t-tests T S and T KH. In addton to the mproved accuracy n group effect estmates and sgnfcance testng compared to the conventonal approach, MEMA also provdes statstcal nference on the heterogenety τ across subjects, compares the two sources of data varablty, and asssts the nvestgator n dentfyng those subjects that have sgnfcantly outlyng effect estmates. To reterate, wth outler modelng combned wth adjusted t- test T KH, MEMA resulted n a hgher statstc power for voxels,, and 3, because effect estmates wth large varance were down-weghted and the use of Laplace dstrbuton accommodates better the presence of outlers. However, the conventonal method provded a hgher group effect estmate and the statstcal power n voxel 4 because subjects showng the largest effect also had the largest varance, thereby reducng ther contrbuton to the group effect estmate n MEMA compared to the Student t-test. Voxel 5 yelded smlar sgnfcance between Student t-test and MEMA when T KH s appled. Ths case demonstrates the mportance of the adjustment adopted n T KH : despte the large wthn-subject varance, the effect s deemed sgnfcant because t s consstent across subjects neglgble nter-subject varance (τ = 0); however, f only the precson nformaton s used n T S, then the statstcal power s lost. Comparsons among varous group analyss approaches As an emprcal comparson between our frequentst and a Bayesan mplementaton, we performed a smlar group analyss on the same datasets wth FLAME and FLAME + (Woolrch, 008) of FSL (verson 4..4). Sgnfcance maps are compared among sx group analyss approaches: Student t-test, three MEMA methods, FLAME and FLAME + (Fg. ). Results from Student and all MEMA t-tests were converted to z-scores for easy comparson wth FLAME n FSL. FLAME + wth and wthout the outler assumpton generated dentcal results. All sx methods rendered smlar onetaled sgnfcance map at the 0.05 level, especally for the two man regons of nterest, blateral superor temporal sulc (STS) for audtory functon (upper panel n Fg. ) and the vsual cortex (lower panel). The results from T S wth Gaussan assumpton and FLAME (not shown n Fg. ) were vrtually dentcal n sgnfcance map. Runtme comparson s shown n Table, and was markedly dfferent, wth MEMA beng smlar to FLAME, but 0 to 50 tmes faster than FLAME + at comparable settngs. The subtle dfference among the sx testng statstcs s more revealng n scatterplots and hstograms (Fg. 3). There are some small to large dfferences n z-scores between T KH and Student t-test (panel (A) n Fg. 3). Among the voxels where these two methods dffered by more than 0.5 n z-score, 63.% had hgher statstc value wth the MEMA test. The adjustment n T KH made a bg dfference relatve to ts Studentzed counterpart T S, resultng hgher statstc values n 85.9% of voxels (panel (B)). The dfference between Gaussan and Laplacan assumpton s relatvely small (panel (C)), ndcatng few outlers n the group. FLAME + gave some sgnfcantly dfferent results from T KH. Although the latter had hgher statstc values at 60.8% of voxels among those voxels that dffered by more than 0.5, FLAME + had extremely hgh statstc values at small proporton of voxels, also shown n the sgnfcance maps n (E) of Fg.. The equvalence between T S and FLAME s demonstrated n (E) of Fg. 3. The moderate dfferences between the two methods wth those voxels not sgnfcant (gray n (E)) at onesded level of 0.05 were due to the fact that, to save runtme for such voxels, 3dMEMA adopts MOM and avods the unnecessary REML teratons. Moreover, 3dMEMA has the flexblty to allow a small proporton of subjects to have mssng ndvdual subject t-statstcs at voxel level, as shown n those voxels on the y-axs n (D) and (E) of Fg. 3, whch also gves slghtly dfferent results than FLAME. In addton to provdng more accurate group effect estmates and sgnfcance testng, the MEMA modelng approach can also assess to what extent the subjects wthn a group dffer wth each other n terms of effect sze. 3dMEMA outputs three measures of such heterogenety: (a) the Q-statstc (8) measures the overall varablty wthn the group; (b) λ n (8) shows the percentage of total varablty that comes from the th subject; and (c) the Wald test (9) for each subject ndcates the sgnfcance level of how much the subject devates from the weghted average effect of the group. The results of the three measures for the experment data are shown n Fg. 4. The Q-statstc (Fg. 4A) ndcates that there was sgnfcant amount of varablty n the vsual cortex across the ten subjects whle moderate amount of heterogenety exsted n the STS area. Such heterogenety, measured wth τ, was partly due to the ntrnsc dfferences across subjects and partly due to the mperfect algnment from ndvdual brans to a template n standard space, and t s a dauntng job to tease apart these two components. The ICC-type measure λ (Fg. 4B) shows that the data varablty s domnated by wthn-subject varance, and that the percent of voxels wth the rato of cross-subject to total varance below 0.0, 0.0, 0.30 and 0.50 was 7.4%, 79.6%, 89.8%, and 95.5%, respectvely, among all voxels n the bran. The hstogram dstrbuton for those voxels wth one-taled sgnfcance level of 0.05 under T KH s not shown n Fg. 4B but s very smlar, and the percentage of voxels wth the rato of cross-subject to total varance below 0.0, 0.0, 0.30, and 0.50 was 75.8%, 8.7%, 88.9%, and 93.6%, respectvely. Consstent wth the heterogenety assessment of the Q-statstc at the group

10 756 G. Chen et al. / NeuroImage 60 (0) R L 80 Number of voxels Rato of cross-subject to total varance (A) Sgnfcance of -test (B) Hstogram of - for 0 subjects R L R L R L R L 4 Subject Subject 4 Subject 7 Subject 9-4 (C) Outlyng regons of ndvdual subjects Fg. 4. Outler detecton wth MEMA. (A) Homogenety of subjects (τ =0) under Gaussan dstrbuton assumpton for cross-subject random effects can be tested wth Q-test (8) wth a χ -dstrbuton. (B) Hstogram of cross-subject relatve to the total varance among,89,050 voxels (resoluton mm 3 ) n the brans of 0 subjects. The number of cells at the x-axs s 00 wth a resoluton of 0.0 for the varance rato. The cross-subject varance τ s estmated wth REML (4). (C) The Wald test O result for four subjects n outler dentfcaton s shown. In both (A) and (C) the upper panel (Z=59) shows the vsual cortex regon n axal vew whle the lower panel (Z=74) focuses on the STS. Onetaled sgnfcance level was set at 0.05 wthout cluster thresholdng. level, the Wald test from (9) shows more specfc outlers at the ndvdual subject level (Fg. 4C). For example, subject 7 was relatvely close to the group average n both vsual and audtory response, and so was subject 9 n audtory response. Subject mostly had sgnfcantly lower vsual response, whle the vsual response from subjects 4 and 9 was largely hgher than average. Smlarly, subject had lower response n the audtory regon STS, and subject 9 had hgher response. These Wald test results can assst the researcher n pnpontng those specfc subjects that may need further nvestgaton, ncludng algnment mprovement and ncorporatng auxlary varables that may account for such outlyng effects. MEMA: Model performance wth smulated data Descrpton of the smulatons Smulated data were generated to assess power and controllablty for type I errors n a much broader and more controlled spectrum than s possble wth the results from real data. We amed to compare varous testng statstcs from the followng three perspectves: sample sze n (number of subjects), heterogenety among wthn-subject varances (how dfferent are σ 's across subjects?), and the relatve rato of wthn- to cross-subject varance. Sx sgnfcance testng statstcs were consdered: Student t(n-p-), T S (n-p-) and T KH (n-p-) wth the