Are the Folded F-test and Bartlett s equivalent?

Transcription

1 Are the Folded F-test and Bartlett s equivalent? I can think of two ways to solve this problem: (1) Look up what they actually are. (2) Simulate under many conditions and look closely at the results. Looking at a UCLA website at the folded F test is defined by F = max(s 2 1, s 2 2)/ min(s 2 1, s 2 2) where si 2 refers to the sample variance for group i. For k groups, the Bartlett test statistic is T = (N k) log s2 p k i=1 (Ni 1) log si 2 ( k ) 1 3(k 1) i=1 N i 1 1 N k SAS Programming November 20, / 76

2 Are the Folded F-test and Bartlett s equivalent? As test statistics they look quite different, but it s more difficult to tell whether they will generate different p-values because. Bartlett s test is compared to a χ 2 distribution with k 1 degrees of freedom. The Folded F test is based on an F distribution. It is called folded because the larger variance is always placed in the numerator, thus making the F statistic constrained to be greater than or equal to 1. Why are the p-values so close, you ask? SAS Programming November 20, / 76

3 Are the Folded F-test and Bartlett s equivalent? The F test and the χ 2 distributions in that the F distribution has a d denominator and a numerator degrees of freedom and F m,n χ 2 m as n. The arrow means convergence in distribution, something slightly different from the usual limits you study in Calculus, but something you ll study in Casella and Berger. Basically this means that P(F m,n > x) P(χ 2 m > x) for large n, which also means that they generate very similar p-values, but they aren t equivalent for finite n, even if they are close. If you are graphing, they might appear to be identical due to limited resolution, so it is good to look at the numbers themselves to see how close they are. SAS Programming November 20, / 76

4 Are the Folded F-test and Bartlett s equivalent? How to investigate this with simulation? We might guess that as the sample size increases, then the tests are more likely to be similar, so we ll try with a small sample size. Unequal sample sizes might also have an effect, so I ll try three observations in one group and 10 in the other. Both definitions look like they easily generalize to more than two groups, for example you could use max(s 2 1,..., s 2 k )/ min(s2 1,..., s 2 k ) as a test statistic. If all variances are the same, this should still be close to 1. The distribution is now not clear, so you could either use theory to derive the distribution (might be hard!) or simulate the distribution SAS Programming November 20, / 76

5 Are the Folded F-test and Bartlett s equivalent? Let s simulate to see how different the p-values are for the Folded-F versus Bartlett s SAS Programming November 20, / 76

6 Are the Folded F-test and Bartlett s equivalent? Let s simulate to see how different the p-values are for the Folded-F versus Bartlett s SAS Programming November 20, / 76

7 Are the Folded F-test and Bartlett s equivalent? Let s simulate to see how different the p-values are for the Folded-F versus Bartlett s SAS Programming November 20, / 76

8 Are the Folded F-test and Bartlett s equivalent? SAS Programming November 20, / 76

9 Are the Folded F-test and Bartlett s equivalent? SAS Programming November 20, / 76

10 Are the Folded F-test and Bartlett s equivalent? So the type 1 error rate was different between the two tests. Note that although the Folded F was slightly elevated above 0.05, this was not significant since sqrt(.05.95/1000) =.042. I also tried equal samples sizes of 10 per group, and there the p-values disagree only in the 4th decimal place. In 1000 iterations, it wouldn t be surprising if they always agreed on whether or not to reject the null. Looking at the actual p-values instead of the power gives you more resolution for what is going on. SAS Programming November 20, / 76

11 Are the Folded F-test and Bartlett s equivalent? If I use 3 observations in each group, then the tests appear to be closer, but still not identical. SAS Programming November 20, / 76

12 HW hints: How to plot two identical curves on top of each other? Metallica recurrence data. SAS Programming November 20, / 76

13 HW hints Rolling Stones recurrence data. SAS Programming November 20, / 76

14 HW hints Megadeth recurrence data (much easier colors to read...) SAS Programming November 20, / 76

15 HW hints: how to plot two nearly identical curves on top of each other SAS Programming November 20, / 76

16 PROC IML! PROC IML! PROC IML! SAS/IML is the Interactive Matrix Language. It is probably the last topic of programming to cover in the course. After this, we will look at other topics using more statistical procedures from SAS. SAS/IML, like macros, can be used To Boldly Go Where No SAS Proc Has Gone Before...that is, it is partly used to give SAS programmers more flexibility and to implement new techniques that haven t found their way into SAS procedures yet. SAS Programming November 20, / 76

17 PROC IML! PROC IML! PROC IML! Another use for learning SAS/IML is to program with matrices is to improve your knowledge of statistics. This can be done with R or MATLAB just as (or perhaps more) easily than with SAS. It is often too difficult to work out multiple regression estimates by hand for instance, but instead of just relying on a canned procedure to get the answers for you, you can implement the matrices involved yourself and see if you can duplicate the results from programs like SAS or R or STATA. This can give you better appreciation for the methods and involved and deeper knowledge of what s going on under the hood of these programs. SAS Programming November 20, / 76

18 PROC IML basics We ll start by just doing some of the basics of how to use PROC IML. Despite the interactive title, it works pretty much like other SAS procedures, with typing in the Code editor and viewing the results in the Results viewer. The syntax for entering vectors and matrices is more similar to MATLAB than R. Here are some basic examples that show assigning values, vectors, and matrices and displaying their properties. Something that might be annoying is that you have to tell it PROC IML; reset log print each time you run it. If you just highlight your latest bit of code and run it, it will not know that you are using IML or know the previously defined variables. SAS Programming November 20, / 76

19 PROC IML basics SAS Programming November 20, / 76

20 PROC IML basics You can perform operations on matrices such as concatenation either horizontally or vertically... SAS Programming November 20, / 76

21 SAS Programming November 20, / 76

22 PROC IML basics The transpose of a vector or matrix is handled by the left single-quote, which looks a little weird. Some references say you can use the transpose function t(), which is how R does it, but it has generated errors for me. SAS Programming November 20, / 76

23 PROC IML basics You can create arrays of numbers similarly to R, and even arrays of strings that have consecutive suffixes. Try the following in the Code window and then look at the Results viewer. proc iml; reset log print; index=1:10; rows = 1:3 ; rindex=10:1; series(0,100,20); strings = "x1":"x8"; /* turns strings into a vector of, strings "x1", "x2",..., "x8" */ SAS Programming November 20, / 76

24 PROC IML basics Try these also to set up special matrices used often in statistics. proc iml; reset log print; b=j(6,1); *6x1 matrix of ones; c=j(2,3); *2x3 matrix of ones; a = I(6); *6x6 identity matrix; d = diag( {1 2 4} ); * diagonal matrix; b = 1:3; d2 = diag(b); d3 = diag(1:5); SAS Programming November 20, / 76

25 PROC IML basics SAS Programming November 20, / 76

26 PROC IML basics Some other functions and operators that can be useful: B = A##2; /* squares A elementwise */ C = A <> B; /* element-wise maximum */ C = A < B; /* element-wise 0-1 test */ B = sqrt(a); /* element-wise square root */ D = block(a,b,c); /* creates a block diagonal matrix using A, B, and C as matrices */ c = NCOL(A); /* number of columns of A */ d = NROW(A); /* number of rows of A */ E = exp(a); /* element-wise exponentiation */ F = log(a); /* element-wise log */ d = det(a); /* determinant */ B = inv(a); /* inverse of A */ v = vecdiag(a) /* v is a vector with the diagonal entries of A */ SAS Programming November 20, / 76

27 PROC IML basics You can extract individual elements or submatrices from a matrix: d1_12 = d1[1,2]; /* extact first row, second column */ d1_ = d1[1,]; /* extract first row */ d_2 = d1[,2]; /* extract second column */ d = d1[1:2,{1 3}]; /* extract a 2x2 submatrix */ mycols = {1 3}; d = d1[1:2,mycols]; /* alternative gives the same results */ Remember that to run this code you also need to run proc iml; reset log print; each time. SAS Programming November 20, / 76

28 PROC IML basics To get column sums and column sums of squares, there is special notation csum = d1[+,] #column sum is a row vector rsum = d1[,+] #row sum is a column vector rsum2 = d1 [,+] #rsum2 is a row vector a = d1[+,+]; /* sum of all elements in the matrix */ b = sum(d1); /* a and b are the same but b is more efficient */ SAS Programming November 20, / 76

29 PROC IML: searching matrices Often we want an index of where certain values occur. For example, we might want to know which values in a vector or a matrix are less than 0, or where the maximum value occurs. This can be done using the LOC function. The LOC function is easier to understand with vectors. Here it returns a vector of indices of the vector satisfying the condition. For example, with d 2 = { 4 0 4}, trying m = max(d_2); negvalue = loc(d_2<0); maxvalue = loc(d_2=m); returns the vector {1 3} for negvalue and returns {2} for maxvalue. The LOC function is very similar to the which() function in R. SAS Programming November 20, / 76

30 PROC IML searching matrices If you apply the LOC function to an entire r c, it counts the cells from 1 to r c and gives a single number indexing the cells rather than the row and column number, and counts cell row-wise. Thus, for a 2 3 matrix like d1, d1[5] extracts element (2,2). SAS Programming November 20, / 76

31 PROC IML: missing values Matrices can be defined with missing values, which will sometimes appropriately create missing values when operated on, and other times missing values are ignored. SAS Programming November 20, / 76

32 PROC IML: formatting and labels You can format values to have dollar signs, commas or any other format you like and can also label rows and columns. SAS Programming November 20, / 76

33 PROC IML: Reading data to and from SAS data sets Typically, you ll want to read in data from a SAS data set instead of starting from scratch and entering data into your matrix. You also might want to manipulate data using PROC IML then output it again to a SAS data set to use usual SAS PROCs. SAS Programming November 20, / 76

34 PROC IML: Creating a SAS data set Typically, you ll want to read in data from a SAS data set instead of starting from scratch and entering data into your matrix. You also might want to manipulate data using PROC IML then output it again to a SAS data set to use usual SAS PROCs. SAS Programming November 20, / 76

35 PROC IML: Creating a SAS data set Note that I ve changed the variable name from temperature to temp and also changed the 5th observation from age 73 to age 67. I needed the three statements, CREATE, APPEND, and CLOSE to create the data set. If you don t CLOSE, then the data set is created but is still empty. Instead of creating a new data set, you can instead edit an existing data set using the EDIT statement. However, you can only input one data set at a time into PROC IML. SAS Programming November 20, / 76

36 PROC IML: more on reading in data You can choose to read in less than all of the data in a data set. The syntax for the READ statement is READ <range> <VAR operand> <WHERE (expression)> <INTO name <[rowname=variable colname=matrix]>>; where items within angled brackets are options. Typically for the range you put all if you want to read in all of the data. You can also read in the next n observations using NEXT n, or a list of specific observations using point {3,10,11} (for example to read in the 3rd, 10th, and 11th observations only. SAS Programming November 20, / 76

37 PROC IML: more on reading in data VAR allows you to specify a list of which variables to read in if you don t want to read them all in. The default is to only read in numeric variables, however you can use CHARACTER to specify that you only want to read in character variables. The WHERE option works similarly to WHERE statements in data steps in PROCs. A common example might be to only read in observations without missing data, e.g. where var1 ne.. An alternative to using the READ statement to control what is read in is to create a new data set using data steps to first create the subset of the data you want to manipulate in PROC IML, although this could be less efficient than going directly through PROC IML for large data sets SAS Programming November 20, / 76

38 PROC IML: logical expressions, loops Logical expressions using IF THEN, ELSE, and loops using DO follow the same syntax as in data steps. They allow you to loop through a matrix in a way that can be a bit easier than looping through a data set in a data step. If your loop is indexing observation (row) i, then you can examine row i 1 without having to use lag functions and retain statements. You also have access to observation i + 1. SAS Programming November 20, / 76

39 PROC IML: simple linear regression As an exercise, let s use PROC IML to find the regression coefficients when we do a simple linear regression of temperature on age. For now, we ll ignore the sex of the individuals. We want to use the model Y = X β + e, or y i = β 0 + β 1 x i + e i, i = 1,..., 130 First, we ll read in the data, separate the data into observations Y, and the predictor, and create the design matrix. The design matrix will have a column of 1s and a column for the predictors. Note that X is and the vector of coefficients in the regression model is β = (β 0 β 1 ), which is a 2 1 matrix (or column vector). If we set Y = X β + e, where we use these matrix expressions, then the ith row of Y is y i = x i1 β 0 + x i2 β 1 + e i. SAS Programming November 20, / 76

40 PROC IML: simple linear regression The idea is to solve for β using the equation Y = X β. To do this we first multiply both sides by the transpose of X, so X Y = X X β (X X ) 1 X Y = (X X ) 1 X X β β = (X X ) 1 X Y Now we can use PROC IML to do these matrix calculations and see if they result in β that matches the output of SAS procedures. SAS Programming November 20, / 76

41 PROC IML: simple linear regression Typically, you ll want to read in data from a SAS data set instead of starting from scratch and entering data into your matrix. You also might want to manipulate data using PROC IML then output it again to a SAS data set to use usual SAS PROCs. SAS Programming November 20, / 76

42 PROC IML: simple linear regression output from PROC IML and PROC REG SAS Programming November 20, / 76

43 PROC IML: simple linear regression How do you interpret the matrix X X? Think about how to multiply this. The matrix is p p where p is the number of parameters (β terms), so it is 2 2 in this case. The (1,1) entry is times, so it is 130. You can also think of this as the sample size. The (1,2) entry is x x x 130 = The (2,1) entry is The (2,2) entry is i=1 130 x x x = i=1 130 x 1 x 1 + x 2 x 2 + x 130 x 130 = i=1 x i x i x 2 i SAS Programming November 20, / 76

44 PROC REG You can also output the X X matrix from PROC REG using model temperature = age / xpx; which is an abbreviation for x prime x. Of course, PROC REG outputs much more than this, and you can use PROC IML to see if you can reproduce residuals, fitted values, F statistics, and so on. SAS Programming November 20, / 76

45 Getting other quantities for regression Other quantities of interest for a linear model include the hat matrix, X (X X ) 1 X, the predicted values, X X etc., all of which can easily be obtained from PROC IML. Some of these can also be obtained from PROC REG. If a new diagnostic test is developed not implemented in PROC REG or PROG GLM, then it would be beneficial to have access to these matrices directly from PROC IML. SAS Programming November 20, / 76

46 Comparing shapes An example where you might need something like PROC IML is if you have to rotate your data, which can be accomplished through matrix multiplication of your original data. This comes up in statistics when you want to compare to sets of points (usually either in 2D or 3D, but could be higher-dimensional), and the points are not oriented the same way or scaled the same way. This can come up particularly in shape analysis, where you want to determine whether two shapes are roughly equivalent, or you want to compare two photographs taken from slightly different positions. In addition to statistical testing, sometimes you just want to best visualize the difference between two sets of points, and this is best accomplished by lining up the points as nearly as possible. SAS Programming November 20, / 76

47 Comparing shapes As an example, consider two photographs of hands. SAS Programming November 20, / 76

48 Comparing shapes We have reference points on the hands, and we want to line up the reference points as closely as possible by rotating the images, rescaling if necessary (suppose you have photos that are cropped or zoomed in and still want to compare the shapes). In general we might also allow mirror images. In this case, we assume the points are two-dimensional so that they each have just an x and y coordinate. This would typically be the case for analyzing photographic images, although in general you can imagine have three dimensional data as well. SAS Programming November 20, / 76

49 Comparing shapes Applications for this are widespread. In medical imaging, you might take an x-ray of a patient over time to compare how their spine is changing with osteoporosis. The x-rays won t be taken at identical distances, angles and so forth, so you need to align the images by stretching and rotating. Other examples will include MRI scans of the brains, where you might want to either compare the same individual at different time points, the left versus the right hemisphere to look for asymmetries, or two separate individuals to see how closely aligned two brains are. Here we want to ignore the fact that one brain might be slightly larger than the other. If eyes or fingerprints are used for ID, again it will be easiest to compare two images by rotation and rescaling. SAS Programming November 20, / 76

50 Comparing shapes If you have satellite images of regions on earth, you might want to measure things like habitat loss. Successive photos of the same region won t be exactly the same, so you might try to align two photos using certain geographical reference points. Once the photos are aligned, you can use differences in the area that is green as a measure of vegetation loss, for example. The name Procrustes refers to a bandit from Greek mythology who made his victims fit his bed either by stretching their limbs or cutting them off. (Wikipedia) SAS Programming November 20, / 76

51 Procrustes illustrations You can find some interesting illustrations online... SAS Programming November 20, / 76

52 Comparing shapes Back to the hand example. Here we have two sets of coordinates. We might call them and X = {(x 11, x 12 ), (x 21, x 22 ),..., (x n1, x n2 )} Y = {(y 11, y 12 ),..., (y n1, y n2 )} How should we align the points? If we use the distances between corresponding points, we can minimize the distance between points over all possible angles of rotation, rescalings, and reflections. To deal with reflections, it might help to center the points so that (x 1, x 2 ) = 0 and (y 1, y 2 ) = 0. For many problems (like with satellite photographs or the same patient over time), reflections won t matter. SAS Programming November 20, / 76

53 Comparing shapes The distance between two individual points x i = (x i1, x i2 ) and y i = (y i1, y i2 ) is naturally defined as d(x i, y i ) = (x i1 y i1 ) 2 + (x i2 y i2 ) 2 This is the Euclidean distance between two points in the plane. We might define the overall squared distance from the set of points X to the set of points Y as n d 2 (X, Y ) = d 2 (x i, y i ) i=1 SAS Programming November 20, / 76

54 Comparing shapes We can then minimize the sum of squared distances between points (this is equivalent to minimizing the distances why?). This is fairly similar as a criterion to what we do in regression, so hopefully doesn t seem too weird. In other words, we want to minimize d 2 (x i, y i ) = (x i1 y i1) 2 + (x i2 y i2) 2 over all choices of θ and c. To do this, we need to write y i as a function of y i, θ, and a scaling factor c. SAS Programming November 20, / 76

55 Example SAS Programming November 20, / 76

56 Example Lines connecting corresponding points SAS Programming November 20, / 76

57 Example Rotating π/8 radians = 22.5 degrees clockwise, we get SAS Programming November 20, / 76

58 Example How the squares were shifted by π/8 radians = 22.5 degrees SAS Programming November 20, / 76

59 Example Shifting by another π/8 radians (45 degrees total), we get SAS Programming November 20, / 76

60 Rotating your data To rotate 2-dimensional data, you can use a rotation matrix. If a set of points is in an n 2 matrix, then we need a 2 2 matrix to multiply this matrix. The rotation matrix (from linear algebra) is [ ] cos θ sin θ R = sin θ cos θ SAS Programming November 20, / 76

61 Minimizing squared distances Let X be the matrix for the first data set and Y the matrix for the second data set. To simplify the problem, we ll consider only doing optimal rotations without worrying about scaling. We can think of rotating the observations Y to match X using a rotation matrix R. Thus X = RY This usually can t be solved exactly, so we want to find R for which X RY 0 To minimize the sum of squared distances, we minimize tr[(x RY ) (X RY )] where tr is the trace, or sum of the diagonals. After some matrix algebra, this is equivalent to minimizing tr(ry X ) SAS Programming November 20, / 76

62 Minimizing squared distances A technique for solving this problem involves the singular value decomposition, which again comes from linear algebra and which we won t review, but can be used in SAS, Matlab, and other matrix-oriented languages. However, the matrix Y X can be decomposed into UDV, where D is diagonal and U and V are orthogonal. The solution is R = VU which minimizes the sum of the squared distances. If you have software which can do the singular value decomposition, then you can use it get the optimal rotation. SAS Programming November 20, / 76

63 Using PROC IML for Procrustes Rotation SAS Programming November 20, / 76

64 Using PROC IML for Procrustes Rotation SAS Programming November 20, / 76

65 Using PROC IML for Procrustes Rotation SAS Programming November 20, / 76

66 Using PROC IML for Procrustes Rotation SAS Programming November 20, / 76

67 Using PROC IML for Procrustes Rotation In this case, the rotation matrix has values near 1 and -1 for sin θ and sin θ for the (1,2) and (2,1) entries, respectively, suggesting that the optimal rotation is near 90 degrees or π/2 radians. This means 90 degrees counterclockwise, and if you look at the photos, moving the right-hand photo 90 degrees counterclockwise will indeed line up the wrists (from top left to bottom left) and the rest of the hand. Since these photos look identical, the slight discrepancy might be due to truncating measurements in the positions of the points. SAS Programming November 20, / 76

68 Using PROC IML for Procrustes Rotation Most of the work of the Procrustes rotation in terms of code was for centering the data. Once the data was centered, there were only three lines of code needed. It is possible to do the optimal rotation without using matrices, but this would involve more work in terms of coding. For the centering itself, this could have been done outside of PROC IML, and PROC IML could have read in a SAS dataset with X and Y already centered. If you had the original data in a SAS data set, how would you center the data using data step programming? SAS Programming November 20, / 76

69 Centering the data Often if something is very tedious, there might be a procedure to help you out. Googling z-score SAS quickly reveals PROC STANDARD, which can center your data. If you have a data set called temperature, you can use something like proc standard data=temperature mean=0 std=1 out=ztemp; var degrees; run; SAS Programming November 20, / 76

70 Minimizing squared distances using calculus To minimize this distance, we can use calculus, but in this case, we need to minimize over rotations (angles) and rescalings. It helps to think of one data set as fixed, say X, and we rotate and rescale Y to match X as closely as possible. If we rotate the Y data by an angle θ and stretch their values by c, then [ cos θ sin θ sin θ cos θ ] [ cyi1 cy i2 ] = c [ yi1 cos θ y i2 sin θ y i1 sin θ + y i2 cos θ So we use y i1 = c(y i1 cos θ y i2 sin θ) and y i2 = cy i1 sin θ + cy i2 cos θ and and plug these values into n d 2 (X, Y ) = (x i1 y i1) 2 + (x i2 y i2) 2 i=1 Then the idea is to minimize with respect to θ and c. This can be done taking partial derivates with respect to θ and c and using the usual calculus techniques of optimization. SAS Programming November 20, / 76 ]

71 The calculus approach I ll give the formulas in terms of summations for the optimal values for θ and c. This will be equivalent whether you use the calculus approach or the matrix approach. The optimal rotation angle is for integer k, and θ = arctan D B + kπ c = B A cos θ + D A sin θ where k {0, 1} should be chosen to let c > 0. Here A = n i=1 y 2 i1 + y 2 i2 B = C = n x i1 y i1 + x i2 y i2 i=1 n x i1 y i1 x i2 y i2 i=1 SAS Programming November 20, / 76

72 Using PROC IML for Procrustes Rotation Part of the advantage of the matrix approach to Procrustes rotations is that it generalizes more easily than the calculus approach. In particular, you can compare three-dimensional (or higher dimensional) shapes or patterns in the data as well as two-dimensional, which makes the rotations more complicated. We just illustrated rotations rather than scaling. Optimizing the scaling factor as well as rotations is sometimes called extended Procrustes Analysis. Generalized Procrustes also more than two data sets at a time to be used. For Generalized Procrustes, and mean shape is the set of points that minimizes the sum of the Procrustes distances to each of the input data sets. SAS Programming November 20, / 76

73 Comparing data sets and Outlier detection In addition to rotating your data, Procrustes rotations give you a way to quantify how different two shapes are. You might or might not want to rescale (stretch) the data depending on the application, for example using the minimum sum of squared distances. Thus, given three data sets, you can look at all pairwise distances to determine which two datasets are most similar. The Procrustes rotation can also be used to look for outliers. Try removing one observation at a time and recomputing the squared distances each time. (This means you will do Procrustes rotations with n 1 instead of n data points each time.) This gives you a measure of which observations have the biggest effect for one dataset not being able to be rotated to match the other data set. SAS Programming November 20, / 76

74 Writing functions in PROC IML SAS Programming November 20, / 76

75 Writing functions in PROC IML SAS Programming November 20, / 76

76 Writing functions in PROC IML SAS Programming November 20, / 76