Macros and ODS. SAS Programming November 6, / 89

Transcription

1 Macros and ODS The first part of these slides overlaps with last week a fair bit, but it doesn t hurt to review as this code might be a little harder to follow. SAS Programming November 6, / 89

2 ODS in SAS studio We re now ready to try combining macros with ODS. Here is an example of a simulation study that we can do with what we have. Suppose X 1,..., X 10 are i.i.d. (independent and identically distributed) exponential random variables with rate 1. If you perform a t-test at the α = 0.05 level for whether or not the mean is 1, what is the type 1 error? If X 1,..., X 10 are normal with mean 1, standard deviation σ, then you expect that you will reject H 0 5% of the time when H 0 is true. In this case, H 0 is true (the mean is 1), but the assumption of normality in the t-test is violated, and this might effect the type 1 error rate. SAS Programming November 6, / 89

3 ODS in SAS studio First we ll do one data set and run PROC TTEST once with TRACE ON to figure out what table we want to save. SAS Programming November 6, / 89

4 ODS in SAS studio It looks like we want the third table, which was called TTests SAS Programming November 6, / 89

5 ODS in SAS studio If you look at the data set in Work Folder, the name of the variable with the p-value is again Probt, although when PROC TTEST runs, it labels the variable by Pr > t. SAS Programming November 6, / 89

6 ODS in a macro Here s a start, I run PROC TTEST 3 times on 3 generated data sets. This creates 3 small data sets with p-values. SAS Programming November 6, / 89

7 ODS in a macro: using concatenation merge to put all p-values in one data set SAS Programming November 6, / 89

8 ODS dataset of p-values You should be able to now use your dataset of p-values to analyze your p-values. Particular questions of interest would be (1) how many p-values are below 0.05 (This is the type I error rate), and (2) the distribution of the p-values. SAS Programming November 6, / 89

9 ODS dataset of p-values In my case, I had trouble doing anything directly with the dataset pvalues. Nothing seemed to print in my output (for example PROC MEANS). So, in my case, I just output my dataset pvalues to an external file and then read it in again using a new SAS program. This is slightly inelegant, but it means I can start from scratch in case any of my ODS settings changed things and caused problems.// This approach could also be useful in case I wanted to generate p-values in SAS and analyze them or plot them in another program like R, or if I just want to save those p-values for later reference. SAS Programming November 6, / 89

10 ODS dataset of p-values SAS Programming November 6, / 89

11 ODS dataset of p-values Here I analyze the output of a SAS program using a second SAS program. pvalues2.txt is a cleaned up version of pvalues.txt that removed header information and so forth. SAS Programming November 6, / 89

12 ODS dataset of p-values Note that the MEANS procedure calculated the mean of the 0s and 1s and got This means there were 84 (out of 1000) observations that had a p-value less than How did it get the standard deviation? Note that each observation is a Bernoulli trial, and the standard deviation of a Bernoulli trial is p(1 p), so we would estimate this to be.084(1.084) = Why is this (slightly) different from the standard deviation reported by PROC MEANS? Also, is there evidence that there is an inflated type I error? Is significantly higher than α = 0.05? SAS Programming November 6, / 89

13 Statistical inference and simulations Sometimes we find ourselves using statistical inference just to interpret our own simulations, rather than for interpreting data. Some scientists have the attitude that if a phenomenon is real, then you shouldn t need to statistics to see it in the data. Although I don t share this point of view, because a lot of data is too complicated to interpret by eye, I sort of feel this way with simulations, though. If you re not sure whether is significantly higher than 0.05 (meaning there really is inflated type 1 error), you could either get a confidence interval around 0.084, or you could just do a larger simulation so that you could be really sure without having to construct confidence intervals. In this case, the confidence interval does exclude 0.05, so there is evidence to think that the type 1 error rate is somewhat inflated due to violation of the assumption of normally distributed samples. SAS Programming November 6, / 89

14 What about the distribution of p-values? The p-value is a random variable. It is a function of your data, much like X and σ 2, so it is a sample statistic. What should the distribution of the p-value be under the null hypothesis for a test statistic? If you use α = 0.05, this means that 5% of the time the p-value should be below α. More generally, for any α, P(p-value < α) = α. The p-value therefore has the same CDF as a uniform random variable. So p-values should be uniformly distributed for appropriate statistical tests when the null hypothesis and all assumptions are true. This is true for tests based on continuous test-statistics. For discrete problems, it might not be possible for P(p-value < α) = α. SAS Programming November 6, / 89

15 The distribution of p-values Here is a more technical explanation of why p-values are uniformly distributed for continuous test statistics, when the null is true, and for hypotheses H 0 : µ = µ 0, H A : µ > µ 0 (i.e., I ll just consider a one-sided test). For this one-sided test, the p-value is P(T t), where T is the test-statistic. 1 F (t) = P(T > t) = P(F (T ) > F (t)) = 1 P(F (T ) F (t)) P(F (T ) F (t)) = F (t) Because 0 F (t) 1, this means that F (T ) has a uniform distribution (since it has the same CDF). If U is uniform(0,1), then so is 1 U, so 1 F (t) is also uniform(0,1). But note that 1 F (t) = P(T t), which is the p-value. SAS Programming November 6, / 89

16 ODS dataset of p-values Here is the distribution of the p-values represented by a histogram. Typically uniform distributions have flatter looking histograms with 1000 observations, so the p-values here do not look uniformly distributed. Again, this would be clearer if we did more than just 1000 simulations. P- value SAS Programming November 6, / 89

17 A different way to simulate this problem Instead of simulating 1000 data sets of 10 observations, I could have just simulated all of the data all once and indexed the 1000 sets of 10 observations (similar to what I did for the Central Limit Theorem example. In this case, I would want to use PROC TTEST separately on each of the 1000 experiments. Moral: There s more than one way to do things. SAS Programming November 6, / 89

18 PROC TTEST using a BY statement P- value SAS Programming November 6, / 89

19 SAS Programming November44rr , / 89 PROC TTEST using a BY statement P- value

20 How to do this in R? It s a little easier in R. Here s how I would do it: x <- rexp(10000) x <- matrix(x,ncol=1000) pvalue <- 1:1000 for(j in 1:1000) { pvalue[j] <- t.test(x[,j],mu=1)$p.value } sum(pvalue<=.05)/1000 # this is the type I error rate hist(pvalue) SAS Programming November 6, / 89

21 Another example is a test for homogeneity of variances. Textbooks often warn that Bartlett s test can be used to test for equality of variances, but that it is extremely sensitive to the assumption of normality, even though many procedures, such as t-tests and ANOVA are reasonably robust to assumptions of normality. It is instructive to do a simulation to find out just how sensitive Bartlett s test is to the assumption of normality. In this example, we ll again create samples from two independent exponential distributions with rate λ = 1, so that both have equal variances of size λ 2 = 1. This time, we ll let the sample size vary from n = 10, 20, 30,..., 100 and see how the test does with increasing sample sizes. Again we ll look at the type I error rate for the test. For t-tests, we expect that as the sample size increases, the Central Limit Theorem tells us that X n is becoming increasingly similar to a normal distribution, so we expect the type I error rate to improve (get closer to α) as n increases. The Central Limit Theorem doesn t apply to S 2, the estimate of the variance, so the result of increasing n isn t as clear here. SAS Programming November 6, / 89

22 Testing type 1 error for Bartlett s test There are different ways to test for homogeneity of variance in SAS depending on the procedure that you are using. To get a statistic for Bartlett s test, you can use PROC GLM, which can use a two-sample case as a special case, although PROC GLM is much more general. PROC GLM can also be used for doing ANOVA, including with unbalanced designs (PROC ANOVA is for balanced designs), MANOVA (multivariate ANOVA with multiple response variables as well as multiple independent variables), polynomial regression, random effects models, repeated measures, etc. SAS Programming November 6, / 89

23 Testing type 1 error for Bartlett s test First, we ll generate example data, keeping in mind that we want to generalize our parameters. SAS Programming November 6, / 89

24 Testing type 1 error for Bartlett s test Note that PROC GLM wants data in the narrow style, NOT two columns, one for group A, one for group B. The data doesn t have to be sorted by group. SAS Programming November 6, / 89

25 Testing type 1 error for Bartlett s test Note that we could have generated the data in a wide format or with group A generated first followed by group B. To generate in a wide format, we could have done this with only one output statement and different variables for the two groups. data sim; do i=1 to &iter; do j=1 to &n; x = ranexp(2014*&n + &iter); y = ranexp(2013*&n + &iter); output; end; end; keep group x y i; run; SAS Programming November 6, / 89

26 Testing type 1 error for Bartlett s test To generate the As first then the Bs, we could have done this instead with an extra do loop, which still generates a narrow data set. data sim; /* generate two exponentials for each combination of i and j */ do i=1 to &iter; do j=1 to &n; x = ranexp(2014*&n + &iter); group = "A"; output; end; do j=1 to &n; x = ranexp(2013*&n + &iter); group = "B"; output; end; end; /* i is the iteration */ SAS Programming November 6, / 89

27 Testing type 1 error for Bartlett s test Back to the original data. We ll look at the output from PROC GLM and PROC TTEST to compare them. SAS Programming November 6, / 89

28 Testing type 1 error for Bartlett s test Here s output from PROC GLM. SAS Programming November 6, / 89

29 Testing type 1 error for Bartlett s test SAS Programming November 6, / 89

30 Testing type 1 error for Bartlett s test By default, PROC TTEST does a test for equal variances using the Folded F-test, whereas PROC GLM does not. Note that the p-value from PROC GLM matches the p-value from PROC TTEST when equal variances are assumed. SAS Programming November 6, / 89

31 Testing type 1 error for Bartlett s test Put the trace on to figure out how to save the right table. SAS Programming November 6, / 89

32 Testing type 1 error for Bartlett s test Look in the log file for the table. SAS Programming November 6, / 89

33 Testing type 1 error for Bartlett s test SAS Programming November 6, / 89

34 Testing type 1 error for Bartlett s test Now we can extend to more iterations and use BY to get them in one data set. SAS Programming November 6, / 89

35 Testing type 1 error for Bartlett s test SAS Programming November 6, / 89

36 Testing type 1 error for Bartlett s test We can scale this up to as many iterations as we want. Then we want to keep track of the number of p-values below 0.05 to get the type 1 error rate. SAS Programming November 6, / 89

37 Testing type 1 error for Bartlett s test SAS Programming November 6, / 89

38 Testing type 1 error for Bartlett s test Now we want to repeat this same idea but for different sample sizes n. Of course we could just repeat the code over and over and over again, changing the value of n. Or we can loop over different values of n. This creates a 3-level loop instead of 2-levels. SAS Programming November 6, / 89

39 Testing type 1 error for Bartlett s test SAS Programming November 6, / 89

40 Testing type 1 error for Bartlett s test SAS Programming November 6, / 89

41 DO loops versus Macros Instead of having an additional DO loop, I could have created a macro, say %macro bartlett(n,iter) and then run the macro multiple times for different values of n: %bartlett(10,1000) %bartlett(20,1000) %bartlett(30,1000) And so on. If you re data step is getting too complicated, then this might be reasonable. Also if you want different combinations of parameters, the macro approach is more flexible. For example, if you want 1 million iterations for n = 10 but 1000 iterations for n = 100 due to time constraints, then the macro approach is more flexible. SAS Programming November 6, / 89

42 Suppressing output? Unfortunately, these simulations create a lot of output as they are. If you want to run a procedure to create an output data set, but not generate output, then you can do this for most procedures, for example using the NOPRINT option in the same line as the procedure, but still creating an output data set, such as OUTPUT out= in PROC MEANS. Unfortunately, the NOPRINT option in the procedure means that nothing is printed for ODS to use. As a result, when using ODS, you end up with lots of output. I m not sure of a good way around this, but it is pretty annoying and slows SAS down to do a lot of I/O. You can reduce the output by only selecting what you will need to save: SAS Programming November 6, / 89

43 Testing type 1 error for Bartlett s test SAS Programming November 6, / 89

44 Results SAS Programming November 6, / 89

45 Results: interpretation These results are based on 10,000 iterations. Since we are generating essentially Bernoulli random variables (reject or don t reject H 0 ), we can think of the mean of the as the proportion of rejections. This should be α = 0.05, but is higher, with 23% for n = 10, 27% for n = 20, and 29% for n = 50. For 10,000 iterations, a confidence interval for this proportions has a margin of error of roughly 2 (.25)(.75)/10000 = A handy rule of thumb is that for a binomial, the 95% confidence interval has margin of error less than or equal to 1/ n = 1/100 for n = 10, 000, the point being that 29% appears to be significantly larger than 27%, so that type 1 error rates are increasing as the sample size increases. SAS Programming November 6, / 89

46 The moral of the story The moral of the story is that increasing your sample size doesn t always improve your inferences. In this case, the method is sufficiently non-robust that increasing the sample size makes it perform worse. So when textbooks say that Bartlett s test isn t very good for testing equality of variances, they really mean it, although they rarely explain why it is so bad. So what exactly is Bartlett s test testing? The usual description is that it tests H 0 : σ1 2 = σ2 2 assuming that the two samples are from normally distributed populations. However, considering that the test is likely to reject H 0 when σ1 2 = σ2 2 but the data are not normal, you could instead think of the null hypothesis as H 0 : X 1,..., X n iid N(µ1, σ 2 ), Y 1,..., Y m iid N(µ2, σ 2 ) i.e., the normality is part of what is being tested. In this case a rejection of H 0 could mean either that the data are not normal or that variances are unequal. SAS Programming November 6, / 89

47 Statistical inconsistency A related issue regarding increasing sample sizes is statistical inconsistency. An estimator θ n for a parameter θ (which might be an ordered-tuple of parameters), is said to be statistically consistent if for any ɛ > 0 and for any θ Θ (the parameter space), lim P( θ n θ > ɛ) = 0 n where n is the sample size. In other words, the estimator gets close to the actual parameter with high probability. You can have estimators that don t have this property, so that increasing the sample size doesn t increase the probability of your estimate being close to the true value. If you work with a discrete parameter space, then statistical consistency requires that the probability approaches 1 of making the correct inference for the parameter. SAS Programming November 6, / 89

48 Statistical power So far, we ve focused on type 1 error. How about power? Power is nearly identical from the point of view of simulation. In this case, you simulate for some values for which H 0 is false, and again count the number of times the null is rejected. In this case, the more frequently H 0 is rejected, the better the method (assuming it has good type 1 error as well). As an example, we ll consider the power for testing H 0 : µ 1 = µ 2 in a t-test when X 1..., X n iid N(0, 1), Y1,..., Y n iid N(1, 1). In this case the variances are equal, but the means are different. SAS Programming November 6, / 89

49 Statistical Power SAS Programming November 6, / 89

50 Statistical Power SAS Programming November 6, / 89

51 Statistical Power Power of the t-test for rejecting H 0 when X s are i.i.d N(0,1) and Y s are i.i.d N(1,1). 1 SAS Programming November 6, / 89

52 Uses of Power Analyses Why is it useful to study power? The main reasons for studying power for a particular problem are sample size determination for study design determining the effect size detectable for a given sample size choosing between different methods investigating robustness of methods to model violation SAS Programming November 6, / 89

53 Power: sample size determination A power analysis can be useful for determining the sample size you need to have a good chance of rejecting H 0 when H 0 is false. This is useful for initially trying to decide what sort of sample size to aim for when designing a study, and can be useful in grant applications. In the previous t-test example, if we believe that treatment 2 results in values an average of 1 point higher than treatment 1, then we can estimate that we d need a sample size of roughly 20 people per group to have about 80% power to detect a difference. If you wanted a better than 80% chance of being able to reject H 0, you d want larger samples. SAS Programming November 6, / 89

54 Power: sample size determination Often for grant proposals, you might try to justify your grant budget based on an estimated effect size (i.e., µ 1 µ 2 ) based on preliminary data. The idea is that if you think the effect size might be some particular value, then you want the sample size to be large enough to have a reasonable chance (80% is often used) to reject the null hypothesis. Since larger samples require more money, this can be used to justify how much money you need to request for your study. Similarly, if you don t justify your proposed sample size, then a reviewer might complain that your study is likely to be underpowered, meaning it is unlikely to detect anything if there is a difference (i.e., if a new drug is more effective). SAS Programming November 6, / 89

55 Power: sample size determination Sample sizes for studies aren t usually completely under the researcher s control, but they are analyzed as though they are fixed parameters rather than random variables. If you recruit people to be in a study for example using flyers around campus, the hospital, etc., then you might have historical data to predict what a typical sample size would be based on how long and widely you advertise. Study designers can therefore often indirectly control the sample size. SAS Programming November 6, / 89

56 Power: sample size determination Random sample sizes might be worth considering, however, For the t-test example, you might have better power to reject the null hypothesis if your sample sizes are equal for the two groups than if they are unequal. For example, suppose you are recruiting for testing whether a drug reduces headaches, and you recruit both men and women. Suppose you suspect that the drug is more effective for men than women. If you recruit people for the study, you might not be in direct control of how many men versus women volunteer to be in the study. Suppose 55 women volunteer to be in the study and 45 men volunteer. You could make the sample sizes equal by randomly dropping data from 10 of the women, but this would be throwing away information. It is better to use information from all 90 study participants, although you might have less power with 45 men versus 55 women than with 50 d for each sex. SAS Programming November 6, / 89

57 Power: sample size determination On the other hand, if for your study, you are collecting expensive information, such as doing MRIs for each participant, you might decide to accept the first n women volunteers and the first n men volunteers. A power analysis could help you decide whether it was important to have a balanced design or not. SAS Programming November 6, / 89

58 Power: effect of unbalanced designs How could we simulate the effect of unbalanced versus balanced designs? Assuming we knew that there were a fixed number of participants (say n = 100), we could compare the effect of a particular unbalanced design (for example 45 versus 55) versus the balanced design (50 per group). We could also let the number of men versus women in each iteration of a simulation be a binomial random variable, so that the degree of imbalance is random. SAS Programming November 6, / 89

59 Power: determining effect size In addition to graphing power as a function of sample size, it is common to plot power as a function of the effect size for a fixed sample size. Ultimately, power depends on three variables: α, n, and the effect size such as µ 1 µ 2 for the two-sample t-test example. We usually fix two of these variables and plot power as a function of the other variable. The t-test example is easy to modify to plot power as a function of the effect size for a given sample size (say, n = 20). SAS Programming November 6, / 89

60 Power: determining effect size SAS Programming November 6, / 89

61 Power: determining effect size SAS Programming November 6, / 89

62 Power: determining effect size 1 SAS Programming November 6, / 89

63 Power: plotting both sample size and effect size SAS Programming November 6, / 89

64 Power: plotting both sample size and effect size SAS Programming November 6, / 89

65 Power: plotting both sample size and effect size SAS Programming November 6, / 89

66 Power: determining effect size 1 SAS Programming November 6, / 89

67 Power: determining effect size Note that the data set sim that has all of my simulated data has 840,000 observations. SAS is still reasonably fast, and the log file gives information about how long it took. NOTE: SAS Institute Inc., SAS Campus Drive, Cary, NC USA NOTE: The SAS System used: real time seconds cpu time 9.43 seconds We could make the plots smoother by incrementing the effect size by a smaller value (say.01), although this will generate 50 times as many observations. When simulations get this big, you start having to plan them how long will they take (instead of 30s, will it take 25min?, 25 days?), how much memory will they use, and so on, even though this is a very simple simulation. SAS Programming November 6, / 89

68 Length of simulations The log file also breaks down how long each procedure took. Much of the time was actually due to generating the PDF file with ODS. From the log file: NOTE: The data set WORK.SIM has observations and 5 variables. NOTE: DATA statement used (Total process time): real time 0.19 seconds cpu time 0.18 seconds NOTE: The data set WORK.PVALUES has observations and 9 variabl NOTE: The PROCEDURE TTEST printed pages NOTE: PROCEDURE TTEST used (Total process time): real time 9.12 seconds cpu time 8.97 seconds... NOTE: PROCEDURE SGPLOT used (Total process time): real time seconds cpu time 0.19 seconds SAS Programming November 6, / 89

69 Length of simulations When designing simulations, there are usually tradeoffs. For example, suppose I don t want my simulation to take any longer than it already has. If I want smoother curves, I could double the number of effect sizes I used, but then to keep the simulation the length of time, I might have to use fewer iterations (say 500 instead of 1000). This would increase the number of data points at the expense of possibly making the curve more jittery, or even not monotonically increasing. There will usually be a tradeoff between the number of iterations and the number of parameters you can try in your simulation. SAS Programming November 6, / 89

70 Length of simulations for R If you want to time R doing simulations, the easiest way is to run R in batch mode. In Linux or Mac OS X, you can go to a terminal, and at the shell prompt, type time R CMD BATCH myprogram.r and it will give a similar print out of real time versus cpu time for your R run. SAS Programming November 6, / 89

71 Power: determining effect size Power n=30 n=20 n= mu1 mu2 SAS Programming November 6, / 89

72 Power: determining effect size SAS Programming November 6, / 89

73 Power: determining effect size SAS Programming November 6, / 89

74 Power:tradeoff between number of parameters and number of iterations (500 vs 100 iterations) Power n=30 n=20 n=10 Power n=30 n=20 n= mu1 mu2 mu1 mu2 SAS Programming November 6, / 89

75 Using Power to select methods As mentioned before, power analyses are useful for determining which method is preferable when there are multiple methods available to analyze data. As an example, to consider the two sample t-test again when we have exponential data. Suppose we wish test H 0 : µ = 2 when λ = 1, so that the null hypothesis is false. Since the assumptions of the test are false, researchers might prefer using a nonparametric test. SAS Programming November 6, / 89

76 Using Power to select methods As an alternative, you can use a permutation test or other nonparametric test. Here we might wish to see which method is most powerful. If you can live with the inflated type 1 error for the t-test (or adjust for it by using a smaller α-level, then you might prefer it if is more powerful. A number of nonparametric procedures are implemented in PROC NPAR1WAY, as well as PROC MULTTEST. In addition, there are macros floating around the web that can do permutation tests without using these procedures. SAS Programming November 6, / 89

77 Using power to select methods Here we ll try PROC NPAR1WAY and just one nanparametric method, the Wilcoxon rank-sum test (also called the Mann-Whitney test). The idea is to pool all of the data, then rank them. Then calculate the sum of the ranks for group A versus group B. The two sums should be approximately equal, with greater differences in the sums of the ranks being evidence that the mean for one group is larger than the mean for the other group. SAS Programming November 6, / 89

78 Using power to select methods Note that there are many other methods we could have selected such as a median test or a permutation test. This is just to illustrate, and we are not necessarily finding the most powerful method. SAS Programming November 6, / 89

79 Power: comparing methods SAS Programming November 6, / 89

80 Power: comparing methods SAS Programming November 6, / 89

81 Power: comparing methods SAS Programming November 6, / 89

82 Power: comparing methods SAS Programming November 6, / 89

83 Power: comparing methods For these parameter values (exponentials with means of 1 and 2), the t-test was more powerful than the Wilcoxon test at all sample sizes. The Wikipedia article on the Mann-Whitney test says: It [The Wilcoxon or Mann-Whitney test] has greater efficiency than the t-test on non-normal distributions, such as a mixture of normal distributions, and it is nearly as efficient as the t-test on normal distributions. Given our limited simulation, we have some reason to be a little bit skeptical of this claim. Still, we only tried one combination of parameters. It is possible that for other parameters or other distributions, the t-test is less powerful. Also, the t-test has inflated type 1 error, so the comparison might be a little unfair. We could re-run the experiment using α =.01 for the t-test and α =.05 for the Wilcoxon to make sure that both had controlled type 1 error rates. SAS Programming November 6, / 89

84 Power: comparing methods Here s an example from an empirical paper, SAS Programming November 6, / 89

85 Power: comparing methods SAS Programming November 6, / 89

86 Speed: comparing methods For large analyses, speed and/or memory might be an issue for choosing between methods and/or algorithms. This paper compared using different methods within SAS based on speed for doing permutation tests. SAS Programming November 6, / 89

87 Use of macros for simulations The author of the previous paper provides an appendix with lengthy macros to use as more efficient substitutes to use as replacements for SAS procedures such as PROC NPAR1WAY and PROC MULTTEST, which from his data could crash or not terminate in a reasonable time. In addition to developing your own macros, a common use of macros is to use macros written by someone else that have not been incorporated into the SAS language. You might just copy and paste the macro into your code, possibly with some modification, and you can use the macro even if you cannot understand it. Popular macros might eventually get replaced by new PROCs or new functionality within SAS. This is sort of the SAS alternative to user-defined packages in R. SAS Programming November 6, / 89

88 From Macro to PROC An example of an evolution from macros to PROCS is for bootstrapping. For several years, to perform bootstrapping, SAS users relied on macros often written by others to do the bootstrapping. In bootstrapping, you sample you data (or the rows of your data set) with replacement and get a new dataset with the same sample size but some of the values repeated and others omitted. For example if your data is bootstrap replicated datas set might be etc. SAS Programming November 6, / 89

89 From Macro to Proc Basically to generate the bootstrap data set, you generate random n random numbers from 1 to n, with replacement, and extract those values from your data. This was done using macros, but now can be done with PROC SURVEYSELECT. If you search on the web for bootstrapping, you still might run into one of those old macros. Newer methods might still be implemented using macros. A webpage from 2012 has a macro for Bootstrap bagging, a method of averaging results from multiple classification algorithms. There are also macros for searching the web to download movie reviews or extract data from social media. Try searching on SAS macro 2013 for interesting examples. SAS Programming November 6, / 89