Introduction to RStudio

Transcription

1 First, take class through processes of: Signing in Changing password: Tools -> Shell, then use passwd command Installing packages Check that at least these are installed: MASS, ISLR, car, class, boot, leaps, glmnet, pls, splines, gam, akima, tree, randomforest, gbm, e1071, ROCR Creating vectors v1 = c(1, 4, 2, 7) # print result afterward v2 = -1:5 v3 = seq(2, 8,.25) v4 = c('phineas','ferb','candace','jeremy','isabella','buford','baljeet') v5 = (1:7) == 3 v6 = (1:5)^2 v7 = 2*exp(-2*v1) v8 = sample(c(0,1), 12, replace=true, prob=c(.25,.75)) Exercise 1. How might you simulate 10 rolls of dice, as in the game Monopoly? Manipulating/accessing vectors 5*v1 v2 v2[3] v2[4:6] v2[c(1,3,6)] v2[-c(2,5)] v2 > 3 v2[v2 > 3] v4[v2 > 3] v4[v5] Basic function plotting 2

2 curve(x^2, 0, 1.5, ylab="y") curve(x^3, 0, 1.5, add=true, col="blue") curve(x^4, 0, 1.5, add=true, col="red") legend(0.1, 2.1, c(expression('x'^2),expression('x'^3),expression('x'^4)), lty=c(1,1,1), lwd=2.5*c(1,1,1), col=c("black","blue","red")) y x 2 x 3 x x Distribution models (emphasis on normal family) plotting distributions Two normal curves, where the black has mean µ = 2 and standard deviation σ = 1, the blue has µ = 0, σ = 2. Both are symmetric, bell-shaped. In general the curve (probability density function, or pdf) of a normal distribution is f (x) = 1 σ 2π e (x µ)2 /(2σ2). We have plotted a pdf from the exponential family of models in red. Its formula, for a given parameter λ > 0 (below I have taken λ =.4), is 0, if x < 0, f (x) = λe λx, if x 0. 3

3 curve(dnorm(x, 2, 1), from=-6, to=6, ylab="y") curve(dnorm(x, 0, 2), add=true, to=6, from=-6, col="blue") curve(dexp(x,.4), add=true, to=6, from=-6, col="red") # an exponential pdf y x assessing probabilities from such (continuous) models: integration If X has a pdf f X (x), then P(a < X < b) := b a f X (x) dx. For a random variable X having the standard normal distribution, we can find P(0 < X < 1) via a command like integrate(dnorm, 0, 1) Exercise 2. Find other probabilities, such as P( 1 < X < 1), P( 2 < X < 2), P( 3 < X < 3). Try adapting the attempts so that, if X Norm(1, 2), we obtain the probability P(1 < X < 3). We run into the problem that integrate() does not allow you to specify arguments (like µ, σ) to your function. One work-around is to write a user-defined function with these arguments hardcoded: tlsfn = function(x) { } return( dnorm(x, 1, 2) ) # mu=1, sigma=2 is hardcoded in 4

4 integrate(tlsfn, 1, 3) But by the FTC, if we had an antiderivative function, one such as F X (x) := x f X (s) ds, = P(X < x), (1) then we could obtain probabilities using it: P(a < X < b) = b a f X (x) dx = F X (b) F X (a). For a given pdf f (x), an antiderivative F(x) defined by means of (1) is called the corresponding cumulative distribution function, or cdf. In R, pnorm(1,0,1) - pnorm(0,0,1) pnorm(1) - pnorm(0) pnorm(1) - pnorm(-1) pnorm(2) - pnorm(-2) pnorm(3,1,2) - pnorm(1,1,2) # same as above Exercise 3. Assume IQ scores in the U.S. adult population are well modeled by Norm(100, 15). If you pick an adult at random, what is the chance that person s IQ exceeds 130? Exercise 4. Suppose a part on an automobile has a lifetime X, measured in hours, modeled by an exponential distribution with parameter λ = What is the chance this part fails in the first 200 hours of use? Notes: There are many different sorts of functions f which can serve as a pdf (a probability model). It must be the case, however, that 1. f (x) dx = 1. That is, F(+ ) = P( < X < ) = f (x) 0 for all x. Quantiles, Percentiles, Median qnorm(.3, 64, 3) # Norm(64, 3) is model for female heights in inches qnorm((0:10)/10, 64, 3) 5

5 Means, Variances, Standard Deviations definitions: If X has a pdf f (x), take the mean µ X (expected value E(X)) and variance Var(X) to be E(X) := E((X µ X ) 2 ) := x f (x) dx, (2) (x µ X ) 2 f (x) dx. (3) The standard deviation, then, is σ X = Var(X), and hence Var(X) may sometimes be written as σ 2 X. Explore R commands to compute mean, variance for Norm(µ, σ), Exp(λ). Suppose you intend to flip a coin n times, and the coin (not necessarily fair) has probability p of being a head. Let X be the number of heads in the n flips. Use a probability tree to work out the probability mass function (pmf), then consider how one calculates things like the cumulative distribution P(X x), the mean and standard deviation. Sampling large samples begin to take on the characteristics of the population from which they are drawn x1 = rnorm(50, 18, 4) # draws sample of size 50 from Norm(18,4) x2 = rnorm(500, 18, 4) # draws sample of size 500 from Norm(18,4) hist(x1, xlab="", ylab="density", freq=false, main="hist. of x1", col="gray90") hist(x2, xlab="", ylab="density", freq=false, main="hist. of x2", col="gray90") curve(dnorm(x,18,4), 5, 30, xlab="", ylab="", main="norm(18,4) dist") Hist. of x1 Hist. of x2 Norm(18,4) dist density density using samples to estimate means, standard deviations, quantiles 6

6 Obtaining Data Complex data sets involve many measurements (variables) taken on a collection of like-objects variously called cases, subjects or units, depending on the context. The typical arrangement is to place these measurements in a table what, in R is known as a data frame. Each row of the table represents a unit studied, and the columns correspond to the variables. Data in packages data() help(faithful) head(faithful) # often gives details about the data # displays the first few records in the data Using delimited files The file at the specified url is a comma separated value (csv) file, containing responses to a survey conducted in 2004 by students then enrolled in introductory statistics classes at Calvin. You may view the questions as they were posed at this link Respondents were typically the students conducting the survey, along with other students with whom they came into contact. I could give the data frame any name I want, and have chosen ss for student survey." ss = read.csv(" names(ss) dim(ss) # shows size of table/data frame Viewing Data Frequency tables are convenient for exploring univariate categorical data. While larger populations/samples result in larger counts/frequencies within the various values of the categorical variable, one might expect the proportions of occurrences of these values to be relatively stable. ss$selfhandedness # produces vector containing selfhandedness responses ss[,5] # produces vector corresponding to 5th column ss[31:35, c(3,5,8)] # one way you can pair down a data frame subset(ss, select=c(gender,cds)) # another way xtabs(~ selfhandedness, data=ss) # Note the need for "cleaning" the data cleanedss = droplevels(subset(stusurvey, selfhandedness!="")) xtabs(~ selfhandedness, data=cleanedss) prop.table(xtabs(~ selfhandedness, data=cleanedss)) 7

7 Exercise 5. Look over the list of variables in this student survey dataset. Determine which are categorical and which are quantitative. Considering only those which are quantitative, further determine which are discrete, and which are continuous. Exercise 6. We might consider the students who took this survey as a population (as opposed to a sample from a larger population of students). Write a command that takes a sample (with replacement) of region values of size n = 10 from this population, and shows the proportion of respondents from the three region types (Rural, Suburban, Urban). Do these proportions look similar to those for the overall population (i.e., the dataset as a whole)? As you increase the size of your sample (Note that, since sampling with replacement, n can be made to be larger than the actual number of cases in the dataset!), does the distribution of values for the region variable appear increasingly like the distribution for the population? Exercise 7. Another variable in this student survey dataset is momhandedness. Create a frequency table of its values, noting that there are instances where survey participants did not respond to the corresponding question. Write a command, or sequence of commands, which produces a data frame that has been cleaned in the sense that the records where either selfhandedness or momhandedness are blank have been removed. Call your final data frame twicecleanedss. When two categorical variables are of interest, the counterpart to the frequency tables above is a contingency table. The commands given next produce tables of various sorts, some of which are contingency tables (those giving actual frequencies rather than fractional values). Execute them and reflect on the results. xtabs(~selfhandedness + momhandedness, data=ss) prop.table(xtabs(~selfhandedness + momhandedness, data=ss)) xtabs(~selfhandedness + momhandedness, data=twicecleanedss) prop.table(xtabs(~selfhandedness + momhandedness, data=twicecleanedss)) prop.table(xtabs(~selfhandedness + momhandedness, data=twicecleanedss), margin=1) prop.table(xtabs(~selfhandedness + momhandedness, data=twicecleanedss), margin=2) Exercise 8. The result of each command given above is, of course, closely related to that of the other commands. Describe the various contexts in which one might find a particular version of greater use than its conterparts. If you were imagining momhandedness in the role of explanatory variable and selfhandedness as response variable, which command(s) would seem most useful? Explain your choice? 8

8 Exercise 9. selfhandedness? Explain. Do you think momhandedness is useful in the prediction A mosaic plot is a visual depiction of the information in a contingency table. Try this command to view a mosaic plot depicting the information of the tables above. mosaicplot(xtabs(~selfhandedness + momhandedness, data=twicecleanedss)) For univariate quantitative data, the main type of plot is a histogram. We already used one R command, hist(), above. There is an alternate command, from the lattice package, which I tend to prefer. It is used below to produce a histogram of the number of cds owned (yes, that was back in the day"!) by respondents in our survey, the one on the left. As you see, there is a value (one student who said he owned 601 cds) that is far removed from the others, an outlier. One gets a better view of the rest of the responses on the right, the result of removing this outlier. histogram(~cds, data=ss) Percent of Total Percent of Total cds CDs owned Exercise 10. Write a command which reproduces the histogram on the right. This course focuses on relationships between variables, which suggests we again turn our attention to displays of two variables at a time. It may be that we may wonder whether there is an association between gender, a categorical variable, and number of cds owned. If so, a model predicting the number of cds a person owns may be enhanced by taking gender into account. A plot giving separate histograms of cds owned by gender is likely the first step in investigating whether an association exists. histogram(~cds gender, data=ss, n=20, layout=c(1,2)) 9

9 M Percent of Total 60 F cds Try leaving out the layout=c(1,2) part, and see if you think it helps to include it. Whatever you decide about that, it seems once again, we might be able to make a better comparison if we leave out Mr. over 600 CDs". Exercise 11. Write a command which produces histograms broken down by gender, as above, but does so without the outlier. What would you expect to see if there were no association between gender and numbers of cds owned? Is that what you see in this data? Exercise 12. Is there a difference in cd ownership for students coming from different regions/living environments? Do the analysis and explain your results. Prepare a nice report, employing R Markdown, of your answer. Of course, one may be interested in whether an association exists between two quantitative variables. For this purpose, we use a scatterplot. For each subject in the data set, we plot a single point whose x-coordinate comes from the explanatory variable and y-coordinate from the response. Below we 10

10 produce a scatterplot between two quantiative variables, pulse rate and gpa, in the student survey data. xyplot(gpa ~ pulse, data=ss, pch=19, cex=.5) pulse gpa Exercise 13. Imagine the appearance of a scatterplot between two quantitative variables when they are not associated. Given the result of the command above, are you quite convinced that an association exists between pulse and gpa? Why or why not? 11

11 Tonight s Assignment: Read Chapter 1 from the textbook. Do the exercises that appear in this document, writing them up (preferably in R Markdown). As per the examples we heard today in the video lecture, think of a challenging data question of particular interest to you. Do not use one you have heard of elsewhere. You need not limit your scope to problems you could, with training like that of the authors, tackle single-handedly. Write out what the problem is, and what response variable(s) and predictor variable(s) you would use. 12