Exploratory data analysis with one and two variables

Transcription

1 Exploratory data analysis with one and two variables Instructions for Lab # 1 Statistics Probability and Statistical Inference DUE DATE: Upload on Sakai on July 10 Lab Objective To explore data with histograms and scatter plots. Review A few highlights to review from the previous lab: 1. The Working Directory It is always important to know where the files that you are using are saved on the computer. This is so both you and Stata can access the correct files. Let s see how this would work for this problem. Download the data set movies2012.dta from the course website (link next to the lab handout) and save/put it in the folder you are going to work in (e.g., C:\Users\John\Documents\Stats111\ lab1). Now you need to direct Stata to this folder, which will be the folder you will be working from. There are two ways to accomplish this. (a) Use the menu bar and navigate to File -> Change Working Directory. This will pull up a window that you can then use to find the folder where you saved the data. (b) Issue a command to change the working directory by typing cd followed by the filepath to your file, (e.g., cd C:\Users\John\Documents\Stats111\lab2). Typing pwd into the command line well tell you the current directory and can be used to verify that you are in the right place. 2. Do-files All the commands you enter into the Command Line for the lab can (and should) be put into a "do-file to allow replication and access at a later date. You should save a "do-file in the

2 Stats Exploratory Data Analysis folder where the data is. You can save the example.do from the preovious lab session from the class website and use it as an example of your "do-file". The first three lines and the last two are creating a example.log file that will have all the commands that were executed, as well as the output of them. If you want to use this example.do file as a model for your lab script, save it as lab1.do in the correct directory. Also, change the name of the log file that you want to create (in the third line of the do file, replace example.log with lab1.log). The comments and the display comands are just a silly example. These are the lines that you will want to replace with the commands that you use during the lab. For example, after you have loaded the data (you can also save this command in your do file) and changed to the correct working directory, you may want to summarize a variable. In this case, you would type summarize varname in the do file. In order to run each command at a time, you can select the commands you want to run and click on the Run Selection button (or press a shortcut). Feel free to add as many comments (lines starting with *) as you want. They will help organize your script file so you can understand it the next time you open it. At the end of the lab, you should save your do file. It will allow you to recover all the analysis that you did. 3. Fonts Different fonts generally mean very specific things in the labs. This is to help you more easily distinguish Stata code from the rest of the text: Typewriter text refers to Stata commands. Italicized text, either in typewriter or in regular font, refer to variable names. Sometimes these are specific variables in a data set (Domestic) while at others they are simply a place holder for any variable or name that you may choose (cont_var). Red text is for Data Analysis Tips. Blue text are generally links to datasets, examples or places in the document. Lab Procedures Remeber to write your answers in a report that will be graded. Along with your answers, you can include outputs, graphs and commands to your report. Let s go to the movies! What are the characteristics of U.S. movies that make the most money? Let s address this question with the data set movies2012.dta, which you can download from the course website. It comprises

3 Stats Exploratory Data Analysis data on the 250 top domestic grossing movies of all time as of November The variables are: Variable Ranking Title Year Domestic Foreign Worldwide Budget Rating Best_Picture Genre All_Genres Director Description Ranking on Domestic gross sales Movie title Release year Domestic gross sales Foreign gross sales Worldwide gross sales Budget MPAA rating Academy Awards Best Picture (nominated or won) Main/first genre of the movie List of all genres the movie falls into Name of the director There are missing data in this file. We will ignore them for simplicity. In general, when confronted with missing data, it is best to get the advice of a professional statistician before doing analyses. Data Analysis Tip: The unit of measurement for the monetary variables is not stated. That is bad practice. Always include a description of the units somewhere on the file. Based on knowledge of movie revenues, it is clear that that the unit of measurement is $1,000,000. Questions: 1. After reading in the data, describe the distributions of foreign and domestic grosses. That is, say where most values are, note any outliers, and say whether the distribution is tightly packed around its mean or is spread out. Also, report the mean and standard deviation. In addition to summarizeing the data, you can use histograms to get a visual representation of the distribution of the data. The command is histogram varname, name(graph1, replace) where varname is the variable of interest. As with most commands, there are many options available with histogram. One of the most useful options is name(graph1, replace). This stores a version of the graph in temporary memory to be used later and it causes the graph to be displayed in a window titled graph1. This allow multiple graph windows to be open at the same time 1. You use the replace option within the perentheses of the name option to overwrite the previous version of the graph. Further, if you want to compare multiple histograms in one window you can combine them by typing 1 The default is for each graph, regardless of type, to overwrite the previous graph

4 Stats Exploratory Data Analysis graph combine graph1 graph2, name(graph_all, replace) where again graph1, graph2, and graph_all are just examples for names of the graphs. You can also navigate to Graphics-Histogram to get a wizard to help with the graph. Data Analysis Tip: The default histogram in Stata is a true histogram, where the areas of the bins sum to one. Often people want just the heights to sum to one. This is accomplished with the fraction option. Further, if you want the y-axis to simply count how many observations are in each bin, you can use the frequency option. 2. Which sentence best describes the distributions of domestic and foreign grosses? You can just write the letter of your choice on the lab report. (a) Domestic and foreign grosses are very similar. (b) Domestic and foreign grosses have similar distributional shapes, but foreign grosses tend to be larger than domestic grosses. (c) Domestic and foreign grosses have similar distributional shapes, but domestic grosses tend to be larger than foreign grosses. (d) The two distributions look nothing like each other, because one has a long left tail and the other has a long right tail. 3. What are the names of the two movies that are the largest outliers on all three monetary variables? 4. We can examine the relationship between world-wide gross and movie genre using a box plot. Use the variable Genre for this analysis. The command for a box plot is graph box cont_var, name(graph1, replace) or graph box cont_var, name(graph1, replace) over(cat_var) where cont_var represents the continuous variable that you are trying to graph. The option over(cat_var) allows you to break the box plot down by different values of a categorical variable. Alternatively, you can navigate to Graphics-Box Plot, type the continuous variable in the Main tab and the categorical variable in the Categories tab. If you want to clean up the graph, you can test out some of the other tabs in the wizard. Answer the three questions below. (a) Out of Comedy and Animated movies, which one has a distribution of world-wide grosses that is most similar to the distribution of world-wide grosses for Action movies? Justify your choice in at most two sentences. (b) Compare the distributions for Drama movies and Adventure movies. Do they have reasonably similar medians? Is one more spread out than the other (if so, say which one)?

5 Stats Exploratory Data Analysis (c) If you directed a movie and wanted to make lots of money worldwide, which type appears to give you the best chance of doing so? Base your answer on the results of the box plot. 5. Describe the relationship between domestic gross and foreign gross. To make a scatter plot, we will tell Stata that we want to do a twoway graph as a scatter: graph twoway scatter varname1 varname2 Alternatively, you can navigate to Graphics-Twoway graph (scatter, line, etc.), go to the Plots tab, and hit the Create button. Choose Scatter and the Y and X variables. Items to include in your description are the general trend of the relationship (e.g., positive and linear, negative and linear, some other pattern, no clear pattern) and whether there are any outliers or points that do not fit the pattern. 6. Report the three pairwise correlations between Foreign, Domestic, and World-wide gross. Further, graph the raw distribution of the data for each pair. To find correlations in Stata, type correlate varname1 varname2... which will show a matrix of the correlations between all the variables used as input (you can use as many as you would like). Note that the diagonal is always 1. Make sure you know why that is. In addition, you can create a scatterplot matrix which will create scatter plots of all the different outcomes by typing graph matrix varname1 varname2... Alternatively, you can navigate to Graphics-Scatterbox Matrix. And again, you can use multiple variables. Do the correlations suggest strongly positive linear relationships, weakly positive linear relationships, no linear relationships, weakly negative linear relationships, or strongly negative linear relationships? 7. Why are the correlations between Domestic and Worldwide, and Foreign and Worldwide, stronger than the correlation between Domestic and Foreign? The answer has to do with the definitions of the variables. 8. (extra) Outliers can have a strong effect on correlations. Let s check to see if excluding Avatar and Titanic changes the correlations substantially. To exclude Avatar and Titanic, let s again use the if functionality of Stata by typing if Title!= Avatar & Title!= Titantic at the end of the previous commands. Data Analysis Tip: Note that!= is defined as not equal to and & is the AND operator.

6 Stats Exploratory Data Analysis Now, re-calculate the correlations in (6). Did the correlations get stronger or weaker? Does the substance of your conclusions in (6) change very much when excluding Avatar and Titanic? Data Analysis Tip: It is not acceptable to exclude outliers from analyses unless you have a scientific reason to do so (e.g., a data entry error, or maybe the outlying unit is not part of your target population). Hiding outliers is fudging data to get results you want. That is dishonest and unethical. When you see outliers, do analyses with and without them. When the results do not change much, report the results based on the full data set, and tell your audience that the results were not sensitive to the outliers. When the results do change substantially, report both sets of analyses: one with and one without the outliers. This honestly informs people that your conclusions are not on very solid ground, because particular data points affect the results greatly.