Earthquake data in geonet.org.nz

Transcription

1 Earthquake data in geonet.org.nz There is are large gaps in the 2012 and 2013 data, so let s not use it. Instead we ll use a previous year. Go to At the screen, click on the CLEAR button, so that it doesn t assume you want data from last week, last month, or last year. Then enter in a date range starting in If your birthday is December 10, then use December 10, 2010 to January 09, SAS Programming September 11, / 72

2 Earthquake data in geonet.org.nz Towards the bottom of the page, click on the botton that says Build Query instead of Show of Map. This way, it will generate a data file for you. Use the drop-down menu to select CSV as the file format. SAS Programming September 11, / 72

3 Earthquake data After you click the search button, you should be given a url for a website where you can download the data as a.csv file. If it opens up automatically in Excel, I recommend closing Excel without saving. Instead get the file from your Downloads or Desktop or wherever it was saved so that Excel doesn t change anything. SAS Programming September 11, / 72

4 Earthquake An alternative is to download a dataset I ve downloaded of all earthquakes from January 1st, 2010 to January 30th, 2011, and extract the month that you need. The dataset has about observations, and it will be up to you to figure out how to get the exact dates you need. You could, for example, open in Excel to figure out which rows you need, then use firstobs= and obs= in SAS to get just those rows. In linux you can do this $ grep " " quake csv > AugSep.csv $ grep " " quake csv >> AugSep.csv This will grab all lines in the files that have earthquakes in August into a file, then append that file with all earthquakes in September. SAS Programming September 11, / 72

5 Earthquake A quick way to determine the line number that your data should start with is this, assuming that your birthday is say, August 15th. $ nl quake csv > foo $ grep " " foo head -1 $ grep " " foo tail -1 Here nl creates a version of the file with line numbers in the first column, and this saved to the file foo, which you can later delete. The first grep and head will spit out the first line in foo that has your birthday. The second grep and tail will spit out the last line that has an event in the month starting on your birthday. SAS Programming September 11, / 72

6 Earthquake data I tried uploading a file with 5000 lines of earthquakes in SAS Studio, and this worked fine. SAS Programming September 11, / 72

7 Earthquake data I tried uploading a file with 5000 lines of earthquakes in SAS Studio, and this worked fine. SAS Programming September 11, / 72

8 Earthquake data I tried uploading a file with 5000 lines of earthquakes in SAS Studio, and this worked fine. SAS Programming September 11, / 72

9 The SUM statement Because the data step processes one observation at a time, it can be awkward to write functions of an entire column of data unless using a special procedure like PROC MEANS. The SUM statement is a helpful way around this to create a counter or cumulutive sum in a data set. SAS Programming September 11, / 72

10 SUM statements The idea with the sum statement is to have a new variable which is a counter, and which is usually incremented conditionally. As an example, we ll keep track of the number of cars that have more than 100,000 miles in the cars.txt data set. SAS Programming September 11, / 72

11 SUM statement with cars.txt dataset SAS Programming September 11, / 72

12 SUM Statement Some new aspects of the code are the syntax of using high + 1 defines a new variable and have it increment by 1 each time the condition is true. In most languages, you would write something like high = high +1 instead, but that doesn t work and just creates missing values the SUM function can also be used but requires the variable high to be initialized first, so here just gives 1 for every nonmissing value SAS Programming September 11, / 72

13 SUM statement with cars.txt dataset SAS Programming September 11, / 72

14 SUM statement with cars.txt dataset 21:00 Monday, September 8, SAS Programming September 11, / 72

15 SUM and RETAIN statements with cars.txt dataset You can fix previous code with a retain statement to initialize variables SAS Programming September 11, / 72

16 SUM and RETAIN statements with cars.txt dataset Note the order of the columns. The RETAIN statement effects the order of the columns defined in this datastep, but additional RETAIN statements won t change the order of the variables read in from the SET statement. SAS Programming September 11, / 72

17 MIN and RETAIN statements with cars.txt dataset What will this do? Notice that lag1() returns the value of the variable in the previous observation. SAS Programming September 11, / 72

18 MIN and RETAIN statements with cars.txt dataset Notice that the MIN function behaves intelligently for missing values it returns missing only when all values are missing. The minimjm of a missing value and a nonmissing value is the nonmissing value. minmiles returns the minimum value encountered so far for the value of year. SAS Programming September 11, / 72

19 First and Last observations for a given category (Chapter 24). Often we want to only output one value for each level of a given category. For example, I might only want to know the minimum mileage for each model year of the cars. Or I might want to know how many cars there were in a given year, with one observation per car. In a medical setting, it is common to have datasets with patientid as one variable with one record per doctor visit. The number of visits per patient can vary widely, so sometimes it is useful to summarize this with a dataset with one record (observation) per patientid. For this one record, you might want to keep track of, for example, the total number of visits, the sum of the fees for all visits, or the time of the most recent visit. For medical information, you might want to keep track of the highest blood pressure recorded on all visits. SAS Programming September 11, / 72

20 First and Last observations We ll practice using the car data again. Let s create a dataset that has the year and the minimum mileage for that year. If you use a BY statement after a SET statement in a data set, SAS creates internal variables that start with FIRST. and LAST.. These allow you to determine when a variable in the last one of its category (assuming the data is already sorted on the same BY variable). SAS Programming September 11, / 72

21 First and Last observations SAS Programming September 11, / 72

22 First and Last observations SAS Programming September 11, / 72

23 First and Last observations Here is simpler code that doesn t use the lag1 function. However, it is a bit less general and assumes more sorting. With the lag1 function, I could keep track of both the minimum and maximum at the same time and put those into the data set on one row. SAS Programming September 11, / 72

24 First and Last observations SAS Programming September 11, / 72

25 Example with Earthquake data: related to next homework! As an example, we ll use some earthquake data from New Zealand. This data set was downloaded from which has a database of New Zealand earthquakes. You specify a geographic range (using latitude and longitude) and a time frame. These were all earthquakes recorded from Jan 1st to March 31st in There were 4892 recorded earthquakes in that interval. The goal here will be to keep a cumulative count of the number of earthquakes that were at least 4.0 on the Richter Scale. The data is sorted decreasing by time, and we want it to be sorted increasing by time. SAS Programming September 11, / 72

26 Cumulative sums: Earthquake data The data comes as a.csv file. It is hard to read by eye as a plain text file, but easy to read into computer programs like SAS. SAS Programming September 11, / 72

27 Cumulative sums: Earthquake data Here is what part of it looks like read into Excel. Several variables on the right were chopped off. As is typical for empirical data, the data set includes many variables that you might not be interested in. SAS Programming September 11, / 72

28 Dates and Times in the earthquake data Something a little tricky about the earthquake data is that the times are not in a standard format. They have a T in the middle separating dates from hours, minutes, and seconds, and they have a Z at the end. There are different things you could do to deal with this. You could replace T and Z with spaces in the original data file and hope that this doesn t interfere with other variables such as eventtype. I m not really sure a good way to do this without linux tools. In emacs I would do a global replace of T with a space, and it would tell me how many replacements took place, which, if it is 2 per observation and there are no missing values, then it worked. What we ll do as a SAS solution is to read in the dates as character strings, but only read the first 10 characters, and this will chop off the hour-minute-second information. We ll have to convert the string to a date (an integer), which can be done using the input() function. SAS Programming September 11, / 72

29 SAS code for dealing with earthquake times SAS Programming September 11, / 72

30 SAS code for dealing with earthquake times SAS Programming September 11, / 72

31 input() function The input function can also be used to convert a string iwith commas for large numbers into a numeric value. For example data popsize; input population $9.; pop=input(population,comma9.); datalines; 2,115,353 ; run; It is often convenient to be able go back and forth between thinking of a value as either a string or a number. SAS Programming September 11, / 72

32 Earthquake counter SAS Programming September 11, / 72

33 Earthquake counter SAS Programming September 11, / 72

34 More on First and Last observations for a given value SAS Programming September 11, / 72

35 More on First and Last observations for a given value SAS Programming September 11, / 72

36 More on First and Last observations for a given value SAS Programming September 11, / 72

37 More on First and Last observations for a given value SAS Programming September 11, / 72

38 More on First and Last observations for a given value SAS Programming September 11, / 72

39 Example: average of top three salaries For one retirement plan at UNM, employees are given a proportion of the average of their top three years of salary, where the proportion depends on the number of years of service. Here we want to determine which three years have the highest salary for each employee and get the average salary for those three years. How can you do this in SAS? SAS Programming September 11, / 72

40 Example: average of top three salaries A bit tricky: need a RETAIN so that high3 is not missing. SAS Programming September 11, / 72

41 Example: average of top three salaries After sorting on ID and SALARY (not on year), we want a counter to rank the salaries for each ID. SAS Programming September 11, / 72

42 Example: average of top three salaries Here we used a subsetting IF to only include the observation with the average of the top three salaries. SAS Programming September 11, / 72

43 Example: average of top three salaries How could we have done this if salaries were in ascending order instead? In this case you could use LAST.ID to determine the highest salary. You could also LAG1 and LAG2 to determine previous salaries (second and third highest). You d have some code that looked something like this salary1 = lag1(salary); salary2 = lag2(salary); highest3 = mean(salary,salary1,salary2); if last.id; This code also assumes that each individual has at least three years worth of salary, which might be a reasonable assumption if the person qualifies for retirement, but still something to be careful about. SAS Programming September 11, / 72

44 Example: average of top three salaries SAS Programming September 11, / 72

45 A caution about LAG functions Use of LAGS can be very useful but can be a tricky when they are used conditionally (in IF or WHILE statements). The safest way of using them is to create a variable which is the lagged variable, and then do more sophisticated stuff with them. The following code generates incorrect output (but not errors) SAS Programming September 11, / 72

46 Imitating SAS in R The following code imitates the first.id and last.id variables in SAS, and I use it on the cars data set. SAS Programming September 11, / 72

47 Imitating SAS in R SAS Programming September 11, / 72

48 Imitating SAS in R SAS Programming September 11, / 72

49 DO loops An important part of programming is loops. SAS automatically loops through datasets that are read in and processes each observation one at a time. However, sometimes you need to control loops yourself, create loops without reading in data, loop through subsets of a dataset, and so forth. SAS Programming September 11, / 72

50 DO loops In the previous example, the data step generated simulated normal random variables and outputted them to the dataset called n01 because it was Normal(0,1) values. The value in parentheses is the random number seed. You will get different values depending on the seed, which allows you to reproduce results or generate new results. The syntax requires that the DO loop be closed with END; which you can think of as being like a closing parentheses. SAS Programming September 11, / 72

51 DO loops To get into SAS s head so to speak, consider the following code and try to anticipate what it does. The data set address.txt has three observations (the addresses used for Homework problem 1.2). SAS Programming September 11, / 72

52 Using DO to create plots Note that log is the natural log. SAS Programming September 11, / 72

53 Using DO to create plots This plot is not very good. Axis labels are too small, title is oddly large, and there is no legend. But blue is for odds, and orange is for log-odds. SAS Programming September 11, / 72

54 Using DO with LAG functions Some recursive problems can be solved using LAG functions within DO loops. An example from Ross, A First Course in Probability Chapter 2, is: Let f n denote the number of ways of tossing a coint n times such that successive heads never appear. Argue that f n = f n 1 + f n 2 n 2, where f 0 1, f 1 2 HINT: How many outcomes are there that start with a head, and how many that start with a tail? If P n denotes the probability that successive heads never appear when a coin is tossed n times, find P n (in terms of f n ) when all possible outcomes of the n tosses are equally likely. Compute P 10. SAS Programming September 11, / 72

55 Using DO with LAG functions The recursion f n = f n 1 + f n 2 works for the first term because if the first toss is tails, then we have n 1 tosses without restrictions, and there are f n 1 ways to not get successive heads with the remaining tosses. If the first toss is heads, the second toss must be tails (if n > 2), so there are now n 2 tosses without restrictions, and there are f n 2 ways to accomplish this. The probability of not getting successive heads in n tosses is therefore P n = f n /2 n, since each of the 2 n sequences of tosses is equally likely. To compute this, we use a LAG function within a DO loop. SAS Programming September 11, / 72

56 Using DO with LAG functions SAS Programming September 11, / 72

57 Using DO with LAG functions SAS Programming September 11, / 72

58 Using DO with LAG functions This code doesn t work, and the problem seems to be putting the LAG function in an IF statement. SAS Programming September 11, / 72

59 DO WHILE You can also DO something conditionally until you have the desired number of successes or observations. Suppose I want to simulate normal random variables conditional on having values greater than 3. I might want to use this to get, say the median of X X > 3, which is not easy to deal with analytically. The following accomplishes this, although it is somewhat inefficient. To do this more efficiently, you could use importance sampling. SAS Programming September 11, / 72

60 DO WHILE caution Because the condition that you require might never occur or might take a very long time to execute, your program could run a VERY long time and appear to be hanging, creating an infinite loop or just very long loop. If we wanted 1000 observations of standard normal variables being larger than 3, we d need generate an average of 740,000 random numbers. If we wanted 1000 normal variables being larger than 6, we d need to generate an average of 1,013,594,635,000 random variables, which probably wouldn t work, and you might need something more efficient like importance sampling. If you aren t sure whether something will take too long, you could add a counter and stop after a maximum number of iterations. SAS Programming September 11, / 72

61 DO WHILE testing SAS Programming September 11, / 72

62 DO UNTIL testing An alternative approach is to use DO UNTIL which executes the code in the block first, then tests the condition to see whether it should continue to execute. It is mostly a matter of style whether you use DO WHILE or DO UNTIL. For both DO WHILE and DO UNTIL, infinite loops are easy to mistakenly creep into your code. A DO WHILE will be an infinite loop if the condition is always true, and a DO UNTIL will be an infinite loop if the condition is always false. The following uses a DO UNTIL instead of DO WHILE to get the same results. SAS Programming September 11, / 72

63 LEAVE and CONTINUE with DO loops In addition to testing at the top (WHILE) or bottom (UNTIL) of a loop, you can break out of a loop using the LEAVE and CONTINUE statements. With the LEAVE statement, you break out of the DO loop entirely and move past the END statement. For the CONTINUE statement, you return to the top of the loop to do the next iteration of the loop. This is an alternative to putting extra testing conditions in the WHILE or UNTIL conditions. SAS Programming September 11, / 72

64 LEAVE and CONTINUE SAS Programming September 11, / 72

65 CONTINUE Continue will mostly be useful if you have more complicated loops where there is computational savings if you jump to the next iteration once you have done everything needed. This can also be useful for running an example with many iterations, but only printing a few of them. In Markov chain Monte Carlo experiments for example, you might want to run a Markov chain for several million iterations, but this is too many to plot. Instead of plotting each iteration, you could just plot every 1000th iteration. This is most easily achieved with the MODULUS function, mod(), which returns the remainder when divided by the modulus. The modulus function can also be useful for separating a dataset into its odd and even observations numbers, or some other unusual grouping. SAS Programming September 11, / 72

66 CONTINUE example with modulus SAS Programming September 11, / 72

67 MODULUS example SAS Programming September 11, / 72

68 MODULUS operator to splice up a dataset SAS Programming September 11, / 72

69 Another way to do the phone data... SAS Programming September 11, / 72

70 Application of modulus to longitude data The New Zealand data includes longitudinal coordinates on both sides of the international date line, although they are primarily to the the West of it. Some of the longitude values are recorded as negative, so you might have one earthquake with longitude 179 (meaning 179 degrees east of Greenwich) and another with longitude -179 (meaning 179 degrees west of Greenwich), but really there are only two degrees of longitude separating them. On a scatterplot, it would make sense to have them close together. How can we fix this problem? SAS Programming September 11, / 72

71 Application of modulus to longitude data: no modulus This data is for eventtype = earthquake only SAS Programming September 11, / 72

72 Application of modulus to longitude data: with modulus When I first read in the data, I added the line in the datastep longitude = mod(longitude+360,360); This doesn t effect positive longitudes, but allows something with a longitude of -179 to have a longitude of 181, which is where we want it. SAS Programming September 11, / 72