DSCI 325: Handout 2 Getting Data into SAS Spring 2017

Size: px

Start display at page:

Download "DSCI 325: Handout 2 Getting Data into SAS Spring 2017"

Alisha Thomas
6 years ago
Views:

1 DSCI 325: Handout 2 Getting Data into SAS Spring 2017 Data sets come in many different formats. In some situations, data sets are stored on paper (e.g., surveys) and other times data are stored in huge electronic databases. SAS is one of the most popular and, more importantly, efficient statistical software packages for the management of data. In this handout, we will explore methods for getting data into SAS. SOME RULES FOR SAS DATA SETS Data Types There are just two data types in SAS, numeric and character. o o Numeric: these variables take on values that are numbers and thus can be manipulated as such. Character: these variables can be up to 32,767 characters long and may contain numbers, letters, and some special characters (e.g., $ or!). Realize that at times it might be appropriate to treat numbers as characters, and SAS allows us to do this. For example, a ZIP code most likely should be treated as a character and not a numerical value. Missing Data o o A missing observation for a numeric variable is identified with a period. A missing observation for a character variable is identified with a blank. Rules for SAS Variable Names o o o o Variable names must contain between 1 and 32 characters Variable names must start with a letter or underscore Variable names can contain only letters, numbers, and underscores. Special characters such as %,!, etc., are not allowed in variable names; blanks also cannot appear in SAS variable names. SAS variable names are not case-sensitive. 1

USING THE SNIPPETS IMPORT TOOL You can use the Snippets Import Tool to get data into the SAS Studio environment when running SAS University Edition. Suppose you want to read in the file CarAccidents.

2 USING THE SNIPPETS IMPORT TOOL You can use the Snippets Import Tool to get data into the SAS Studio environment when running SAS University Edition. Suppose you want to read in the file CarAccidents.csv. First, save the desired data set in your shared folder. Select Snippets > Data > Import CSV file. The following code is added to your CODE tab. /** FOR CSV Files uploaded from Windows **/ FILENAME CSV "<Your CSV File>" TERMSTR=CRLF; /** FOR CSV Files uploaded from Unix/MacOS **/ FILENAME CSV "<Your CSV File>" TERMSTR=LF; /** Import the CSV file. **/ PROC IMPORT DATAFILE=CSV OUT=WORK.MYCSV DBMS=CSV REPLACE; /** Print the results. **/ PROC PRINT DATA=WORK.MYCSV; /** Unassign the file reference. **/ FILENAME CSV; 2

Now, you must customize the filenames in this code. /** FOR CSV Files uploaded from Windows **/ FILENAME CSV "/folders/myfolders/caraccidents.csv" TERMSTR=CRLF; /** Import the CSV file.

3 Now, you must customize the filenames in this code. /** FOR CSV Files uploaded from Windows **/ FILENAME CSV "/folders/myfolders/caraccidents.csv" TERMSTR=CRLF; /** Import the CSV file. **/ PROC IMPORT DATAFILE=CSV OUT=WORK.CarAccidents DBMS=CSV REPLACE; /** Print the results. **/ PROC PRINT DATA=WORK.CarAccidents; /** Unassign the file reference. **/ FILENAME CSV; Note: The OUT statement is used to specify where to save the SAS data table in the format Library.Filename. In this example, the data table is named CarAccidents and is saved to the Work library (a temporary location). MORE ON SAS DATA SETS We have now created a SAS data set. This is technically a file containing two parts: a descriptor portion and a data portion. The following programming statements can be used to view the descriptor portion. PROC CONTENTS DATA = CarAccidents; The descriptor portion contains information about the data set: The programming statements below can be used to print the data portion of the SAS data set. Note: using the WHERE ID <=10 statement tells SAS to print only the first 10 observations. PROC PRINT DATA=CarAccidents; WHERE ID <= 10; 3

The observations (in rows) are collections of data values that belong to the particular object or subject (in this case person) being measured.

4 The data set is comprised of variables and observations. The variables (in columns) are collections of data values that describe a particular characteristic of what is being measured. The observations (in rows) are collections of data values that belong to the particular object or subject (in this case person) being measured. MORE ON SAS VARIABLES The descriptor portion of a SAS data set also stores the following attributes of each variable: The variable s name The variable s type The variable s length The variable s format The variable s informat The variable s label 4

READING RAW DATA INTO SAS You can also provide SAS with internal raw data by typing data directly into your SAS program with a DATA step. For example, let s enter a portion of the NC_Birth data.

5 READING RAW DATA INTO SAS You can also provide SAS with internal raw data by typing data directly into your SAS program with a DATA step. For example, let s enter a portion of the NC_Birth data. Note that the INPUT line is used to identify variable names (the $ is used to identify character variables). The DATALINES statement must be the last statement in the data step, and all lines following this statement should contain data. Finally, note that information about the father s minority status and age are not known for the 4 th observation, so a period (.) is used to identify missing values. DATA NC_Birth; INPUT FatherMinority$ FatherAge MotherMinority$ MotherAge Apgar1 Apgar5; DATALINES; Nonwhites 50 White White 19 White White 37 White Nonwhite Nonwhite 39 Nonwhite White 20 Nonwhite White 30 White White 28 White White 22 White Nonwhite 25 White ; PROC PRINT DATA=NC_BIRTH; The PROC PRINT step produces the following output. 5

6 SAS can work directly with external raw data, as well. In these cases, the INFILE command is used within a DATA step to specify the location of the external raw data. For example, the data used in the previous example also is provided in a *.dat file named DataStep_NCBirth.dat. The following code can be used to read in this external data in raw format. DATA NC_Birth; INFILE '/folders/myfolders/datastep_ncbirth.dat'; INPUT FatherMinority$ FatherAge MotherMinority$ MotherAge Apgar1 Apgar5; PROC PRINT DATA=NC_Birth; At times, it may be necessary to specify the length of a line when reading in external data. You can do this with the LRECL= option (which stands for logical record length) in the INFILE statement. If you use this option, make sure that you specify a record length at least as long as the longest record in your data file; otherwise, you may encounter problems. For example, let s take a close look at what SAS does if we specify 20 as the record length in the above program. DATA NC_Birth; INFILE '/folders/myfolders/datastep_ncbirth.dat'lrecl=20; INPUT FatherMinority$ FatherAge MotherMinority$ MotherAge Apgar1 Apgar5; PROC PRINT DATA=NC_Birth; 6

A print-out of the ENTIRE dataset read in by SAS with this program is shown below: Question: What did SAS do to you? Answer: Look at your log file! First, consider line 1 of our data set.

7 A print-out of the ENTIRE dataset read in by SAS with this program is shown below: Question: What did SAS do to you? Answer: Look at your log file! First, consider line 1 of our data set. SAS truncates information after 20 spaces and moves to the next line because SAS is trying to find the values for Apgar1 and Apgar5. You told SAS to look for six variables in the INPUT statement and it s trying to find valid information for all six variables! NOTE: Invalid data for Apgar1 in line RULE: White 19 White 18 8 FatherMinority=Nonwhite FatherAge=50 MotherMinority=White MotherAge=2 Apgar1=. Apgar5=19 _ERROR_=1 _N_=1 NOTE: Invalid data for MotherAge in line NOTE: Invalid data for Apgar5 in line White 20 Nonwhite 19 FatherMinority=Nonwhite FatherAge=39 MotherMinority=Nonwhite MotherAge=. Apgar1=20 Apgar5=. _ERROR_=1 _N_=3 NOTE: Invalid data for Apgar5 in line White 28 White 29 4 FatherMinority=White FatherAge=30 MotherMinority=White MotherAge=27 Apgar1=9 Apgar5=. _ERROR_=1 _N_=4 NOTE: Invalid data for Apgar5 in line Nonwhite 25 White 25 FatherMinority=White FatherAge=22 MotherMinority=White MotherAge=19 Apgar1=9 Apgar5=. _ERROR_=1 _N_=5 7

8 As mentioned above, SAS will continue to search for variables (even across multiple lines) until it finds valid information for each variable. Consider our dataset again in which observation 5, 6, and 10 have information on multiple lines. The same code as used above will work here, as well, since SAS automatically continues its search for information across one or more lines. DATA NC_Birth; INFILE '/folders/myfolders/datastep_ncbirth_multiline.txt'; INPUT FatherMinority$ FatherAge MotherMinority$ MotherAge Apgar1 Apgar5; PROC PRINT DATA=NC_BIRTH; 8

SPECIFYING COLUMNS WHEN READING RAW DATA INTO SAS We can specify exactly which columns should

The following code can be used to read in this data.

txt'; INPUT Subject $ 1-4 Number 5-7 Students 8-9; PROC PRINT DATA=CourseData; Note that SAS

9 SPECIFYING COLUMNS WHEN READING RAW DATA INTO SAS We can specify exactly which columns should be used for the creation of specific variables. For example, consider the CourseData.txt file. The following code can be used to read in this data. DATA CourseData; INFILE '/folders/myfolders/coursedata.txt'; INPUT Subject $ 1-4 Number 5-7 Students 8-9; PROC PRINT DATA=CourseData; Note that SAS missed the value of Students for row 98 for some reason. External raw data Data read into SAS To fix this, ensure that numeric variables are right justified in the raw external file. 9

READING IN NONSTANDARD DATA At times our datasets may contain variables other than strictly numeric or character strings. For example, the following dataset contains a date.

10 READING IN NONSTANDARD DATA At times our datasets may contain variables other than strictly numeric or character strings. For example, the following dataset contains a date. Note: The variables in the above dataset remain in nice straight columns. We will take advantage of this when reading in the data. The following code can be used to read in this dataset. DATA faculty; INFILE '/folders/myfolders/faculty.dat'; INPUT Name $ Age Discipline $1. +1 Date MMDDYY10. (Rating1 Rating2) (4.1); PROC PRINT Data=faculty; Note the use of informats after the variable names in the INPUT statement. Discuss the purpose of each of the following components of the INPUT statement: Name $ Age 2. Discipline $1. Date MMDDYY10. (Rating1 Rating2) (4.1) 10

This default setting can be overridden (when printing) using a FORMAT statement in the PROC

11 A print-out of this dataset is given here. Note: The dates here are expressed as the number of days since January 1, This default setting can be overridden (when printing) using a FORMAT statement in the PROC PRINT procedure. We ll discuss this in detail in a future handout. Using a FORMAT statement to print actual dates: PROC PRINT Data=faculty; FORMAT Date MMDDYY10.; 11

MIXING INPUT STYLES AND DEALING WITH MESSY RAW DATA Consider the following dataset which has a mixture of

The dataset contains the following information for each park: Name, State(s), Year Park was established, and

To read in this data set, start by identifying the column location of the variables that are properly aligned.

12 MIXING INPUT STYLES AND DEALING WITH MESSY RAW DATA Consider the following dataset which has a mixture of input styles. The dataset contains the following information for each park: Name, State(s), Year Park was established, and Acreage. Some variables are in nice straight columns, but States and Year are not. To read in this data set, start by identifying the column location of the variables that are properly aligned. Then, use the following code: DATA NationalParks; INFILE '/folders/myfolders/nationalparks.txt' ; INPUT Name $ 1-22 State $ Acreage COMMA9.; PROC PRINT DATA=NationalParks; A print-out of the correct dataset in SAS is shown below: 12

13 Questions: 1. Change the input line as follows: INPUT Name $ State $ Acreage COMMA9.; What impact does this have on the result? 2. Change the input line as follows: INPUT Name $ 1-22 State $ Year Acreage COMMA9.; What impact does this have on the result? What did SAS do here? Discuss each of the following components of the INPUT Statement. Name $ 1-22 State $ Acreage COMMA9. 13

(verify this on your own). Note that SAS treats a tab differently than a space. 2.

characters, as well. Consider a typical log file from a website.

14 Comments: 1. If a tab is used in the creation of the raw data file, the data will not read in correctly (verify this on your own). Note that SAS treats a tab differently than a space. 2. When reading in messy data, symbol can be used to move across a line to find specific characters, as well. Consider a typical log file from a website. Suppose we want to read in the Date and Filename. Consider the following program and output: DATA Logfile; INFILE '/folders/myfolders/logfile.txt'; Date File :$53.; PROC PRINT DATA=Logfile; 14

15 Questions: 1. Change the INPUT line to read: Date File $; What is the effect of using this input line? Discuss. 2. Change the INPUT line to read: Date File $52.; What is the effect of using this input line? Discuss. 3. Change the INPUT line to read: Date File :$100.; What is the effect of using this input line? Discuss. 4. What is the purpose of the colon in our original input line? Discuss. 5. Verify whether or not SAS picks up the space after GET but before the / when reading in the file name. 15

OBSERVATIONS ON MULTIPLE LINES AND MULTIPLE OBSERVATIONS PER ROW Consider the following dataset that contains information about an observation across

The second line contains the variables Mother Minority and Mother s Age, and the third line contains the Apgar Scores from 1 minute and 5 minutes.

16 OBSERVATIONS ON MULTIPLE LINES AND MULTIPLE OBSERVATIONS PER ROW Consider the following dataset that contains information about an observation across multiple rows. The first line of each observation contains the variables Father Minority Status and Father s Age. The second line contains the variables Mother Minority and Mother s Age, and the third line contains the Apgar Scores from 1 minute and 5 minutes. The following code can be used to read in this data. DATA NC_Birth; INFILE '/folders/myfolders/datastep_ncbirth_multiline_2.txt'; INPUT FMinority $ FAge / MMinority $ MAge #3 Apgar1 Apgar5; PROC PRINT DATA=NC_Birth; 16

17 Tasks: 1. Rewrite the code above to read in only the following variables: FMinority, FAge, Apgar1, and Apgar5. Paste the code here. 2. Rewrite the code above to read in only FMinority and Apgar1, Apgar5. Paste your code here. 3. Rewrite the code above to read in only FMinority, Mage, Apgar1. Paste your code here. 17

Another issue that may arise is that multiple observations may be present in each row. In the following setup, each row contains two observations instead of one.

18 Another issue that may arise is that multiple observations may be present in each row. In the following setup, each row contains two observations instead of one. Once SAS reads in information for each variable, it automatically jumps to the next line. In the case above, we want to force SAS to continue reading information in the first line, since data from the second observations is also present in the first line. This can be done using in the INPUT line. DATA NC_Birth; INFILE '/folders/myfolders/datastep_ncbirth_multiobs.txt'; INPUT FMinority $ FAge MMinority $ MAge Apgar1 With the use of symbols, all observations could technically be contained on the first line. SAS will continue reading in variables from the first line until it reaches the end of this line. 18

We used the following program to read in the entire dataset: DATA NC_Birth; INFILE

19 READING PART OF A DATA FILE Consider once again the subset of the NCBirth dataset that is given here. We used the following program to read in the entire dataset: DATA NC_Birth; INFILE '/folders/myfolders/datastep_ncbirth.dat'; INPUT FMinority $ FAge MMinority $ MAge Apar1 Apgar5; The SAS print-out of the data is shown below. 19

Now, suppose the goal is to read in only information for which the Father Minority = Nonwhite. You could modify the code as follows: DATA NC_Birth; INFILE '/folders/myfolders/datastep_ncbirth.

20 Now, suppose the goal is to read in only information for which the Father Minority = Nonwhite. You could modify the code as follows: DATA NC_Birth; INFILE '/folders/myfolders/datastep_ncbirth.dat'; INPUT FMinority $ FAge MMinority $ MAge Apar1 Apgar5; IF FMinority = 'White' then DELETE; The SAS print-out of the data is shown below: Comment: The placement of the IF statement above requires all of the information for that observation get read in and then we immediately delete this information. This is inefficient. The following code instead reads in only the Father Minority status, then moves to the IF statement symbol does this). If the FMinority variable is deleted, SAS moves on to the next observation (i.e., it returns to the beginning of the data step). If FMinority exists for that observation, SAS moves on to the second INPUT statement and continues to read in information for that observation. DATA NC_Birth; INFILE '/folders/myfolders/datastep_ncbirth.dat'; INPUT FMinority IF FMinority = 'White' then DELETE; INPUT FAge MMinority $ MAge Apgar1 Apgar5; Task: Write code to read in only the observations for which FMinority = Nonwhite and MMinority = Nonwhite. Your code should require SAS to read in a minimum amount of information from each observation. Test your code in SAS and make sure it works. 20

The code we ve been using to read in this data will not work.

21 SPECIFYING OPTIONS IN THE INFILE STATEMENT The following data set is similar to one with which we ve been working; however, note that here the first row contains variable names. The code we ve been using to read in this data will not work. DATA NC_Birth; INFILE '/folders/myfolders/ncbirth_with_names.dat' ; INPUT FMinority $ FAge MMinority $ MAge Apar1 Apgar5; 21

The FIRSTOBS option in the INFILE statement can be used to tell SAS to skip over the first row when reading in the data file. DATA NC_Birth; INFILE '/folders/myfolders/ncbirth_with_names.

22 The FIRSTOBS option in the INFILE statement can be used to tell SAS to skip over the first row when reading in the data file. DATA NC_Birth; INFILE '/folders/myfolders/ncbirth_with_names.dat' FIRSTOBS=2; INPUT FMinority $ FAge MMinority $ MAge Apar1 Apgar5; Likewise, the OBS option can be used to STOP reading from a data file at some point. For example, the following would read in only the 2 nd and 3 rd line of the data file. DATA NC_Birth; INFILE ' /folders/myfolders/ncbirth_with_names.dat' FIRSTOBS=2 OBS=3; INPUT FMinority $ FAge MMinority $ MAge Apar1 Apgar5; On another note, observations that do not contain the same amount of information can be problematic, especially if this missing information is at the end of a data line. Suppose the data file had been constructed as follows. (Notice that putting a. in for the missing information would prevent any problems.) Our usual SAS code to read in this data is shown below. DATA NC_Birth; INFILE '/folders/myfolders/datastep_ncbirth_missing.dat'; INPUT FMinority $ FAge MMinority $ MAge Apar1 Apgar5; 22

These statements produce the following output: Note that SAS reached the end of the first data line before it had assigned values to all variables

The MISSOVER option in the INFILE statement can be used to overcome this issue.

23 These statements produce the following output: Note that SAS reached the end of the first data line before it had assigned values to all variables in the INPUT statement. By default, SAS will go to the next line to try to complete the first record. The MISSOVER option in the INFILE statement can be used to overcome this issue. This tells SAS to simply assign missing values to any remaining variables once it reaches the end of a data line. DATA NC_Birth; INFILE ' /folders/myfolders/datastep_ncbirth_missing.dat' MISSOVER; INPUT FMinority $ FAge MMinority $ MAge Apar1 Apgar5; PROC PRINT; 23

The following SAS code can be used to read in this data file.

24 READING IN DELIMITED FILES Consider again the Car Accidents data set. Note that this is a comma-delimited file which has variable names in the first row. The following SAS code can be used to read in this data file. DATA Car_Accidents; INFILE '/folders/myfolders/caraccidents.csv' DLM=',' FIRSTOBS=2; INPUT ID Gender $ SeatBelt$ Age $ CellPhone $ Usage $; The output for the first 10 observations is shown below. 24

25 Next, consider reading in a tab-delimited file. SAS uses standard ASCII codes for characters. For example, 09 is the ASCII equivalent of a tab, and the notation 09 X in the programming statements below refers to a hexadecimal 09 character. (To see more ASCII codes, you can visit The following code can be used to read the tab-delimited file. DATA Car_Accidents; INFILE '/folders/myfolders/caraccidents.txt' DLM='09'X FIRSTOBS=2; INPUT ID Gender $ SeatBelt$ Age $ CellPhone $ Usage $; 25

26 USING THE IMPORT PROCEDURE The PROC IMPORT procedure was introduced earlier in this handout when discussing the Import Wizard. At this point, we can better understand how the IMPORT procedure works. PROC IMPORT OUT= WORK.CarAccidents DATAFILE="/folders/myfolders/CarAccidents.csv" DBMS=CSV REPLACE; GETNAMES=YES; DATAROW=2; Note that lines 1, 2, and 3 are OPTIONS under the PROC IMPORT statement. Identify the purpose of each line: Code Purpose OUT= WORK.CarAccidents DATAFILE= "/folders/myfolders/caraccidents.csv" DBMS=CSV REPLACE; GETNAMES=YES; DATAROW=2; Reading data files from other formats can easily be done (e.g. JMP, SPSS, Excel, etc.). The import options vary between different file formats. For example, when Excel files are imported, information about which sheet the data is contained in must be specified. 26

USING LIBRARIES IN SAS All SAS data sets are stored in a folder or directory known as a SAS library. You can view the active libraries by clicking on the Libraries tab.

You can also create a new permanent library in SAS by clicking on the New Library icon.

27 USING LIBRARIES IN SAS All SAS data sets are stored in a folder or directory known as a SAS library. You can view the active libraries by clicking on the Libraries tab. The Work library is a temporary storage location for SAS data sets (and it is also the default library). Data sets in this library will be deleted once you end the SAS session. You can also create a new permanent library in SAS by clicking on the New Library icon. In the New Library window, enter the name of the library you want to create and the path to where you want your data to be stored. Keep in mind that SAS places the following restrictions on library names: Names can be at most 8 characters Names can contain only letters, numbers, and underscores When you are using the SAS University Edition, any libraries that you create must be assigned to a shared folder. You access your shared folder with this pathname: /folders/myfolders/. Finally, be sure to check the Re-create this library at startup box so that you don t have to define your library reference each time you start SAS. 27

28 You can also use the LIBNAME statement to create a new library. For example, the following statements would have produced the same library that was created above using the New Library window. LIBNAME Hooks '/folders/myfolders/'; Creating Permanent SAS Data sets All SAS data sets have a two level name such as WORK.Car_Accidents. To create a permanent data set in SAS, you can use this two-level naming option in the DATA statement while referencing any library other than WORK. For example, consider the following programming statements, which save the Car_Accidents data to the Hooks library. DATA Hooks.Car_Accidents; INFILE '/folders/myfolders/caraccidents.txt' DLM='09'X FIRSTOBS=2; INPUT ID Gender $ SeatBelt$ Age $ CellPhone $ Usage $; SAS also has the ability work directly with data sets when they reside in a SAS library. For example, the following code can be used to print the Car_Accidents data set from the Hooks library. PROC PRINT Data=Hooks.Car_Accidents; 28

VIEWING THE CONTENTS OF A SAS DATA SET As shown earlier, PROC CONTENTS can be used to view the contents of a dataset. Consider the following example. PROC CONTENTS DATA=Hooks.

characters). For example, consider the following programming statements. DATA Hooks.Car_Accidents; INFILE '/folders/myfolders/caraccidents.

29 VIEWING THE CONTENTS OF A SAS DATA SET As shown earlier, PROC CONTENTS can be used to view the contents of a dataset. Consider the following example. PROC CONTENTS DATA=Hooks.Car_Accidents; You can also use LABEL statements in the DATA step to specify labels for each variable (these labels can be up to 256 characters long, whereas variable names are limited to 32 characters). For example, consider the following programming statements. DATA Hooks.Car_Accidents; INFILE '/folders/myfolders/caraccidents.txt' DLM='09'X FIRSTOBS=2; INPUT ID Gender $ SeatBelt $ Age $ CellPhone $ Usage $; LABEL Gender = 'Gender of subject' SeatBelt = 'Whether a seatbelt was in use at time of accident' Age = 'Age of subject at time of accident' CellPhone = 'Whether a cell phone was in use at time of accident' Usage = 'How cell phone was used'; PROC CONTENTS data=hooks.car_accidents; 29

Chapter 2: Getting Data Into SAS

Chapter 2: Getting Data Into SAS Data stored in many different forms/formats. Four categories of ways to read in data. 1. Entering data directly through keyboard 2. Creating SAS data sets from raw data