Other Data Sources SAS can read data from a variety of sources:

Transcription

1

2 Other Data Sources SAS can read data from a variety of sources: Plain text files, including delimited and fixed-column files Spreadsheets, such as Excel Databases XML Others

3 Text Files Text files of various types can be read via the data step. Two essential statements for this operation include: infile file-reference <options>; Directs the data step to a file. The file reference can be a path (in quotes) or a filename reference. input variable list; Lists variables to read, with some instructions for how to read them.

4 Basic Input With no options assigned, the infile statement will assume data values are space delimited. In this case, the most basic form of the input statement is: input variable_1 <$> variable_n <$>; $ is an indicator that the variable is character variable must follow SAS naming conventions

5 Simple Example Read the data file flights.prn from the raw data subfolder of the data sets folder:

6 Simple Example Code: data test; infile '\\seashare\blumj\sas Programming Data\raw data\flights.prn'; input flight_no date $ destination$ first_class economy; run;

7 Simple Example Code: data test; infile '\\seashare\blumj\sas Programming Data\raw data\flights.prn'; input flight_no date $ destination$ first_class economy; run; Infile statement includes a full-path reference to the file to be used.

8 Simple Example Code: data test; infile '\\seashare\blumj\sas Programming Data\raw data\flights.prn'; input flight_no date $ destination$ first_class economy; run; List (legal) variable names in order corresponding to columns in the data. Each character variable must be followed by the $. It can be appended to the name or a space can be placed between.

9 Results Something is not quite right in the date column.

10 data test; Process infile '\\seashare\blumj\sas Programming Data\raw data\flights.prn'; input flight_no date $ destination$ first_class economy; run; During compilation, the PDV is set up based on the specifications in the input statement. PDV flight_no date destination first_class economy

11 data test; Process infile '\\seashare\blumj\sas Programming Data\raw data\flights.prn'; input flight_no date $ destination$ first_class economy; run; During execution data is loaded into an input buffer and parsed as specified by the input and infile statements. Input Buffer / 1 1 / L A X PDV flight_no date destination first_class economy

12 data test; Process infile '\\seashare\blumj\sas Programming Data\raw data\flights.prn'; input flight_no date $ destination$ first_class economy; run; Spaces are taken as delimiters between values. Multiple, consecutive spaces are seen as a single delimiter Input Buffer / 1 1 / L A X PDV flight_no date destination first_class economy /11/20 LAX

13 Default length of character variables is 8 A length statement (before input) would be useful. data test; Process infile '\\seashare\blumj\sas Programming Data\raw data\flights.prn'; input flight_no date $ destination$ first_class economy; run; Input Buffer / 1 1 / L A X PDV flight_no date destination first_class economy /11/20 LAX

14 This record is output to the data set, and the process continues until the end of the raw file is reached. data test; Process infile '\\seashare\blumj\sas Programming Data\raw data\flights.prn'; input flight_no date $ destination$ first_class economy; run; Input Buffer / 1 1 / L A X PDV flight_no date destination first_class economy /11/20 LAX

15 The FILENAME Statement The filename statement looks similar to the libname statement: filename fileref 'path'; The rules for the fileref are the same as those for a libref. The path can point directly to an individual file or a folder. One method: filename myfile '\\seashare\blumj\sas Programming Data\raw data\flights.prn'; data test; infile myfile; input flight_no date $ destination$ first_class economy; run;

16 The FILENAME Statement A more flexible method: filename rawdata '\\seashare\blumj\sas Programming Data\raw data'; data test; infile rawdata('flights.prn'); input flight_no date $ destination$ first_class economy; run;

17 The FILENAME Statement A more flexible method: filename rawdata '\\seashare\blumj\sas Programming Data\raw data'; data test; infile rawdata('flights.prn'); input flight_no date $ destination$ first_class economy; run; Since the filename points to a folder, individual files must be requested when the fileref is used.

18 Pitfalls Consider the data file flights2.prn, which is only slightly different than the previous one (the zero is a blank):

19 Process at 9 th Record data test2; infile rawdata ('flights2.prn'); length date $ 10; input flight_no date $ destination$ first_class economy; run; Input Buffer / 1 5 / L A X PDV date flight_no destination first_class economy 12/15/ LAX 187 The multiple spaces are seen as a single delimiter (not as missing). What happens with economy?

20 Process at 9 th Record data test2; infile rawdata ('flights2.prn'); length date $ 10; input flight_no date $ destination$ first_class economy; run; Input Buffer / 1 5 / D F W PDV date flight_no destination first_class economy 12/15/ LAX SAS wants to fill in a value, so it gets more information from the raw file the next record

21 Results Only 9 records, and the 9 th is incorrect.

22 Pitfalls As a general rule, spaces are lousy delimiters consider flights3.prn which adds the pilot s name as the first column. What will happen with this?

23 Other Delimiters Instructions for which delimiter is present are set in the infile statement. Consider flights.csv :

24 Other Delimiters The dlm= option allows for specification of the delimiter, which can be a literal keyboard character. Code: data test3; infile rawdata('flights.csv') dlm=','; length date $ 10; input flight_no date $ destination$ first_class economy; run;

25 More Infile Options 9 th Record still problematic DSD: (delimiter sensitive data) Ignores delimiters inside quoted values. Treats consecutive delimiters as containing a missing value. Automatically changes the default delimiter to a comma. Update: data test3; infile rawdata('flights.csv') dsd; length date $ 10; input flight_no date $ destination$ first_class economy; run;

26 Results 9 th record now has appropriate missing first class value and 10 th record reads correctly.

27 More Infile Options Consider flights2.csv : Here the 9 th record is incomplete, with the current code the result is:

28 More Infile Options This comes from the flight number on the 10 th record.

29 More Infile Options This is missing since it tries to read the date (12/15/2000) here, which is not valid as a number. Once again, SAS tries to fill in information from the next record.

30 More Infile Options MISSOVER: Forces all values that cannot be filled with the information in the current input buffer to be set to missing. Update: data test4; infile rawdata('flights2.csv') dsd missover; length date $ 10; input flight_no date $ destination$ first_class economy; run;

31 Results 9 th record now has appropriate missing first class and economy values and 10 th record reads correctly.

32 More Infile Options In flights3.csv, column headers are preserved: The firstobs= option: Selects the first row to read from the raw data.

33 Firstobs=

34 TAB as a Delimiter TAB is not a standard keyboard character in the SAS editor it is just a series of spaces. For flights.txt (which is space delimited), what is the correct form of the dlm= option? ASCII hexadecimal codes can be given in the form: '##'x The hex code for TAB is 09 data test6; infile rawdata('flights.txt') dlm='09'x; length date $ 10; input flight_no date $ destination$ first_class economy; run;

35 TAB as a Delimiter See if you can read in flights2.txt, flights3.txt and flights4.txt.

36 Informats In the previous examples the date was read as character instead of what SAS recognizes as a date. Many formats can be used in a dual role as informats, which provide instructions for how to read raw data. Code: data test7; infile rawdata('flights.txt') dlm='09'x; input flight_no date:mmddyy10. destination$ first_class economy; run;

37 Informats In the previous examples the date was read as character instead of what SAS recognizes as a date. Many formats can be used in a dual role as informats, which provide instructions for how to read raw data. Code: The colon (:) is used as an operator data test7; to attach an informat to a variable. infile rawdata('flights.txt') dlm='09'x; input flight_no date:mmddyy10. destination$ first_class economy; run;

38 Informats The dates are converted, but no format is applied. Use a format statement to pick the format desired.

39 Fixed Column Files In flights.dat, data are in fixed positions: Flight number, columns 1-3 Date, columns 4-11 Destination, columns First Class, columns (or just 16 & 17) Economy, columns 18-20

40 Fixed Column Files Here the form of the input statement is quite different. Two potential forms can be used on each variable variable <$> start-stop start is the starting column position, stop is the ending column position. $ is used to denote character variables. This form cannot use variable informat. n is the starting column position The informat includes the total width The informat determines if the variable is character or numeric.

41 First version: Fixed Column Files data test8; infile rawdata('flights.dat'); input flight_no 1-3 date$ 4-11 destination$ first_class economy 18-20; run; In this case, information in the input buffer is parsed into pieces corresponding to the specified columns.

42 Result

43 Second version: Fixed Column Files data test9; infile rawdata('flights.dat'); flight_no date destination first_class economy 3.; format date date9.; run; In this case, information in the input buffer is parsed into pieces corresponding to the start column and format width.

44 Result

45 Fixed Column Files One method or the other can be chosen for each variable, but different variables can use different methods in the same input statement: data test10; infile rawdata('flights.dat'); input flight_no date mmddyy8. destination$ first_class economy 18-20; format date date9.; run; Date is the only variable that needs an informat, so the others can be read using the column span.

46 Exercise 1 Read the projects.txt data, which has the columns of: state, job id, date, region, equipment cost, personnel cost and pollution code. Compute a variable for total job cost and another for pollution type using the encoding: 1 = TSP 2 = LEAD 3 = CO 4 = SO2 5 = O3

47 Exercise 2 Read the employee list.csv data, which has the columns as noted at the right. In addition to the variables present, you should also compute a bonus and final salary for each employee. The bonuses are 9% for all pilots and 7.5% for all mechanics and 6.75% for all others. Description of Field Employee s Last Name Employee s First Name Country City Phone Number Employee ID Job Code Salary Data Type Character Character Character Character Character Character Character Currency ($xxx,xxx)

48 Exercise 3 Read the delay.dat data, which has the columns as noted at the right. The airline classifies departure times in one of four ways: midnight to 7 a.m. as early morning, 7 a.m. to noon as morning, noon to 6 p.m. as daytime and 6 p.m. to midnight as evening. Create an additional variable to reflect these classifications. Additionally, using the scheduled time of departure and the delay, define a variable that contains the actual departure time. Description of Field Columns Data Type Flight number 1-3 Character Departure date 4-10 ddmonyy Departure time Time/Character hh:mm Destination Character (3-letter airport code) Flight distance (mi) Numeric Ticketed passengers boarding Transfers from other flights Numeric Numeric Non-revenue passengers Numeric Aircraft Capacity Numeric Departure Delay (min negative indicates an early departure) Numeric