Analysis of variance and regression. November 13, 2007

Transcription

1 Analysis of variance and regression November 13, 2007 SAS language The SAS environments Reading in, data-step Summary statistics Subsetting data More on reading in, missing values Combination of data sets Lene Theil Skovgaard, Dept. of Biostatistics, Institute of Public Health, University of Copenhagen

2 SAS language, November SAS exercises on this course Two teachers to help you Private user names and passwords!! two share each machine many of you know SAS ANALYST from course on basic statistics, but here we focus on the SAS language References: Aa. T. Andersen, T.V. Bedsted, M. Feilberg, R.B. Jakobsen and A. Milhøj: Elementær indføring i SAS. Akademisk Forlag (in Danish, 2002) by Aa. T. Andersen, M. Feilberg, R.B. Jakobsen and A. Milhøj: Statistik med SAS. Akademisk Forlag (in Danish, 2002) R.P Cody og J.K. Smith: Applied statistics and the SAS programming language. 4. ed., Prentice Hall, SAS language, November Menus vs. Language Menus + No learning by heart + No syntax error + Stepwise learning Inflexible A bit hard to find your whereabouts Does not contain everything Tedious in the long run SAS language, November Menus vs. Language Language: Some learning by heart Many syntax errors in the beginning + Logical coherent + Reproducably + Easier to document + Easier to communicate

3 SAS language, November Basic structure SAS Core Database system ( Engine ) Programming language SAS Base Data manipulation: DATA, SORT, PRINT, (PLOT) Minimal statistics: MEANS, UNIVARIATE, TABULATE Special modules SAS/STAT: TTEST, GLM, GENMOD, etc. SAS/GRAPH: GPLOT SAS/ASSIST, QC, ETS, FSP, IML,... SAS ANALYST SAS Enterprise SAS language, November SAS in a nutshell Program + { } Raw data Data file { } Data file + Log Output Batch SAS: *.sas Program file *.log Log file *.lst Output file SAS language, November SAS Display Manager Environment for program development and data handling Program editor: common or enhanced Output window Log window Graphics window Explorer, Viewtable, Toolbar, Results Note: Program code must be saved

4 SAS language, November Example O Neill et.al. (1983): Lung function for 25 patients with cystic fibrosis. SAS language, November Some of these data may be found in the text file T:\pemax.txt (created using e.g. Wordpad) age sex height weight fev1 pemax SAS language, November Reading in data (more later on...) data sasuser.pemax; infile T:\pemax.txt firstobs=2; input age sex height weight fev1 pemax; To execute the program, we click on running man, and then we look at the log file NOTE: 25 records were read from the infile pemax.txt. The minimum record length was 21. The maximum record length was 21. NOTE: The data set SASUSER.PEMAX has 25 observations and 6 variables. NOTE: DATA statement used: real time 0.11 seconds cpu time 0.01 seconds No output

5 SAS language, November What if it did not work as intended? 1. Find out why! 2. Correct 3. Try again SAS is executed sequentially. If we want to add something, we can just do it later. Recall commands When a program bit has been executed, it may sometimes disappear from the program editor Earlier bits may be recovered using F4 Note, that the bits accumulate: If you use F4 several times, you will get the previous bits successively after one another SAS language, November Definition of new variables, transformation We want to study body mass index, bmi: data sasuser.pemax; infile T:\pemax.txt firstobs=2; input age sex height weight fev1 pemax; bmi=weight/(height/100)**2; proc print data=sasuser.pemax; Obs age sex height weight fev1 pemax bmi SAS language, November Transformations Arithmetics The usual operators: + - * / Raising to a power: **, e.g.. x**2 Square root: sqrt(x) Logarithms: log(x), log10(x), log2(x) All logarithms are proportional log 2 (x) = log(x) log(2) Relations: = < > <= >= <> (unequal) eq lt gt le ge ne (alternative notation) Logical operators: and or not

6 SAS language, November Other types of variable definitions data sasuser.pemax; infile T:\pemax.txt firstobs=2; input age sex height weight fev1 pemax; length csex $ 6 ; /* in order to avoid truncation */ if sex=1 then csex= male ; if sex=2 then csex= female ; fat=(bmi>18); proc print data=sasuser.pemax; var csex age bmi fat; Obs csex age bmi fat 1 female male male SAS language, November Ingrediences in DATA step Specification line (name of new data set) Data source (here: read from file) Variables to read in Possible calculations Possible redefinitions To be concluded with SAS language, November Variables The columns in a data set May be numerical variables (contain numbers) or character variables (contain text strings, letters) Values of a character variable is enclosed in citation signs, e.g. male (except in data files) Period (.) denotes a missing value for a numerical variable

7 SAS language, November Variable names SAS does not care about upper/lower case (SEX, sex and Sex refer to the same variable) Names may be up to 32 characters long (previously only 8) Names may contain English letters, digits and underscore (_) but they are not allowed to start with a digit SAS language, November Calculation of summary statistics in SAS proc means data=sasuser.pemax; The MEANS Procedure Variable N Mean Std Dev Minimum Maximum age sex fev pemax bmi These are default, others may be chosen as options SAS language, November From Help pages: /*Some of the keywords available with PROC MEANS: N - number of observations MEAN - mean value MIN - minimum value MAX - maximum value SUM - total of values NMISS - number of missing values MAXDEC=n - set maximum number of decimal places */ statistic-keyword(s) specifies which statistics to compute and the order to display them in the output. The available keywords in the PROC statement are Descriptive statistic keywords CLM RANGE CSS SKEWNESS SKEW CV STDDEV STD KURTOSIS KURT STDERR LCLM SUM MAX SUMWGT MEAN UCLM MIN USS N VAR NMISS Quantile statistic keywords MEDIAN P50 Q3 P75 P1 P90 P5 P95 P10 P99 Q1 P25 QRANGE Hypothesis testing keyword PROBT T

8 SAS language, November If we want to see the medians: proc means data=sasuser.pemax median; var age bmi fev1; The MEANS Procedure Variable Median age bmi fev Oops: Now, we got only the median! SAS language, November proc means data=sasuser.pemax N mean median; var age bmi fev1; The MEANS Procedure Variable N Mean Median age bmi fev SAS language, November Sorting the data often used because other procedures demand this Example proc sort data=sasuser.pemax out=sorted_pemax; by sex descending weight; If out=xxx is omitted, the original data set will be replaced by the sorted data. Note the option DESCENDING in front of weight

9 SAS language, November BY statement may be found in many procedures: (MEANS, REG, GLM,... ) performs the analyses within each group separately demands sorted data proc sort data=sasuser.pemax; by sex; proc means; where sex ne.; by sex; Remember to delete missing values, otherwise they will form a separate group SAS language, November Output data set proc sort data=sasuser.pemax; by sex; proc means noprint data=sasuser.pemax mean; by sex; var age bmi fev1; output out=summa mean=mage mbmi mfev1; proc print data=summa; /* a temporary data set */ Obs sex _TYPE FREQ_ mage mbmi mfev SAS language, November Alternative procedure: UNIVARIATE proc univariate data=sasuser.pemax normal; var bmi; a lot of output... (shown on the next page) Several tests for normality created by the option normal: Tests for Normality Test --Statistic p Value----- Shapiro-Wilk W Pr < W Kolmogorov-Smirnov D Pr > D > Cramer-von Mises W-Sq Pr > W-Sq > Anderson-Darling A-Sq Pr > A-Sq >0.2500

10 SAS language, November The UNIVARIATE Procedure Variable: bmi Quantiles (Definition 5) Moments Quantile Estimate N 25 Sum Weights 25 Mean Sum Observations Std Deviation Variance Skewness Kurtosis Uncorrected SS Corrected SS Coeff Variation Std Error Mean Location Basic Statistical Measures Variability 100% Max % % % % Q % Median % Q % % % % Min Mean Std Deviation Median Variance Mode. Range Interquartile Range Lowest Extreme Observations -----Highest----- Value Obs Value Obs Tests for Location: Mu0=0 Test -Statistic p Value Student s t t Pr > t <.0001 Sign M 12.5 Pr >= M <.0001 Signed Rank S Pr >= S < SAS language, November Tables Two-way tables (cross tabulation): One-way tables: proc freq; tables csex; Cumulative Cumulative csex Frequency Percent Frequency Percent female male proc freq; tables csex*fat / nopercent nocol; Table of csex by fat csex fat Frequency Row Pct 0 1 Total female male Total SAS language, November Filtering data (selecting subsets) In DATA-step regarding observations: IF, WHERE, DELETE regarding variables: DROP, KEEP In procedures Regarding observations: WHERE Regarding variables: VAR-statement (depending on procedure)

11 SAS language, November WHERE If we only want to look at the girls: data pemax; /* temporary data set */ set sasuser.pemax; where csex= female ; proc print data=pemax; var csex age bmi; Obs csex age bmi 1 female female female female SAS language, November IF, DELETE Alternative ways of writing: if csex= female ; if csex ne male ; Look out, if data contains missing values! if sex= male then delete; SAS language, November Filterings may be combined: data pemax; set sasuser.pemax; where csex= female and age>12; proc print data=pemax;; var csex age bmi; Obs csex age bmi 1 female female female female female female female

12 SAS language, November DROP and KEEP Now that we have bmi, we may not need height and weight: data pemax; set sasuser.pemax; drop height weight; If you only want to keep a single variable: data pemax; set sasuser.pemax (keep=bmi); SAS language, November WHERE in procedures If we for a specific procedure only want to look at the girls (but continue to work with all data): proc print data=sasuser.pemax; where csex= female ; var csex age bmi; Obs csex age bmi 1 female female female female Here, we have created no new data set, neither temporary nor permanent. SAS language, November Reading in from file data lines directly in program character variables columns or free fromat data seperation missing values import from Excel

13 SAS language, November Data lines directly in program data sasuser.pemax; input age sex height weight fev1 pemax; bmi=weight/(height/100)**2; datalines; ; SAS language, November Reading in character variables data sasuser.pemax; length sex $ 6; /* to avoid truncation */ input age sex $ height weight fev1 pemax; bmi=weight/(height/100)**2; datalines; 7 male female male male ; SAS language, November Semicolon separated data Until now, data have been nicely separated by blanks. Now it looks a bit different... age;sex;height;weight;fev1;pemax 7;male;109;13.1;32;95 7;female;112;12.9;19;85 8;male;124;14.1;22;100 8;female;125;16.2;41; data sasuser.pemax; * we now specify a list of possible delimiters; infile pemax2.txt firstobs=2 dlm= ; ; input age sex $ height weight fev1 pemax;

14 SAS language, November Formatted input Now, the values are not separated at all! (often useful for many binary observations, e.g. questionnaire data) data sasuser.pemax; length sex $ 1; input age 1-2 sex $ 3 height 4-6 weight 7-10 fev pemax 13-15; datalines; 7M F M M ; SAS language, November Missing values Numeric variables (numbers) must be. (period) Character variables (letters): NA, missing value. etc. blanks?? will of course not work if blanks are used as delimiters Take care with -9, 999 etc. SAS language, November Example 1: List input, numeric variables data sasuser.pemax; input age sex height weight fev1 pemax; datalines; ;

15 SAS language, November Example 2: Formatted input, blanks data sasuser.pemax; length sex $ 1; input age 1-2 sex $ 3 height 4-6 weight 7-10 fev pemax 13-15; datalines; 7M F ; SAS language, November Example 3: Semicolon separated, no symbols for missing data sasuser.pemax; infile pemax2.tal firstobs=2 dlm= ;, dsd; input age csex $ height weight fev1 pemax; bmi=weight/(height/100)**2; datalines; 7;male,;13.1;32;95 7;female;112;12.9;19;85 8;;124;14.1;22; ; Option dsd means: DSD Changes how SAS treats delimiters when list input is used and sets the default delimiter to a comma. When you specify DSD, SAS treats two consecutive delimiters as a missing value. SAS language, November General options (e.g. in first line of the program): options obs=100 nocenter linesize=75 pagesize=60 missing= - ; OBS=: Number of observations (from start) to be included in the analyses CENTER/NOCENTER: Output appearence LINESIZE= Maximum number of characters on each line (at most 256) PAGESIZE= Maximum number of lines on each page MISSING= Specifies the symbol for missing values (default is.)

16 SAS language, November Excel files may be imported directly to SAS proc import datafile= tables.xls out=sasuser.tables dbms=excel2000 replace; getnames=yes; SAS language, November SAS is a programming language A program is like a knitting recipe : A series of instructions which have to be executed in a specific order. Note: SAS is not a spread sheet. Output does not change if you change the data (rerun the program to do this) SAS language, November A simple SAS program data sasuser.pemax; \ infile N:\pemax.txt firstobs=2; input age sex height weight fev1 pemax; Data Step bmi=weight/(height/100)**2; / proc print data=sasuser.pemax; \ proc means data=sasuser.pemax; Proc Steps var age bmi; /

17 SAS language, November DATA steps and PROC steps A SAS program distinguishes between two types of operations ( steps ) DATA steps which define data sets, reading from text files, calculation of derived variables, selection of cases, etc. PROC steps which contain standard procedures, operating on data sets. Note: it is in general not possible to calculate anything in a PROC step. Traditionally, a SAS program is arranged so that the DATA step is at the top, but they may be mixed, if you define new data sets along the way. SAS language, November Basics regarding SAS language Almost everything (except for calculations in a data step) starts with a keyword and ends with a semicolon Statements are bits of code separated by semicolon OPTIONS ls=80; PROC GLM data=sasuser.pemax; MODEL height = age / solution; RUN; Keywords: OPTIONS, PROC, GLM, MODEL, RUN Certain statements belong together in blocks SAS language, November Things to keep in mind The slash (/) is often used to mark the start of options Semicolon and slash are necessary: solution is not a variable name and run is not an option.

18 SAS language, November Formatting code / designing programs SAS generally does not care about extra blanks and line shifts It is however considered good practice to write at most one statement on each line Indenting facilitates the reading considerably. (Sooner or later, you will end up reading your own old code!) SAS language, November How to organize your analyses etc.? If only we knew. But it is to some extent a matter of taste. Some thoughts, though... Interactive program execution is easy, but may be dangerous! Do you remember what was done? What if you have to just correct a few data values? Remember to save the code, at least for the most important analyses. Collect bits and pieces to a more coherent program and save it as a.sas-file, which can stand alone Look out for carrying over effects when using Display Manager. Try out your program in a fresh SAS session. Save also the log-files. They are as important as the output files. SAS language, November SAS libraries SAS is born with four libraries, the most important being WORK and SASUSER Look at Properties in the Explorer to see exactly where they are located WORK is a temporary library, which means that it disappears (with all its contents) when SAS is closed. These data files are denoted work.pemax or simply pemax SASUSER is permanent. Data sets stored here will be there also next time you enter SAS. These files are denoted sasuser.pemax

19 SAS language, November Private libraries, LIBNAME If you want a separate library for each project (surely the best in the long run), you will have to use LIBNAME statement, e.g. libname mysas p:\paper1\sasdata ; data mysas.pemax; infile... Note: The folder has to be created before use! This LIBNAME statement may be saved in the autoexec.sas (too advanced for this course) SAS language, November SAS data set First part (before the period) is the SAS-library The second part is the name of the data set If the first part is omitted, the library is taken to be WORK Seen from Windows point of view, the data files have the extension sas7bdat Data sets have two logical parts, a describing part and the data itself PROC CONTENTS resp. PROC PRINT will show these SAS language, November PROC CONTENTS The CONTENTS Procedure Data Set Name: SASUSER.PEMAX Observations: 25 Member Type: DATA Variables: 6 Engine: V8 Indexes: 0 Created: 14:13 Wednesday, Observation Length: 48 April 14, 2004 Last Modified: 14:13 Wednesday, Deleted Observations: 0 April 14, 2004 Protection: Compressed: NO Data Set Type: Sorted: NO Label:

20 SAS language, November Engine/Host Dependent Information----- Data Set Page Size: 8192 Number of Data Set Pages: 1 First Data Page: 1 Max Obs per Page: 169 Obs in First Data Page: 25 Number of Data Set Repairs: 0 File Name: /saswork/sas_workc3ca _ rasch/pemax.sas7bdat Release Created: M0 Host Created: SunOS Inode Number: 7884 Access Permission: rw-r--r-- Owner Name: lts File Size (bytes): Alphabetic List of Variables and Attributes----- # Variable Type Len Pos age Num fev1 Num height Num pemax Num sex Num weight Num 8 24 SAS language, November Combination of data sets vertically : more cases, identical variables horisontally (merging): new variables, same cases More cases is easy: It is possible to have more data sets in the same SET statement of a data step. Ex. Combining two groups: data all; set group1 group2; If the data sets do not have exactly the same variables, missing values are filled in SAS language, November Merging (horisontal combination) the data sets ought to have a common key variable, e.g. id all data sets have to be sorted according to id proc sort data=info1; by id; proc sort data=info2; by id; data info; merge info1 info2; by id; BY may be omitted, but only if data are complete and are sorted in the same way (not recommended, take good care!)

21 SAS language, November Lists of variables Sometimes you may want to refer to many variables at one time, e.g. in case of repeated measurements or just many connected variables proc freq; tables ques1-ques392; names looking alike: x1-x20 SAS order of appearance: age--weight All character variables: _CHARACTER_ All numerical variables: _NUMERIC_ All variables: _ALL_ SAS language, November Formats: Information of how to read or write a variable Built-in formats (numerical, dates, characters) User defined formats Why use formats? Nicer looking output Grouping for creation of tables Protection against errors in data SAS language, November Formats, continued standard formats: 10.3, best12., E12., $10., date10., yymmdd10.. allways contain a period (keep that in mind!) is associated permanently with the variable in the DATA step, or: is specified ad hoc with a FORMAT statement in PROC steps. User defined formats are created with PROC FORMAT

22 SAS language, November Example of use of formats proc format; invalue sexin M =1 F =2; value sexout 1= male 2= female ; data; informat sex sexin.; format sex sexout.; input sex; datalines; M F m ; proc print; which creates the output: Obs sex 1 male 2 female 3. SAS language, November Example of date formats: data; input x yymmdd10.; /* this covers several formats */ format x yymmddd.; cards; /1/ ; proc print; Obs x The actual value is time in days since 1/ SAS language, November Times in SAS Example with longitudinal measurements of blood pressure during the day: data longitudinal; informat time TIME10.2; input person time bp; datalines; 1 04:45: :21: :15: :18: ; Obs time person bp proc print data=longitudinal;

23 SAS language, November We wish to refer to time since start of treatment First we have to pick out the starting times, i.e. the first observations for each individual: data starttimes; set longitudinal; by person; if first.person; start=time; proc print data=starttimes; Obs time person bp start SAS language, November and then we have to merge the two data sets, and calculate differences: data merge_two; merge longitudinal starttimes; by person; timer=(time-start)/60**2; proc print data=merge_two; Obs time person bp start timer