General Tips for Working with Large SAS datasets and Oracle tables

Transcription

1 General Tips for Working with Large SAS datasets and Oracle tables 1) Avoid duplicating Oracle tables as SAS datasets only keep the rows and columns needed for your analysis. Use keep/drop/where directly on the data step set when reading an Oracle table to avoid bringing the entire Oracle table into SAS memory for processing. Alternately, use similar approach with proc sql. libname oralib oracle user = "grein006" orapw = xxxxxxx path = "ccge11p1" schema = "grein006" ; data local.ds; set oralib.table1(keep=var1 var2 var3 where=(var1 <5 and var2 in ( X, Y, Z ))); data local.ds; set oralib.table1(drop=var4 var5 var6); proc sql; create table local.ds as select * from oralib.table1(keep=var1 var2 var3) where var1 <5 and var2 in (X, Y, Z); proc sql; create table local.ds as select * from oralib.table1(keep=var1 var2 var3 where =(var1 <5 and var2 in (X, Y, Z))); 2) Pay attention to placement of subsetting in statements to reduce the number of observations (rows) processed and overall processing time. set all (keep = st cost1-cost10 ordate);

2 /*Subset first*/ if st = OR and year(datepart(ordate)) in (2000, 2001, 2003); /*Then calculate for reduced number of observations*/ set all (keep = st cost1-cost10 ordate); /*Calculate first on all observations*/ /*Then subset*/ if st = OR and year(datepart(ordate)) in (2000, 2001, 2003); EVEN BETTER: The following subsets on the WHERE clause which means the unneeded observations do not move from the input buffer to the Program Data Vector (PDV). set all (keep = st cost1-cost10 ordate where=( st = OR and year(datepart(ordate)) in (2000, 2001, 2003)); /*Then calculate for reduced number of observations*/ 3) Always use (keep=) or (drop=) on the source data set to reduce the width of the Program Data Vector (PDV) required to create the output dataset. data new(keep= a d); /*keep only final variables needed*/ set sourced(keep= a b c); /*reduce width of PDV by bringing only variables needed*/ d = function (b, c); 4) You can minimize the CPU time that SAS uses when you execute the minimum number of statements in the most efficient order. a) place the subsetting IF statement as soon as logically possible in a DATA step when creating a subset of data.

3 b) Place the statements in groups that are associated with IF-THEN/ELSE statements or SELECT/WHEN statements. Groups of statements execute only when a particular condition is true. Check the most frequently occurring condition first, and continue checking conditions in descending order of frequency. 5) Eliminate unnecessary passes through the data a) Use a single DATA step to create multiple output data sets from one pass of the input data, rather than using multiple DATA steps to process the input data each time that you create an output data set. data new1 new2; set origds; if var1= A then output new1; else output new2; b) Create sorted subsets by subsetting data within the SORT procedure rather than subsetting data in a DATA step and then sorting. proc sort data=origds out=newds; by var1 var2; where var3= xyz ; c) Change variable attributes by using PROC DATASETS rather than a DATA step. 6) Use (rename=) instead of assignment statements. Rename works on the descriptor portion of the dataset, not each observation. data new2 (rename = (st=state)); set bigdata (keep = st company); data new2 (drop = st); set bigdata (keep = st company); state = st;

4 7) Use logical OR rather than logical IN because SAS stops processing the OR list as soon as a comparison makes the expression true. If st = WA or st = OR or st = CA ; If st in ( WA, OR, CA ); BETTER YET: If you know that CA is most frequent and OR is least frequent, the following is better If st = CA or st = WA or st = OR ; 8) If we use WHERE clause on a PROC (print, means, etc.), then we can skip the dataset step and avoid all the system resources that requires. 9) Check the total clock time (real time) and CPU time used for programming steps (see.log file). Particularly for long running programs, experiment with these techniques and compare changes in processing times. 10) Delete temporary datasets after they are no longer needed in the program. This frees up the SASWORK area for ALL SAS users and avoids the possibility of running out of collective memory. proc datasets nolist; delete ds; 11) Clean up file system space OFTEN and avoid storing very large datasets unnecessarily. Check storage space of permanent datasets from within SAS using output from proc contents (see file size (bytes)) or at the UNIX prompt using

5 some file list option (ls lt, for example and size also shown in bytes) or using Windows explorer (shown in KB). 12) When storing large datasets, get into the habit of checking overall file system capacity with the UNIX command df k Check /ccge file system capacity percentage on the following line: linden_ccge_vol % /ccge