%SYSFUNC( BRIDGE TO EXTERNAL DATA ) Hsiwei Yu (Michael), Northrop Grumman, Washington, D.C. ABSTRACT %SYSFUNC, a macro function introduced in SAS Version 6.11 and 6.09E, enables Base SAS functions to be called inside macro environment, thereby greatly enhances the power of SAS macro language. This study attempts to explore %SYSFUNC usage in retrieving data set and external file information via Base functions such as OPEN, FETCH, GETVARN, etc. Finally, %SYSFUNC applications arising from SAS/Graph and SQL are presented. INTRODUCTION %SYSFUNC is documented in SAS Macro Language: Reference, 1 st edition, 1997, with limited illustrations on its potential capability. Yindra (1998) showed using %SYSFUNC to access external file and directory information. He mentioned %SYSFUNC can read records from SAS data set, but no SAS programs were given. Yu (1998) gave outline about using %SYSFUNC to read data set and external file, but short of complete example. Imagine this assignment: For every character variable in a SAS data set, run Proc Freq; for numeric variable, run Proc Univariate; output from each variable s run must be saved to a separate external file; perform this standard analysis on every data set in any given data library. Central to this assignment is obtaining SAS data library and data set information, then generate the required language elements, i.e. Proc Univariate or Freq. Prior to %SYSFUNC, only Data Step can access external data; a Data Step generating dynamic code for later execution could have Macro or Base language elements all mixed in, at various resolution time, in double or single quotes, making it hard on the developer to concentrate on the program logic at hand. Using %SYSFUNC, external data is available in macro environment, and dynamic program generation via macro programming becomes easier. %SYSFUNC can access three sources of external data of interest to macro programmer, i.e. SAS data sets, external files, and directory entries (or members). %SYSFUNC syntax: %SYSFUNC( SAS_function( arguments ) <, format>); where SAS_function can also be a function written with SAS/Toolkit. Arguments can be a macro variable or text expression. If the returned value contains special character or mnemonic operators, such as %, &, AND, EQ, then use %QSYSFUNC to mask these characters. Format is optional and applies to specified SAS format to returned value. Data Step function returns 200 characters at most. A sample call: title %SYSFUNC( date(), weekdate18. ) Absence Report ; SAMPLE 1: READ A DATA SET %Read_Dat can be used as a basis for solving the aforementioned assignment. It deciphers variable information and generates Proc Freq or Univariate accordingly. These Base functions allow the reading of both data set descriptor and actual records. However, we can t write to SAS data set via %SYSFUNC in macro environment. OPEN opens a SAS data set CLOSE closes SAS data set ATTRC returns character value on the requested attribute of a SAS data set, such as SORTEDBY ATTRN returns numeric value on the requested attribute of a SAS data set, such as NOBS, ANOBS, ANY, and NVARS VARNUM returns the number of a data set variable. This number is used later by GETVARC and GETVARN VARTYPE returns the data type of a data set variable FETCH reads the next nondeleted record from data set into Data Set Data Vector (DSDV) FETCHOBS reads a specified record from data set GETVARC returns character variable s value GETVARN returns numeric variable s value MACRO TO READ A DATA SET %macro Read_Dat( dsn=, where= ); %if &where ne %then %do; %let dsn= &dsn(where=(&where)); %let dsid= %sysfunc(open(&dsn,is)); %if &dsid = 0 %then %do; %put Can%str(% )t open DSN=&dsn; %goto quit_bye; %let n_obs= %sysfunc( ATTRN( &dsid, nobs )); %put Current DSN=&dsn has &n_obs records;
%let N_Name= %sysfunc( VARNUM( &dsid, NAME )); %let N_Type= %sysfunc( VARNUM( &dsid, TYPE )); %do %while ( %sysfunc( FETCH( &dsid )) ne -1 ); %let NAME= %sysfunc( GETVARC( &dsid, &N_Name )); %let TYPE= %sysfunc( GETVARC( &dsid, &N_Type )); %if &TYPE = char %then %do; Proc Freq data= sasuser.houses; Tables &NAME; %else %do; Proc Univariate data= sasuser.houses; Var &NAME; %quit_bye: %let rc= %sysfunc( CLOSE( &dsid )); %mend Read_Dat; %Read_Dat( dsn= sashelp.vcolumn, where= %str( libname = SASUSER and memname = HOUSES )); Several Notes Since %SYSFUNC is a macro function, it is not necessary to enclose character values in quotes. Given a variable name as input, VARNUM returns its corresponding number. GETVARC and GETVARN use this number as input to return the variable s value. SAS view returns 1 as number of observations. With where clause in effect, it s simpler to use FETCH and test the return code value of 1 for end-of-file. Or, %let I= 1; %do %while ( %sysfunc( FETECHOBS( &dsid, &I )) = 0 ); %let I= %eval( &I + 1 );.. A data set opened in macro environment can still be read by other Proc, such as Print, it just cannot be updated. COMPARE WITH ALTERNATIVES Prior to %SYSFUNC, if we have to read a SAS data set and generate programs based on records in the data set, one solution could be: data _null_; file pgm_based_on_current_record.sas ; set sashelp.vcolumn( where = ( libname = SASUSER and memname = HOUSES )); if type = num then put '/* Run Proc Univariate */'; else put '/* Run Proc Freq */'; /* watch out UN-balanced quotes! */ %Include 'pgm_based_on_current_record.sas'; OR use Call Execute( ); such as data _null_; set sashelp.vcolumn( where = ( libname = SASUSER and memname = HOUSES )); if type = num then call execute( Proc Univariate data= asuser.houses; var name ; ); else call execute( Proc Freq data= sasuser.houses; tables name ; ); With Data Step approach, an external file must be written and later %Include d. For both the Data Step and Call Execute, the programmer must be careful with quotes. It could be cumbersome at times to count the single and double quotes. Comparing with Call Execute, remember if Call Execute produces macro language elements, they are executed immediately; if Call Execute produces SAS language elements, they are executed after the end of the current Data Step. With %SYSFUNC, since it is invoked in macro environment, timing is straightforward. Finally, macro program based on %SYSFUNC should be easier to read and comprehend. SAMPLE 2: READ AND WRITE EXTERNAL FILE %Write_Fi can be adapted to read a data set, and write to external file. %Read_Fil can be changed to read an external file of different record layout. Here's the sample data file; Street is 16 characters, Style 8, and Price varying length. Each field is separated by a space.
Sheppard Avenue RANCH 64000 Rand Street SPLIT 6 Market Street CONDO 80050 Garris Street TWOSTORY 10 FOPEN opens an external file FCLOSE closes a file, directory, or directory member FGET copies data from the File Data Buffer (FDB) FPUT moves data to the FDB starting at the current column position FWRITE writes a record to external file FREAD reads a record from external file into FDB FRLEN returns the size of the last record read, or the current record size for a file opened for output FPOS sets the position of the column pointer in the FDB MACROS TO READ AND WRITE EXTERNAL FILE %macro Write_Fi( out_fref=, in_dsn=); %let fid= %sysfunc( FOPEN( &out_fref, o )); %let dsid= %sysfunc(open(&in_dsn)); %let N_Street= %sysfunc( VARNUM( &dsid, street )); %let N_Style= %sysfunc( VARNUM( &dsid, style )); %let N_Price= %sysfunc( VARNUM( &dsid, price )); %let i= 1; %do %while ( %sysfunc( FETCH( &dsid )) = 0 ); %let Street= %sysfunc( GETVARC( &dsid, &N_Street )); %let Style= %sysfunc( GETVARC( &dsid, &N_Style )); %let Price= %sysfunc( GETVARN( &dsid, &N_Price )); %if %sysfunc( MOD( &i, 2 ), 1. ) = 0 %then %let Price= %sysfunc( INT( &Price / 10000 )); ; %let i= %eval( &i + 1 ); %let rc= %sysfunc( FPUT( &fid, &Street )); %let rc= %sysfunc( FPOS( &fid, 18 )); %let rc= %sysfunc( FPUT( &fid, &Style )); %let rc= %sysfunc( FPOS( &fid, 27 )); %let rc= %sysfunc( FPUT( &fid, &Price )); %let rc= %sysfunc( FWRITE( &fid )); %let rc= %sysfunc( FCLOSE( &fid )); %let rc= %sysfunc( CLOSE( &dsid )); %mend Write_Fi; %macro Read_Fil( in_fref= ); %let fid=%sysfunc(fopen(&in_fref,i)); %do %while ( %sysfunc( FREAD( &fid )) eq 0 ); %let rc= %sysfunc( FGET( &fid, STREET, 16 )); %let street= %sysfunc( TRIM( &street )); %let rc= %sysfunc( FPOS( &fid, 18 )); %let rc= %sysfunc( FGET( &fid, STYLE, 8 )); %let style= %sysfunc( TRIM( &style )); %let rc= %sysfunc( FPOS( &fid, 27 )); %* Compute the number of bytes occupied by PRICE; %let fld_len= %eval( %sysfunc( FRLEN( &fid )) - 26 ); %let rc= %sysfunc( FGET( &fid, PRICE, &fld_len )); %put Street=&STREET Style=&STYLE Price=&PRICE; %* Do something based on these current values; %let rc= %sysfunc( fclose( &fid )); %mend Read_Fil; filename data@txt data@txt unit=sysda space=(trk,(5,5)) recfm= fb lrecl= 80 blksize= 8000; %Write_Fi( out_fref= data@txt, in_dsn=sasuser.houses ); %Read_Fil( in_fref= data@txt ); Several Notes Fput simply prepares data in the FDB, it is Fwrite that finally outputs the FDB to the external file. %sysfunc( function( arguments(s)) <, format>); accepts optional format argument and
applies it to the functional result. By default, numeric result has format of BEST12. %sysfunc( FGET( &fid, STYLE, 8 )); then a macro variable, &STYLE, is automatically created and filled with its value. Fpos and Fget together are like using column position input in a Data Step. Fget function for reading text files are no substitutes for a Data Step. This example simply shows it can be done. SAMPLE 3: READ DIRECTORY ENTRIES %Dir_Read shows Base functions to retrieve directory and file attribute information. It reads a directory on Unix, Windows, or partition data set (PDS) on MVS. This macro reads all members of a directory then successively opens each file and writes out its contents. DOPEN opens a directory MOPEN opens a directory member file DINFO returns attribute value for a directory DOPTNAME returns a specified directory attribute DOPTNUM returns the number of attributes available to a directory FINFO returns a specified information item for a file FOPTNAME returns the name of an information item for a file FOPTNUM returns the number of attributes available to a file DNUM returns the number of members in a directory DREAD returns the name of a directory member MACRO TO READ DIRECTORY %macro Dir_Read( in_fref= ); %let dir_id= %sysfunc( DOPEN( &in_fref )); %let num_item= %sysfunc( DOPTNUM( &dir_id )); %put Info available on directory:; %do i= 1 %to &num_item; %let attr_nam= %sysfunc( DOPTNAME( &dir_id, &i )); %let attr_val= %sysfunc( DINFO( &dir_id, &attr_nam )); %put I=&i Attr_Name= &attr_nam Attr_Value=&attr_val; %let dnum= %sysfunc(dnum(&dir_id)); %do i= 1 %to &dnum; %* For each member, open and read; %let name= %sysfunc( DREAD( &dir_id, &i )); %put I=&i Memeber_name= &name; %let file_id= %sysfunc( MOPEN( &dir_id, &name )); %let finfo_nm= %sysfunc( FOPTNUM( &file_id )); %do j= 1 %do &finfo_nm; %let fin_nam= %sysfunc( FOPTNAME( &file_id, &j )); %let fin_valu= %sysfunc( FINFO( &file_id, &fin_nam )); %do %while ( %sysfunc( FREAD( &file_id )) = 0 ); %let rc= %sysfunc( FGET( &file_id, my_str, 200 )); %put %superq(my_str); %let rc= %sysfunc( FCLOSE( &file_id )); %let dsid=%sysfunc(dclose(&dir_id)); %mend Dir_Read; filename a.jcl.cntl disp=shr; * filename a d:\sas-programs ; %Dir_Read( in_fref= a ); Note With minor adjustment, it runs on UNIX, PC, and MVS. SAS Macro Language: Reference has a list of portable SAS language functions for %SYSFUNC. PROBLEM 1, DESCRIPTION We wish to show a bar chart representing sites with particular types of location information. Because region covers several states, state covers several counties, and so on, it is only natural to request that the bar representing region should be at the leftmost, and bars toward the right be progressively more geographically restrictive than that on the left. Here s the sample data Region Code 99.99 427259 427263 State Post Code 100.0 427263 427263 County Code Zip Code 99.76 426226 427263 99.92 426918 427263 City Name 99.99 427239 427263 Street Address Lat/Long 99.97 427152 427263 22.75 97223 427263
However, SAS/Graph defaults to alphabetical ordering for discrete bars. See Figure 1 for a simple Proc Gchart output, and Figure 2 for the desired graph. We need to tell Proc Gchart to use data set s original record sequence, without hard-coding. PROBLEM 1, SOLUTION * Data Step generating annotate statements not shown; %macro Midpnts; %let dsid= %sysfunc( OPEN( work.p1vb1, is )); %let nobs= %sysfunc( ATTRN( &dsid, nobs )); %let N_Id= %sysfunc( VARNUM( &dsid, id )); %do i= 1 %to &nobs; %let rc=%sysfunc(fetch(&dsid)); /* The next statement will be seen by Proc Gchart s midpoints= option */ "%sysfunc( TRIM( %sysfunc( GETVARC( &dsid, &N_Id ))))" %let dsid= %sysfunc(close(&dsid)); %mend Midpnts; %macro Values; %let dsid= %sysfunc( OPEN( work.p1vb1, is )); %let nobs= %sysfunc( ATTRN( &dsid, nobs )); %let var_num= %sysfunc( VARNUM( &dsid, id )); /* SAS language elements to be seen by Axis1 s value= option */ ( %do i= 1 %to &nobs; %let rc=%sysfunc(fetch(&dsid)); h= 1 a= 90 f= simplex "%sysfunc( TRIM( %sysfunc( GETVARC( &dsid, &var_num ))))" ) /* End of Axis1 s value= */ %let dsid= %sysfunc(close(&dsid)); %mend Values; pattern1 color= g value= Solid; * Other pattern statements not shown; axis1 value= %Values label= ( h= 1 f= simplex 'Type of Information' ); proc gchart data= p1vb1 annotate= anno; vbar id / sumvar= pct type= sum patternid= midpoint midpoints= %Midpnts quit; maxis= axis1; Discussions This data set was opened three times, first by macro generating midpoints= in Proc Gchart, second by macro generating value= in Axis1, and finally Proc Gchart reads it for graphing. We could have used a Data Step and written to two external files (one for use by midpoints= and one for value=), then the data set will be read only twice. It is a trade-off between execution efficiency and programming simplicity. %SYSFUNC can have nested Base functions calls, %sysfunc(trim(%sysfunc(getvarc( )))); Depending on context, the returned value can be directly used as Base language elements without first assigned to a macro variable. PROBLEM 2, DESCRIPTION Occasionally when I m beginning to analyze a new data set, I like to have a global view of the key variables distinctiveness. For example, sasuser.fitness has 3 distinct Group values, 14 distinct Age values, and 20 distinct values when combining Age and Group. The following SQL is the basic building block: proc sql; select count( distinct age ), count( distinct group ), count( distinct put( age, 8. ) put( group, 8. )) from sasuser.fitness; Since the concatenation operator,, is only applicable to character operands, numeric values are transformed by using the Put function. Also note each key variable is referenced twice by SQL, first in count distinct by itself, and second time in count distinct of all the key combinations. PROBLEM 2, SOLUTION Macro %T_uniq (shown in Appendix 1) is developed to take an arbitrary number of key variables and accumulate the result from a data set across multiple invocations. A sample run and output: %T_uniq( indata= sasuser.fitness, keys= age group, outdata= acum_fnl ); %T_uniq( indata= sasuser.houses, keys= style bedrooms street, outdata= acum_fnl ); proc print data= acum_fnl;
Dataset Attributes Values SASUSER.FITNESS TOTOBS 31 GROUP 3 AGE CTKEYS 14 20 SASUSER.HOUSES TOTOBS STYLE 15 4 BEDROOMS 4 STREET CTKEYS 15 15 The first half of %T_uniq is to break down the list of key variables, in &keys, into individual variable. The second half is to query sashelp.vcolumn view twice. The first read of sashelp.vcolumn is to count distinct on individual keys, and the second read is for concatenating together all the keys into a long, temporary, character variable for count distinct. The length of this variable can only be 200 characters. %sysfunc( REWIND( )) is used to reposition the column attributes view for the second read. Appendix 2 contains the equivalent by using Data Step writing to two external files. The sashelp.vcolumn view is read only once, but two macros are dynamically generated, with one being imbedded in the other. Though these two versions are about the same program length, I believe the Data Step alternative is less intuitive. That s why I decided to rewrite it using %SYSFUNC. SUMMARY %SYSFUNC macro function provides useful ways to bridge traditional Macro and Base language. It simplifies thing we had to do without it and offers more power to traditional macro programming. Make %SYSFUNC part of your tool set. For question and comments, or to obtain complete sample programs, please send email to Hsiwei Yu (Michael) at vhyu@netkonnect.net or yu.michael@epa.gov Reference: SAS Institute Inc. (1997), SAS Macro Language: Reference, 1 st edition, Cary, NC: SAS Institute Inc. Chris Yindra (1998), %SYSFUNC The Brave New Macro World, Proceedings of the Twenty-third Annual SAS Users Group Internal Conference Michael Yu (1998), %SYSFUNC( A Bridge.. ); DC SAS Users Group Newsletter, 1 st quarter 1998 Ian Whitlock (1997), Call Execute: How and Why, Proceedings of the Twenty-second Annual SAS Users Group Internal Conference Robert Virgile (1997), Magic With Call Execute, Proceedings of the Twenty-second Annual SAS Users Group Internal Conference ACKNOWLEDGEMENTS Special thanks to Tom Lewis of Lockheed Martin and Huang Liang for review and constructive critique, helping me to clarify points I was trying to express. SAS, SAS/Graph, are registered trademarks of SAS Institute Inc. in the USA and other countries. APPENDIX 1: T_UNIQ.SAS MACRO /* t_uniq.sas << a macro program >> Uses %sysfunc( ), and does not use external file for generating SAS programs for later execution. */ %macro t_uniq( indata=, keys=, outdata= ); proc datasets lib= work nolist memtype= data; delete orig_dif result; quit; %let inlib= %scan( &indata, 1,. ); %let inmem= %scan( &indata, 2,. ); %if &inmem = %then %do; %let inmem= &inlib; %let inlib= work; %let inlib=%sysfunc(upcase(&inlib )); %let inmem=%sysfunc(upcase(&inmem )); %let keys= %sysfunc(upcase(&keys )); /* Watch out %nrstr(%x) for actually quoting, or blocking any special meaning, the x character, so that macro will treat the x character as simply text. */ %let whr= libname eq "&inlib" and memname eq "&inmem" and name in %nrstr(%(); %let iter= 1; %let t&iter= %scan( &keys, &iter, %str( )); %do %while( &&t&iter ne ); %let whr= &whr "&&t&iter"; %let iter= %eval( &iter + 1 );
%let t&iter= %scan( &keys, &iter, %str( )); /* Again watch out the use of %nrstr(%x). */ %let whr= &whr %nrstr(%)); %** put whr=&whr..; /* The final WHERE string */ /* SASHELP.VCOLUMN is a view; related to dictionary.columns */ %let dsid= %sysfunc( OPEN( sashelp.vcolumn( where= ( &whr )))); %** put dsid= &dsid..; %let fmt= %sysfunc( VARNUM( &dsid, format )); %let nam= %sysfunc( VARNUM( &dsid, name )); %let typ= %sysfunc( VARNUM( &dsid, type )); proc sql; create table orig_dif as select "&inlib..&inmem" as memname length= 17, &iter as nokeys, count( * ) as totobs %do %while ( %sysfunc( FETCH( &dsid )) ne -1 ); %let name= %sysfunc( GETVARC( &dsid, &nam )); SQL */ as &name /* Below is for the PROC, count( distinct &name ) /* REWIND even works on a VIEW! */ %let rc= %sysfunc( REWIND( &dsid ));, count( distinct %do %while ( %sysfunc( FETCH( &dsid )) ne -1 ); %let type= %sysfunc( GETVARC( &dsid, &typ )); %let format= %sysfunc( GETVARC( &dsid, &fmt )); %let name= %sysfunc( GETVARC( &dsid, &nam )); %if &type eq char %then %do; /* Below is for the PROC SQL. Character variable */ &name %else %do; %if &format eq %then %do; /* Below is for the PROC SQL. Numeric var without format */ put( &name, best18. ) %else %do; /* Below is for the PROC SQL. Numeric var with format */ put( &name, &format ) A ) as ctkeys /* Close the data set as soon as possible */ %let dsid= %sysfunc( CLOSE( &dsid )); from &inlib..&inmem; quit; /* End of the PROC SQL */ proc transpose data= orig_dif out= result; copy nokeys memname; proc append base= &outdata new= result; label col1= # of Values _name_= Attributes ; %mend t_uniq; Appendix 2: Data Step Alternative %let keys= %sysfunc(upcase(&keys)); %let indata= %sysfunc( UPCASE( &indata )); data key_vars; set sashelp.vcolumn; where trim( libname ) '.' trim( memname ) = "&indata"; where also ( name in ( %let iter= 1; %let t&iter= %scan( &keys, &iter, %str( )); %do %while( &&t&iter ne ); "&&t&iter" %let iter= %eval( &iter + 1 ); %let t&iter= %scan( &keys, &iter, %str( )); )); %let iter= %eval( &iter - 1 );
%let DIR= %sysfunc(pathname( work )); filename a "&DIR/temp-10.sas"; filename b "&DIR/temp-11.sas"; options ls= 72; data _null_; set key_vars end= EOF; if _n_ = 1 then do; file b new; put %macro b; count( distinct( ; file a new; put proc sql; / create table orig_dif as / select / " &indata as memname length= 17," / " &iter as nokeys," / count( * ) as totobs, / %b ; end; file a; put, count( distinct name ) as name; file b; if type = char then do; put name ; end; else do; put put( name +(-1), best18. ) ; end; if EOF then do; file b; put " " )) as ctkeys / %mend b; ; file a; put "from &indata;" / quit; ; end; %inc b; %inc a;