Going Under the Hood: How Does the Macro Processor Really Work?

Going Under the Hood: How Does the Really Work? ABSTRACT Lisa Lyons, PPD, Inc Hamilton, NJ Did you ever wonder what really goes on behind the scenes of the macro processor, or how it works with other parts of the SAS system? With the knowledge of the macro processing sequence of events, programmers can better understand how macro processing works Since most macros are generic ones, this particular knowledge can save valuable development, debugging, and validation time This paper will help you to understand the timing of macro processing and its relationship with SAS language processing INTRODUCTION The SAS macro facility is a component of base SAS The macro facility is a tool for text substitution, which reduces the amount of text entered for common tasks The SAS macro facility works in conjunction with base SAS to build and execute programs The macro facility has two main components: The macro processor, which is the portion of the system that does the work and; macro language, the syntax used to communicate with the macro processor The technical aspects are not required for most macro programming, however understanding the concepts of how the macro processor works with other parts of the SAS System can help improve programming productivity The concepts in this paper present a logical representation of macro processing MECHANICS OF MACRO PROCESSING Understanding Tokens When a program is submitted, it goes to an area of memory called the input stack The input stack holds a SAS program while it waits for processing by the word scanner The word scanner is a component that examines the stream of characters from the input stack and assembles them into tokens The area within the word scanner which holds the tokens is called the word queue Tokens are the smallest pieces of information that are meaningful to the SAS language This process is called tokenization The word scanner then determines the destination of the tokens (DATA step compiler, macro processor, SAS procedures, etc) There are four general types of tokens: Literal: A string of characters enclosed in single or double quotation marks my program or my program Numbers: Digits, date/time values and hexadecimal numbers 345, 07JUL04, 0 x Names: A string of characters beginning with an underscore or letter and continuing with letters (words in your programs) Data, _test, linesleft, proc, _n_, if Special: Characters other than letters, numbers or underscore which have meaning to the SAS system / + - * ; $ ( ) & % = The underlying building blocks of a SAS program are the tokens the word scanner creates from your language statements Each word, literal string, number or special character is a token A token ends when the word scanner encounters either a blank space after a token or the beginning of a new token Two special symbol tokens signal the word scanner to send information to the macro processor The ampersand (&) and the percent sign (%) followed by an underscore or character triggers the macro processor to examine the tokens associated with the & or % - -

Tokenization of a SAS Program Without Activity In the example below as the word scanner pulls the first token from the input stack it recognizes it as the beginning of the DATA step and triggers the DATA step compiler, which requests more tokens The compiler pulls the tokens from the word scanner and checks the syntax of the tokens It continues to do this until it reaches the end of the DATA step boundary, which in this case is the semicolon after the run statement Figure Word scanner obtains tokens and determines destination SCL Compiler Command DATA Step Compiler data age Word queue data age 3 ( 4 drop 5 = 6 birthdt The word scanner reads the information from the input stack word by word and places the information into the word queue within the word scanner The word queue contains six positions labeled to 6 Word is the top of the queue, while word 6 is the bottom As the word scanner continues to pull from the input stack and place into the word queue, the first word moves to the top of the queue and so on ) ; set demog; age=(visitdt-birthdt)/365; Tokenization of a SAS Program With Activity All SAS programs are compiled and executed in the same manner Once a program is submitted, the statements are held in the input stack and wait for processing by the word scanner The word scanner examines the stream of characters from the input stack, tokenizes them, then determines their destination The appropriate compiler does syntax checking and passes the compiled statements, at boundary steps, on for execution When the word scanner detects a macro trigger, ampersand (&) or percent (%), it sends information and temporarily turns processing over to the macro processor The word scanner interrupts while the macro processor is active As a result, macro language processing occurs after tokenization, but before compilation The macro processor converts the macro language into language statements and places it back to the top of the input stack The word scanner continues by tokenizing the newly built SAS language statements that have been submitted by the macro processor Figures through 5 illustrate the process of tokenization with macro activity SAS creates a symbol table at the beginning of each session to hold in memory automatic and global macro variable values The value of the macro variable is stored in the symbol table for the duration of the SAS session See section Symbol Tables for further detail - -

Figure processor and symbol table SCL Compiler Command DATA Step Compiler 3 4 5 6 SAS Automatic variables SYSTIME 09:00 %let file=demog; data age; set &file; age=(visitdt-birthdt)/365; In figure 3 the word scanner starts at the first character in the input stack and recognizes the percent (%) sign as a macro trigger The word scanner turns control over to the macro processor for processing The macro processor removes the %LET statement, and places the entry in the symbol table Figure 3 processor examines the %LET statement and writes to the symbol table 3 4 5 % 6 let User-Defined Variables FILE DEMOG file=demog; data age; set &file; age=(visitdt-birthdt)/365; - 3 -

When the macro processor is finished, control is turned back over to the word scanner and tokenization resumes In Figure 4, the DATA keyword triggers the DATA Step compiler The word scanner continues to submit tokens until it comes to the macro trigger &FILE The word scanner turns control back over to the macro processor The macro processor concludes that FILE is a macro variable, which exists in the user-defined section of the global symbol table The macro processor replaces the macro variable (FILE) in the input stack with the associated text from the user-defined section of the symbol table (DEMOG) After the text substitution, control is turned back over to the word scanner as shown in Figure 5 The DATA Step compiler continues to request tokens and check syntax If there are no syntax errors in the step, the step is translated into execution The word scanner continues the process until the entire input stack has been read Figure 4 Word scanner resumes tokenization, macro processor substitutes text in input stack DATA Step Compiler data age; set 3 & 4 file User-Defined Variables FILE DEMOG The word scanner recognizes the macro trigger & and turns control over to the macro processor demog; age=(visitdt-birthdt)/365; If the end of the input stack is a DATA step boundary and there are no syntax errors, the compiler compiles and executes the step as shown in Figure 5 SAS then frees the DATA step task Certain user-defined macro variables created within a program remain in the global symbol table for the duration of the SAS session (detail to follow) If the end of the input stack is not a step boundary (eg more statements), the processed statements remain in the compiler until it reaches a boundary in which it executes the statements Figure 5 Word scanner completes processing DATA Step Compiler data age; set demog; age=(visidt-birthdt)/365; run Symbol Table User-Defined Variables FILE DEMOG ; 3 4-4 -

MACRO VARIABLES Basic Concepts of Variables variables are tools that allow you to write reusable programs The length of a macro variable is determined by the text assigned so its length varies with each value it contains variables contain only character data, however there are options to evaluate numbers when needed Depending on the type of macro variable, it will remain the same for the entire SAS session unless explicitly changed Some basic characteristics of macro variables include: Certain macro variables can be referenced anywhere in a SAS program (other than within data lines) variables names must be a valid SAS name (not a reserved SAS word) and start with a letter or underscore They can be used in either open code (defined outside a macro definition) or inside a macro definition or program variables are not dataset variables and do not relate directly to observations in a dataset, they are tools that allow you to write reusable code There are two types of macro variables, automatic and user-defined: Automatic macro variables are defined by the SAS system at the start of each session Most automatic variables contain information about your session, such as date, time when session opened, version, etc User-defined macro variables are defined by the programmer and are specific to the programmers needs Every macro variable has a domain or scope A macro variable s domain determines how values are assigned and how the macro processor resolves references Domains can be nested, and numerous macro variables can be nested and executed within macro programs %macro first(dsout=); SAS code %macro second(var=); SAS code %mend second; Second Domain First Domain Domains or also known as scopes %mend first; SYMBOL TABLES variables are stored in an area of memory known as symbol tables Symbol tables simply hold in memory the macro variable name and its value There are two types of symbol tables, global and local The global symbol table is automatically created at the beginning of each SAS session Global symbol tables contain macro variables that are defined in either open code (outside of a macro definition) or created automatically by the SAS system variables stored in the global symbol table remain the same for the entire SAS session Global macro variables are also available to any part of the SAS session Global macro variables include: All automatic variables (except SYSPBUFF) All macro variables created in open code All macro variables created with %GLOBAL Most macro variables created by the CALL SYMPUT - 5 -

Figure 3 Illustration of the User-Defined Variables FILE DEMOG (BEFORE macro processing) %let file=demog; data age; set &file; if age <=50; (AFTER macro processing) data age; set DEMOG; if age <=50; Global macro variables can be created at any time during a SAS session The values of global macro variables may be changed at any point during a session (except for some automatic SAS variables) Exceptions that preclude referencing the value of a global macro variable are: variables that exists in both global and local symbol tables The macro processor will search for the value in the local symbol table first s variables created with the CALL SYMPUT routine in a DATA step cannot be referenced with an & until the program reaches a step boundary Unlike global symbol tables, which are created at the beginning of the SAS session, local symbol tables are created at the time the word scanner identifies the %macro trigger Therefore, user-defined macro variables created within individual macro programs are stored in the local symbol table only during the execution of the macro program Once the macro program ends, the local macro symbol table and variables are deleted s defined in macro programs that are not explicitly defined as global are typically held in a local symbol table During tokenization the word scanner identifies a macro trigger and temporarily turns processing over to the macro processor Key macro language statements help the macro processor identify when to add a variable to the global symbol table or to create a temporary local symbol table Local symbol tables are empty until the macro program creates at least one macro variable The following can create local symbol tables The presence of one or more macro parameters A macro variable created in %LOCAL statements that define macro variables such as %DO, %LET (within a macro program and when a global macro variable with the same name is not already created) Before the macro processor creates or assigns a value to a macro variable it searches the symbol table to see if the variable already exists The process starts with the local symbol table and if necessary moves to the global symbol table It is important to understand the process of how macro variables are assigned and resolved When the macro processor encounters a macro program trigger (%macro), the processor automatically creates the local symbol table and stores the table in an area of memory Consider the following example in Figure 3-6 -

Figure 3 Illustration of a Local Symbol Table (Before and After Processing) Global symbol table created at the beginning of SAS session User-Defined Global Variables FILE DEMOG Automatic system variables %LET file=demog in open code %macro triggers the creation of the empty local symbol table Local Symbol Table Local Symbol Table VAR SYSBP Word Scanner (BEFORE macro processing) %macro vitinfo(var=); data vitals; set datavitals(keep=pid visit &var); (AFTER macro processing) data vitals; set datavitals(keep=pid visit SYSBP); %mend vitinfo; %subset(var=sysbp); The macro call %subset(var=sysbp) triggers the macro processor to assign the macro variable value in the local symbol table In Figure 3, the word scanner pulls the code (character by character) from the input stack and begins tokenization Once the word scanner recognizes the %macro statement, it triggers the macro processor to temporarily take over control The macro processor creates a local symbol table As it continues to process the macro program it checks to see if the variables defined exist in the local symbol table If the variables do not exist in the local symbol table, the processor continues to check all domains until it determines if the variables already exist in either another domain or the global symbol table If it finds the macro variable in either a local symbol table or global symbol table it then resolves the variable But what about references to macro variables outside of a macro program that is not defined by a %LET (in open code) or %GLOBAL statement? Since the local symbol table is deleted immediately after execution of the macro program, any macro variable referenced outside of a macro program is no longer able to resolve and issues a warning in the log Take the following example; - 7 -

SAMPLE CODE: %macro aegrade(dsout=, var=); data &dsout; set adverse; if &var <=3; %mend aegrade; Because the sample code makes reference to macro variables outside of the macro program, the following warning would appear in the SAS log; WARNING: Apparent symbolic reference VAR not resolved %aegrade(dsout=aesev, var=severity); data ae; set adverse; if &var = ; In order for macro variables to be defined throughout the SAS session the user would need to define the macro variable using the %GLOBAL statement The %GLOBAL statement creates macro variables that are available in all referencing environments Sending Symbol Table Values to the SAS Log During the development of macros, it may be useful to write all or part of the contents of the local and global symbol tables to the SAS log Using either the SAS system option SYMBOLGEN or the %PUT statement with the following options can accomplish this Using %PUT Statement The %PUT statement is part of the macro language and does not need to be part of a DATA or PROC step It only displays text and information about macro variables in the log The %PUT statement can be written as follows: %put &macro_variable_name; To help distinguish it in the log you could add text such as: %put MY MACRO VARIABLE=&macro_variable_name; In addition to writing individual values to the SAS log, there are additional options that allow you to write all macro variables, automatic SAS variables, global or local as well as user macro variables to the log The options along with the %PUT are: _ALL AUTOMATIC GLOBAL LOCAL USER_ lists all defined macro variables regardless of scope lists all automatic SAS variables lists all user-defined global variables lists all user-defined local macro variables defined within the current executing macro lists all user-defined macro variables (global and local) Using the SYMBOLGEN Option When the SYMBOLGEN option is enabled, the results of the resolution of the macro variables are displayed in the log SYMBOLGEN displays the value of a macro variable directly before the statement with the macro variable reference The SYMBOLGEN option helps the user debug macro code, but does display all of the macro variables, where the %PUT statement allows you to selectively display macro variables - 8 -

The following statement enables SYMBOLGEN: options symbolgen; %let file=demog; %let tabvar=gender; proc freq data=data&file; table &tabvar; title Frequencies by &tabvar as of &sysday ; SAS LOG: SYMBOLGEN: variable FILE resolves to demog SYMBOLGEN: variable TABVAR resolves to gender 73 proc freq data=data&file; 74 table &tabvar; SYMBOLGEN: variable TABVAR resolves to gender SYMBOLGEN: variable SYSDAY resolves to Monday 75 title "Frequencies by &tabvar as of &sysday"; 76 COMPILING AND EXECUTING MACRO DEFINITIONS s are compiled programs that can be called in a submitted SAS program or from a SAS command prompt In order to compile a macro, a macro definition must be defined The general form of a macro definition is as follows: %macro macro_name; macro code %mend macro_name; Once a macro definition is submitted, the macro processor compiles the definition and produces a member in the WORK session SASMACR catalog The member in the catalog consists of the compiled macro program and text As previously described, when the word scanner detects a macro trigger such as % followed by a non-blank character it turns processing over to the macro processor The macro processor examines the token and recognizes it as a macro definition The macro processor continues to pull tokens from the input stack and compiles them until it reaches the end of the macro definition (%MEND) During macro compilation the macro processor: Creates an entry in the session catalog Compiles and stores all macro program statements for that macro as macro instructions Stores non-compiled items in the macro as text such as: variable references Text written by %PUT statements functions Arithmetic and logical macro expression Figure 4 Example of Compiled vs Text Items %macro sortby(byval=); %if &byval ne %then %do; proc sort data=mydata; by pid &byval; %end; %mend sortby; - 9 -

During compilation, if the macro processor identifies a syntax error, the macro processor continues checking the remaining macro code and issues messages for any additional syntax errors The macro processor will not store the macro for execution s that the macro processor compiles but does not store are called dummy macros The following macro definition will be used to illustrate how the macro processor compiles, stores and processes a macro definition Figure 4 How a Definition Gets Compiled User-Defined Variables FILE DEMOG 3 ( 4 keep 5 = 6 pid Catalog vitinfo macro %macro vitinfo(var=); data vitals; set datavitals visit &var); %mend vitinfo; %vitinfo(var=sysbp); SOURCE CODE: %macro vitinfo(var=); data vitals; set datavitals(keep=pid visit &var); %mend vitinfo; %vitinfo(var=sysbp); - 0 -

For illustration purposes, the compiled macro appears as the original macro definition in the input stack In actuality the entry would contain compiled macro instructions with constant text Execution of a Compiled execution consists of a series of repetitive actions Once a macro has been compiled and there is a call to a macro (eg %vitinfo), macro execution starts with the macro processor opening the SASMACR catalog to read the appropriate macro catalog entry As the macro processor executes the compiled macro it performs the following actions: Opens the session catalog and creates a local symbol table specific to the compiled macro processor removes tokens for the macro call from the input stack and places any parameter values in the local macro symbol table Executes compiled macro program statements Places non-compiled items back in to the top of the input stack as text Waits for the word scanner to process the text in the input stack Figure 43 to 47 illustrates how the macro processor executes a compiled macro Figure 43 The macro call goes from input stack to word scanner 3 4 5 6 % Catalog sortby macro %macro sortby(byval=); %if &byval ne %then %do; proc sort data=mydata; by pid &byval; %end; sortby; %mend sortby; proc print data=mydata; title Mydata by Gender ; The word scanner examines the input stack and detects % followed by a nonblank character It triggers the macro processor to examine the tokens The macro processor recognizes the %sortby as an entry in the SASMACR catalog and begins to process as follows: - -

The macro processor opens the WORK SASMACR catalog and creates a local symbol table specific to the compiled macro and enters a blank entry for BYVAL The macro processor removes the tokens for the macro call from the input stack and enters the value BYVAL in the SORTBY local symbol table 3 Once the parameter values have been added to the local symbol table, the macro processor turns its attention to the compiled macro In this example it encounters the %IF instruction and recognizes the next item will be text containing a condition 4 The macro processor places the text &byval ne back in the input stack ahead of the remaining text in the program 5 The macro processor waits for the word scanner to tokenize the newly generated text Figure 44 The macro program SORTBY continues executing 3 4 5 6 SORTBY Local Symbol Table Local Variables BYVAL GENDER Catalog sortby macro %macro sortby(byval=); &byval ne proc print data=mydata; title Mydata by Gender ; %if &byval ne %then %do; proc sort data=mydata; by pid &byval; %end; %mend sortby; 6 The word scanner resumes and recognizes the ampersand followed by a nonblank character and triggers the macro processor 7 The macro processor examines the token &BYVAL and first searches the SORTBY local symbol table for a matching entry When the macro processor finds the entry, it replaces the macro variable in the input stack with the value GENDER - -

Figure 45 The macro program SORTBY continues executing 3 4 5 6 SORTBY Local Symbol Table Local Variables BYVAL GENDER Catalog sortby macro %macro sortby(byval=); GENDER ne proc print data=mydata; title Mydata by Gender ; %if &byval ne %then %do; proc sort data=mydata; by pid &byval; %end; %mend sortby; 8 The macro processor waits for the word scanner to tokenize the generated text, the word scanner reads GENDER ne from the input stack 9 The macro processor evaluates the expression GENDER ne, and because the expression is true proceeds to the %THEN and %DO statements 0 The macro processor executes the compiled %DO instructions and recognizes the next item is text It places the text back to the top of the input stack and waits for the word scanner to continue tokenization The word scanner recognizes the beginning of a PROC step and triggers the compiler to begin accepting tokens The word scanner transfers the tokens to the compiler from the top of the input stack - 3 -

Figure 46 The macro program SORTBY continues executing Compiler proc sort data = 3 mydata 4 ; 5 by 6 pid SORTBY Local Symbol Table Local Variables BYVAL GENDER Catalog sortby macro %macro sortby(byval=); &byval; proc print data=mydata; title Mydata by Gender ; %if &byval ne %then %do; proc sort data=mydata; by pid &byval; %end; %mend sortby; When the word scanner detects the ampersand followed by a nonblank character, it triggers the macro processor 3 The macro processor once again first searches the SORTBY local symbol table for a matching entry When the macro processor finds the entry, it replaces the macro variable at the top of the input stack with the value GENDER 4 The word scanner resumes tokenization and continues to send tokens to the compiler 5 The macro processor resumes processing the compiled macro It recognizes the end of the %DO at the %END and continues until the %MEND 6 Once the macro processor reaches the %MEND statement, it removes the SORTBY local symbol table and macro SORTBY ends execution 7 The word scanner resumes tokenization, recognizes the PROC as the beginning of a step boundary and triggers the compiler 8 The compiled PROC step is executed and the compiler is cleared - 4 -

Figure 47 Remaining statements are compiled and executed Compiler proc sort data=mydata; by pid gender; 3 4 5 6 proc print data=mydata; title Mydata by Gender ; Catalog %macro sortby(byval=); %if &byval ne %then %do; proc sort data=mydata; by pid &byval; %end; %mend sortby; sortby macro SUMMARY The macro processor helps facilitate macro compilation and execution The relationship contains repetitive actions that begin with text submitted to the input stack, which is then tokenized by the word scanner Certain tokens trigger the macro processor at which time the word scanner waits for the macro processor to process an activity If the macro processor produces text as part of its activity, it places the text in the top of the input stack and waits while the word scanner resumes tokenization The word scanner then sends the token to the appropriate compiler This repetitive process continues until the entire program has been processed The technical aspects are not required for most macro programming, however understanding the concepts of how the macro processor works can help improve programming productivity REFERENCES SAS Language Reference, First Edition, Cary, NC, SAS Institute Inc, 997 SAS Guide to Processing, Version 6, Second Edition, Cary, NC, SAS Institute Inc, 990 Burlew, Michele M, SAS Programming Made Easy, Cary, NC, SAS Institute Inc, 998 ACKNOWLEDGMENTS Special thanks to my dear friend Adel Fahmy for encouraging me to take on the challenge of writing and presenting this topic TRADEMARKS SAS and all other SAS Institute Inc product or service names are registered trademarks or trademarks of SAS Institute, Inc in the USA and other countries Indicates USA registration Other brand and product names are registered trademarks or trademarks of their respective companies CONTACT INFORMATION Comments, questions and additions are welcomed Contact the author at: Lisa Lyons - PPD, Inc 3575 Quakerbridge Road, Suite 0 Hamilton, NJ 08690 lisalyons@lawrenceppdicom - 5 -