PAPER CC02 Link the Unlinked : Documenting Dependencies Among SAS Programs in a Project Abstract within across related Context and Clarification

Transcription

1 PAPER CC02 Link the Unlinked : Documenting Dependencies Among SAS Programs in a Project Gopal Rajagopal, Merck Research Laboratories, Upper Gwynedd PA Abstract Agency submissions typically involve dozens of tables and SAS programs with numerous interdependencies; some of which are subtle The average SOP emphasizes documentation within a SAS program This paper focuses on the oft unmet need for project level SAS program metadata ie documentation of dependencies across related SAS programs in a project Project level SAS program metadata can add value to the ongoing enhancement of an existing set of related programs It is critical to be selective about what program metadata is collected and how it is presented While one can generate volumes of metadata, 'too much' may be overwhelming rather than helpful UNIX text processing utilities grep, sed and awk along with other UNIX utilities and commands can be used to generate project level SAS program metadata from shared permanent datasets, macros, formats, and map programs to output How these can help you manage SAS programs will be explored in this paper Context and Clarification Necessity is the mother of invention and, sometimes, the inspiration for 'thinking out of the box' One of the author's projects, needing 'enhancement', comprised of 112 PC based SAS programs which used 81 datasets (of which some had millions of records) The programs also contained dozens of macros The 112 SAS programs represented a total of 16,245 lines of code (including comments, blanks etc) The client estimated that 5% of the code needed to change However, the critical question of 'which 5%?' is to be 'changed' needed to be answered SAS on the UNIX environment was chosen for the enhancement round to take advantage of quicker processing The programming team, fortuitously, resorted to repeated ad hoc searches for SAS constructs (macros, datasets etc) to chart out dependencies across programs These searches were then later woven together and organized in UNIX shell script utilities to enhance project level metadata output and to facilitate automationif you are laboring in a non UNIX shop, utilities equivalent to grep, sed and awk maybe available in the Windows platform and other operating system(s) Project level metadata, sagaciously generated, could boost the morale of the team (or at least provide a ground reality perspective) in that it quickly affords a forest for the trees view for a 'big' task The metadata chosen should be based upon the nature of the project and what is needed In addition to attempting parsimony vis-à-vis metadata generation, being thoughtful about how the metadata can be used and presented are significant issues It is said that a 'picture is worth a thousand words'; so presenting the dependencies in a graphical form may add value especially if the information is culled from thousands of lines of SAS code spread across many programs We found the following kinds of metadata to be most useful 1 Metadata on programs as a whole ie a listing of the order in which the programs are run, number of lines for each program, information on families of programs based on the naming convention 2 Data Set to Program mapping 3 Macro to Program mapping 4 Parsing log files and output files during batch runs can also useful 5 Metadata on all the datasets used in Programs If your project needs other kinds of project level metadata, your might find our approach useful

2 Overview of the Metadata documentation approach and utility development Before we delve into specifics, some general observations about the design approach and capabilities of developed utilities may be relevant at this juncture Given significant variability on how programmers write code, developing such utilities is a non trivial task The author believes that the utilities presented in this paper need to be refined further to capture more variations 'out of box' Basically the utilities assume that the SAS programs 'work' Access to corresponding SAS logs will also be helpful The utilities search for specific SAS syntax while attempting to take into account variations such as variable capitalization, spacing etc If the 'data' in 'data xxxyyy' statement is misspelt, the utility may not pick up the data set associated with this specific statement The search strategy, in view of the variations of coding styles, widens the net by using multiple ways of locating SAS objects In addition to the regular 'data xxxyyy' syntax search, for example, the script also searches for all the data sets in the designated directories associated with the libnames The dataset lists from all sources can then be harmonized into one super set of data set names for further processing To the extent possible, all the utilities are designed to be data driven One of the utilities, for example, which maps macros across SAS programs uses a editable list of macros and a editable list of SAS programs The proposed utilities use the SAS code itself as the 'raw' material and do not depend on any pre-existing documentation However, their output can potentially complement internal program documentation The metadata output is always current as long as the most recent version of the SAS program(s) is parsed/processed These utilities are designed to generate 2 types of output: Text and Graphics Both forms of output are designed to complement each other All utilities generate text output Some generate text only The text output contains all the lines in SAS programs which qualify as per the search criteria If the text output file is 'big', then one could search this file too for specific details A Microsoft Visio (a PC/Windows based graphics software) graphics 'ready' text file is generated by some of the utilities The text file is fed, in the form of a spreadsheet, to MS Visio The graphics display the hierarchy of SAS objects such as which macro is used across programs and which line it is used etc The graphics can also be deployed as HTML file(s) This feature allows for easier deployment of the metadata documentation to a development team as a web resource Generate a listing of SAS programs Given the overall design credo of developing data driven utilities to the extent possible, creating complete and comprehensive lists of SAS objects such as SAS programs, macros, data set names etc is a critical starting point The list of SAS programs associated with a project is central to all processing in the utilities This list will be called sas_program_listtxt in this document It can be created with relative ease From the UNIX command prompt, go (or 'cd') to the directory(s) in which the SAS programs reside and use the 'ls *sas > sas_program_listtxt' command syntax In this paper, only relevant sections of the utility code (especially for the larger ones) to generate different kinds of lists and mapping output are referred to in various numbered appendices from this section onwards Since some in the reading audience may not relate to UNIX shell scripting code, a general idea of the algorithm used will be broached at the appropriate juncture Lines, which may be specifically commented upon, will be tagged with a line number Every line of code for every utility will not be presented Readers are welcome to contact the author for details Even a simple list of programs with a modicum of additional metadata can be a surprisingly good start as a source of project level metadata A utility called program_desccmd (Appendix 1) is designed to provide a list of program names and line numbers for each program

3 The material below is a sample excerpt from file $docdir/program_desctxt sorted by alphabetical order import_bpsas 61 import_p_eventsas 95 import_practicesas 26 mexposuresas 373 outcome_c1y1sas 231 outcome_c1y1_agesas 242 outcome_c1y1_fstdosesas 246 The total number of lines for all the 112 SAS programs in our project, as noted before, was 16,245 lines This simple metric, relevant for resource planning, by itself implies that the project is not an overnight task by any means In the listing above there are 2 sets of programs names with a common prefix ie 'import' and 'outcome' If the family assumption is valid, code enhancement and QC might become easier for each family of programs Sorting on the number of lines metric indicates that the file mexposuresas is the largest program This program, at a minimum, assuming that it is not filled with too much redundant code (or empty lines for that matter!), will need 'more' work Some SAS programmers put a number of 'lines' in one line (as in 'data xxxyyy; set aaabbb; where numvar = 5;) ie each SAS code ending with a ';' does not always represent a line Hence, even a 'short' program can have a 'lot' packed in On the 'easier side', programs such as import_practicesas, mercifully contains only 26 lines of code A time ordered listing of program, especially in the absence of a 'batch run' file, buttressed with logs and other information can also provide a rough guide to the order in which the programs have to run Shared permanent SAS datasets to program mapping Utility The main goal of this utility is to map (to the extent possible) the programs in which permanent data sets are created and in which programs they are used (or referred to in program comments etc) The main steps are 1 Generate a list of permanent data sets created/used in all the SAS programs (Utility script Locate_ds2cmd excerpt available in Appendix 2) 2 Use the dataset list and the list of SAS programs to generate a dataset creation to program use/referral map (text and graphic ready version) (Utility script Mak_ds_to_use_map3cmd code excerpt available Appendix 3) Sample excerpt from Text file for Data Set creation to Usage (in programs) map output for data set datadirweight **** Data set datadirweight was created in import_weightsas program(s) and is referenced in program(s) Program Outcome_cny1sas 25: datadirweight Program Outcome_c1y1a_agesas 25: datadirweight Program import_weightsas 27:data datadirweight Program bl_cardio_risksas 24: datadirweight 76:%crisk(datadirweight, bmi, bmi); Program bl_age_sexsas 24: datadirweight Program Outcome_c1y1asas 25: datadirweight Program bl_bloodpresuresas

4 24: datadirweight Program bl_age_sex_agesas 24: datadirweight Program bl_indication_agesas 24: datadirweight **** End of datadirweight usage map Sample excerpt from the graphical map of Data Set creation to Usage (in programs) for data set datadirweight In text file mapping for data set datadirweight, you might notice that the actual line and the corresponding text are shown for each search 'hit' in each SAS program However, in the graphic version only the line numbers are shown At a maximum only the first 10 lines are shown to make the graph less unwieldy Data set datadirtherapy was apparently created in program import_weightsas as indicated by both the text and graphics version The data set creation to usage mapping can help the programming team vis-à-vis It provides a list of all the data sets used in project The 81 data sets of our project inspired us to assign one team member to QC the data set creation programs, given the criticality of 'raw' data quality The datasets used most frequently across SAS programs represent a potential single point of failure The associated SAS programs probably need enhanced QC efforts QC can be low key for infrequently used data sets Unused data sets, which the utility identifies, can be ignored If an upstream SAS program has completed a 'processing block', then programs downstream using this data set do not need to repeat this 'processing block' If a family concept applies to a group of data sets decreasing and reorganizing, if permitted by logic, the number of overlapping data sets in a 'family' can help save disk space However, if programs can use, as per logic, a smaller chunk of a bigger data set, then creating more family members may help performance

5 SAS macro to Program metadata map Utility The design goal of this utility is to produce a macro to SAS program map Macros with the same name across SAS programs are identified and mapped as the same macro even though the underlying code may be completely different A family of macros is inferred by macros across programs which have common prefixes The 2 main steps are 1 Generate a comprehensive list of macros created/used in all the SAS programs Code excerpt for locate_macro1cmd is available in Appendix 4 2 Use the macro list and the list of SAS programs to generate a macro to program creation, use/referral map or a program to macro use/referral map (text and graphic ready version) Code excerpt for mac_to_program_map2cmd is available in Appendix 5 Sample excerpt for text version of the macro to SAS program map for macro bsrx References for MACRO bsrx in SAS programs in /epidb/data002/analysis_021_gprd/pgm_analysis_1 are name is bl_rx_cardiosas Creation reference is 70:%macro bsrx(datain, dataout, gprdcode, gpcode, rname); Invocation reference(s) are 105:%bsrx(datain=bs_rx1, dataout=tdatadirglyco,gprdcode=glyco, 107:%bsrx(datain=bs_rx1, dataout=tdatadirdiu, gprdcode=diu, 109:%bsrx(datain=bs_rx1, dataout=tdatadirbblock, gprdcode=bblock, 111:%bsrx(datain=bs_rx1, dataout=tdatadiracei, gprdcode=acei, 113:%bsrx(datain=bs_rx1, dataout=tdatadirang, gprdcode=ang, 115:%bsrx(datain=bs_rx1, dataout=tdatadirnitr, gprdcode=nitr, 117:%bsrx(datain=bs_rx1, dataout=tdatadirccb, gprdcode=ccb, 119:%bsrx(datain=bs_rx1, dataout=tdatadirorant, gprdcode=orant, 121:%bsrx(datain=bs_rx1, dataout=tdatadirldasp, gprdcode=ldasp, 123:%bsrx(datain=bs_rx1, dataout=tdatadiranexre, gprdcode=anexre, 125:%bsrx(datain=bs_rx1, dataout=tdatadirezet, gprdcode=ezet, 127:%bsrx(datain=bs_rx1, dataout=tdatadirfibrat, gprdcode=fibrat, 129:%bsrx(datain=bs_rx1, dataout=tdatadirstatin, gprdcode=statin, 131:%bsrx(datain=bs_rx1, dataout=tdatadirnicot, gprdcode=nicot, SAS program name is bl_dxrx_diabsas Creation reference is 109:%macro bsrx(datain, dataout, gprdcode, gpcode, rname); Invocation reference(s) are 144:%bsrx(datain=bs_rx1, dataout=tdatadirinsulin, gprdcode=insulin, 146:%bsrx(datain=bs_rx1, dataout=tdatadirantid, gprdcode=antid,

6 Sample excerpt from graphics version of the macro to SAS program map for macro bsrx The text version of the output has a lot more detail in that it provides the line and the associated SAS code where the macro string bsrx occurs It also informs the user where the bsrx macro was created in the program The first line in the text file tells us which line the macro has been created The first line number in the graphic for each macro also corresponds to the macro creation reference Only the first 10 line numbers (and not the code) are available in the graph If a macro is not created in the program, it is described as being implicit For lack of space, an example of an implicit macro is not being shown The macro to program mapping can quickly provide the programming team A ready list of all the implicit and explicit macros used in project There were a total of 25 macros in our project out of which 2 were 'implicit' If there are macros with the same name, some of those macros could be actually identical in terms of code Such macro's 'creation' code can be moved out of the SAS programs as an implicit macro so that propagating any future changes to identified common (and moved out) macros becomes easier and consistent Reduced volume of SAS code and validation efforts If a frequently used macro is 'moved' out and is a 'large' macro, the number of lines of code for the involved SAS programs (and the related maintenance/qc burden) can be reduced QC efforts can sometimes be low key for those macros which are infrequently used Identification of same-named macros coded differently If a macro has a common name across SAS programs, and its code is found to be variable, one could consider renaming the macro with an independent name or give it the appropriate 'family' prefix to help preserve the sanity of current and future SAS programmers working with the SAS programs in question Consolidation of similar macros into one If a family concept applies to a few macro names, the programming team can consider creating a generalized macro which combines the functionality of the 'family' This can help reduce the overhead of maintaining many separate macros Mapping SAS programs to results tables/documents In our project, dozens of reports (or tables) were produced in the form of Microsoft Word files The naming convention of these tables was not explicitly related to the name of the programs generating them These reports were generated by a shared macro where one had to supply specific values to macro arguments The name of the MS Document was tied to a macro argument name 'docfile' Also the title string for each table of MS Word documents was also buried in the macro call's code A simple script doc1cmd (code excerpt available in Appendix 6) was used to generate a list of program names and corresponding MS Word document, related titles and headers It also provides a list of SAS programs that do not create reports

7 Sample output excerpt for the doc1cmd utility 20 Document Reference(s) for the program Outcome_cny1sas is 200: docfile=cny1doc, 182:%let title1= Table xxx Incident Rate of Clinical Outcomes; 183:%let title2= First year follow up and exposure of all courses (Exposure truncated when other DRUGx Rx started); 193:%RTFtable(Ds=outcome1, TitleRow=2, HeadRows=2, FootRows=4, 185:%let header1= Sub-cohort 1&rtfsym1 ^ ^ Sub-cohort 2&rtfsym2^ ^ Sub-cohort 3 &rtfsym3^ ^ ; 186:%let header2= Outcome Case N Incident rate&rtfsym4 Case N Incident rate&rtfsym4 Case N Incident rate&rtfsym4 ; End of Output list for /epidb/data002/analysis_021_gprd/pgm_analysis_1/outcome_cny1sas prog 21 Document Reference(s) for the program Outcome_cnynsas is 200: docfile=cnyndoc, 182:%let title1= Table xxx Incident Rate of Clinical Outcomes; 183:%let title2= Entire available follow up and exposure of all courses (Exposure truncated when other DRUGx Rx started); 193:%RTFtable(Ds=outcome1, TitleRow=2, HeadRows=2, FootRows=4, 185:%let header1= Sub-cohort 1&rtfsym1 ^ ^ Sub-cohort 2&rtfsym2^ ^ Sub-cohort 3 &rtfsym3^ ^ ; 186:%let header2= Outcome Case N Incident rate&rtfsym4 Case N Incident rate&rtfsym4 Case N Incident rate&rtfsym4 ; End of Output list for /epidb/data002/analysis_021_gprd/pgm_analysis_1/outcome_cnynsas prog Searching metadata from SAS logs and generating metadata (data dictionary) for permanent SAS datasets used in programs In addition to SAS programs, searching SAS logs in repeated batch runs to quickly isolate problem code can be valuable This looprunsas script (code excerpt in Appendix 7) was used for batch runs for all the programs (or a specific subset) The order of running SAS programs can be controlled in the utility by changing the order of programs in the sas_prog_listtxt file Furthermore, it examines the log of each SAS program for terms such as 'Error', 'Warning'' and 'initialized' and writes it out to a file called run_errors along with the program name All the errors for the batch run are available in the run_error file Our programming team found this invaluable in that one does not have to go over dozens of logs looking for errors You can save different versions of the run_error file if needed Automated generation of a data dictionary for each of the datasets created/used in the programs is also an important aspect of metadata If you know the dataset libname directory name(s) a priori, one could go in SAS, point a libname to the data set directory in question and use the 'PROC CONTENTS' to get all the data set(s) metadata However, if the libnames are unknown, the utility contents1cmd (excepts are available in Appendix 8) will plough through all the SAS programs and first create a unique list of libnames It will then use this list and produce ' PROC CONTENTS ' output for all the data sets in all the libnames associated with SAS programs being documented Limitations The parser for the SAS language is designed to deal with almost 'any coding style' as long it is syntactically correct For instance it will even deal with a made-up and improbable SAS 'program' such as libname mydata 'c:\temp' ; data temp; var1 = 1; output; var1=2; output;

8 run; data mydatagold ; set temp ; where var1 = 1; run ; This code may look unappetizing but it is syntactically correct The utilities, which use the UNIX grep utility extensively to search for SAS objects, are not suited for multi-line searches The utilities, in a limited way, are attempting to duplicate some of the functionality of the SAS parser; an onerous task under the best of circumstances The utilities try to make up for this limitation in the grep utility to some extent by searching for SAS objects in different ways and then harmonizing the search output from different methods A significant lacuna in our approach was that we were using only Unix utilities, which by definition do not 'know'' SAS, to parse SAS programs An interesting possibility is the use of SAS parser itself to parse the SAS programs (to be documented) and then use the SAS metadata/dictionary tables to provide information on SAS objects like permanent datasets, macros etc Once we get harmonized lists of objects, the searching can be done using SAS or UNIX utilities Using the SAS parser also minimizes the exposure to version changes in SAS Conclusion and Summary Automated generation of project wide SAS program metadata and dependency maps, especially for 'big' projects, has the potential to meet a critical need of SAS programmers The metadata and mapping should be a user friendly resource Consequently it is important to consider the volume and presentation format of the generated metadata Trademark Information SAS and SAS/MACRO software are registered trademarks or trademarks of SAS Institute Inc in the USA and other countries indicates USA registration Other brand and product names are registered trademarks or trademarks of their respective companies Acknowledgments The author wishes to thank S Kasay, DL Usavage, M M Coughlin, R Ganesan, G Havildar, N M Sene, N Rajan, M Tumelty and M Du for their comments and feedback on this paper Contact Information Your comments and questions are valued and encouraged Contact the author at: Gopal Rajagopal Senior Statistical Programmer/Analyst, Scientific Programming Merck Research Laboratories Mail stop: UG1D-80 PO Box 1000, North Wales, PA gopal_rajagopal@merckcom