SQL Metadata Applications: I Hate Typing

Transcription

1 SQL Metadata Applications: I Hate Typing Hannah Fresques, MDRC, New York, NY ABSTRACT This paper covers basics of metadata in SQL and provides useful applications, including: finding variables on one or more data sets; identifying common or different variables on data sets; storing lists of variable names in macro variables; and generating code using text strings of metadata. INTRODUCTION Imagine you have a data set with a small number of variables and are interested in those related to grade point average (GPA). Because the data set has a small number of variables, it is simple to review PROC CONTENTS output. Similarly, PROC CONTENTS output can be used to compare the contents of two data sets. However, if your data sets have many variables, a query on an SQL DICTIONARY table will produce more succinct, and more useful, output. SQL DICTIONARY tables can also be used to define a macro variable that stores a list of variables names. The macro variable definition will be metadata-dependent, and the macro variable is available in any PROC step or DATA step. Metadata can be used to generate code, saving typing and creating code that does not have to be edited if, for instance, variable names change. BASICS OF SQL METADATA DICTIONARY tables are read-only tables available through SQL that store metadata ( data about data ). These tables are stored in the DICTIONARY library. The libref DICTIONARY is automatically assigned by PROC SQL, and the information in the tables is updated each time a DICTIONARY table is accessed. This paper uses DICTIONARY.COLUMNS which contains information about columns (variables) and their attributes. VARIABLES ON DICTIONARY.COLUMNS To better understand the DICTIONARY.COLUMNS table, run the DESCRIBE TABLE statement. describe table DICTIONARY.COLUMNS; The output is sent to the SAS log and is comparable to output produced by PROC CONTENTS. NOTE: SQL table DICTIONARY.COLUMNS was created like: create table DICTIONARY.COLUMNS (libname char(8) label='library Name', memname char(32) label='member Name', memtype char(8) label='member Type', name char(32) label='column Name', type char(4) label='column Type', length num label='column Length', npos num label='column Position', varnum num label='column Number in Table', label char(256) label='column Label', format char(49) label='column Format', <continues> ); Each row in the DESCRIBE TABLE output is information on the columns (variables) in DICTIONARY.COLUMNS. 1

2 DATA IN DICTIONARY.COLUMNS DICTIONARY.COLUMNS contains information about variables in all SAS data sets in all currently defined libraries. Access metadata for a particular data set by using the WHERE clause to specify the respective libname and memname. Note that values of libnames and memnames are stored in uppercase in the DICTIONARY.COLUMNS table. select libname, memname, name, type, length where libname eq "SASHELP" and memname eq "CLASS"; The output produced is: Library Member Column Column Column Column Name Name Name Type Length Label WORK CLASS Name char 8 First Name WORK CLASS Sex char 1 Sex (M/F) WORK CLASS Age num 8 Age WORK CLASS Height num 8 Height (inches) WORK CLASS Weight num 8 Weight (lbs) DICTIONARY.COLUMNS contains one row for every variable in SASHELP.CLASS. (The variables on the SASHELP.CLASS data set do not have labels assigned as shown. However, for the purpose of the examples presented here, we will imagine that the labels are assigned as shown in this output.) APPLICATIONS OF DICTIONARY.COLUMNS DICTIONARY.COLUMNS can be used to: 1. Find columns in a table based on characteristics of the NAME, LABEL or TYPE 2. Find common columns between tables, or find only those common columns with different attributes 3. Create a macro variable that stores a list of column names 4. Generate code 1. FIND COLUMNS IN A TABLE The table WIDEDATA in the WORK library contains thousands of columns. In defiance of the know thy data commandment, we have forgotten some of the column names. We are interested in any columns related to grade point average (GPA). Names and labels are stored in mixed case in the DICTIONARY.COLUMNS table, so the WHERE clause references to names and labels must be case insensitive. select name, type, length, label where libname eq "WORK" and memname eq "WIDEDATA" and (upcase(name) contains "GPA" or upcase(label) contains "GPA"); The output produced is: 2

3 Column Column Column Column Name Type Length Label CCUMGPA num 8 CUM GPA, CALCULATED R1CUMGPA num 8 REL TERM1 CUMULATIVE GRADE POINT AVG R1GPAT num 8 REL TERM1 TERM GPA R2CUMGPA num 8 REL TERM2 CUMULATIVE GRADE POINT AVG R2GPAT num 8 REL TERM2 TERM GPA R3CUMGPA num 8 REL TERM3 CUMULATIVE GRADE POINT AVG R1R4ENWU num 8 KCC: R1-R4 CUM GPA, NO WITHDRAWS 2. FIND COMMON COLUMNS BETWEEN TABLES The following code will identify any variables which have the same (case-insensitive) name on more than one of the three data sets BIF, FAM, or KID. select libname, memname, name, type, length where libname eq "WORK" and memname in ("BIF","FAM","KID") group by upcase(name) having count(*) > 1; To find columns on more than one table that have different attributes, modify the HAVING clause: having count(distinct type) > 1 or count(distinct length) > 1 3. CREATE A MACRO VARIABLE LIST OF COLUMN NAMES First, let s review the basics of creating a macro variable with SQL. The basic syntax is: select <column> into :<macro_variable> separated by ' ' from <table> ; This code assigns the query results to a macro variable with results from each row separated by blanks. Macro variables created using this method have a global scope. proc sql noprint; select name into :varlist1 separated by ' ' where libname eq "WORK" and memname eq "WIDEDATA" and upcase(name) contains "GPA"; %put varlist1 = &varlist1; Generates the following output in the log: varlist1 = CCUMGPA R1CUMGPA R1TGPA R1ATGPA R1BTGPA R1NLCGPA R1ANLCGPA R1LCGPA R2CUMGPA R2TGPA R2ATGPA R2BTGPA R3CUMGPA R3TGPA R3ATGPA R3BTGPA R4CUMGPA R4TGPA R4ATGPA R4BTGPA R5CUMGPA R5TGPA R5ATGPA R5BTGPA R6CUMGPA... 3

4 This macro variable can be used anywhere in our SAS code we would specify a list of variable names. If we had used the %LET statement as above, we would have had to hard-code all variable names. With the SQL DICTIONARY.COLUMNS approach, we avoid typing and have metadata-driven code. 4. GENERATE CODE In this example, we use metadata to create original versions of all variables on a data set. Here s the code we want to generate. The example includes only five variables, so typing out the code is relatively painless. Still, it is prone to typos, and must be modified if variable names change. Typing this code becomes more burdensome the more variables there are on the data set. data class2; set sashelp.class; OName = Name ; OSex = Sex ; OAge = Age ; OHeight = Height ; OWeight = Weight ; Label OName = "ORIGINAL First Name" OSex = "ORIGINAL Sex (M/F)" OAge = "ORIGINAL Age" OHeight = "ORIGINAL Height (inches)" OWeight = "ORIGINAL Weight (lbs)" ; Here s how we can generate the code using SQL: First, we create a table called VARS which contains the variable name and label for all variables on the SASHELP.CLASS data set. proc sql noprint; create table vars as select name, label where libname eq "SASHELP" and memname eq "CLASS"; Next, in the DATA step we use character variable functions to manipulate the strings of text to create the desired code. data vars2; set vars; length oname $ 200 olabel $ 500; oname = 'O' trim(name) ' = ' trim(name) ';'; olabel = 'LABEL O' trim(name) ' = ' quote(catx(' ','ORIGINAL',label)) ';'; Then, we use PROC SQL to assign all values of the modified metadata to macro variables, creating strings of code which can be seen in the %PUT output in the log. 4

5 proc sql noprint; select oname, olabel into :createnew separated by ' ', :editlabels separated by ' ' from vars2; Use %PUT statements to check the generated code. Sample log output: %put %quote(&createnew) ; OName = Name; OSex = Sex; OAge = Age; OHeight = Height; OWeight = Weight; %put %quote(&editlabels) ; LABEL OName = "ORIGINAL First Name"; LABEL OSex = "ORIGINAL Sex (M/F)"; LABEL OAge = "ORIGINAL Age"; LABEL OHeight = "ORIGINAL Height (inches)"; LABEL OWeight = "ORIGINAL Weight (lbs)"; Finally, we execute the generated code in a DATA step by referencing the macro variables that store the generated code. data class2; set sashelp.class; &createnew &editlabels We verify that the technique worked using PROC CONTENTS: proc contents data = class2 varnum; Contents of CLASS2 # Variable Type Len Label 1 Name Char 8 First Name 2 Sex Char 1 Sex (M/F) 3 Age Num 8 Age 4 Height Num 8 Height (inches) 5 Weight Num 8 Weight (lbs) 6 OName Char 8 ORIGINAL First Name 7 OSex Char 1 ORIGINAL Sex (M/F) 8 OAge Num 8 ORIGINAL Age 9 OHeight Num 8 ORIGINAL Height (inches) 10 OWeight Num 8 ORIGINAL Weight (lbs) CONCLUSIONS SQL DICTIONARY tables are powerful tools for exploring data sets, producing concise output, and generating code. Because the metadata is stored in tables, the power and flexibility of SAS can be used to manipulate this data for many applications. 5

6 REFERENCES Ronk, K. M., First, S. and Beam, D. An Introduction to PROC SQL SUGI Paper Code Crafters, Inc. Summary of SAS Dictionary Tables and Views ACKNOWLEDGMENTS SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. indicates USA registration. Other brand and product names are registered trademarks or trademarks of their respective companies. Christopher J. Bost, MDRC, is acknowledged for his assistance in developing and reviewing this paper. CONTACT INFORMATION Your comments and questions are valued and encouraged. Contact the author at: Hannah Fresques MDRC 16 East 34 th Street New York, NY (212)