Programming Gems that are worth learning SQL for! Pamela L. Reading, Rho, Inc., Chapel Hill, NC

Transcription

1 Paper CC-05 Programming Gems that are worth learning SQL for! Pamela L. Reading, Rho, Inc., Chapel Hill, NC ABSTRACT For many SAS users, learning SQL syntax appears to be a significant effort with a low rate of return, because their routine programming tasks can be done just as easily with the base SAS syntax they already know. But there are some tasks that are very simple to do in SQL that are quite difficult in base SAS. If your assignment involves one of these tasks, then the effort to learn a little SQL will pay off handsomely! I will present a few little gems of SQL that have served me well in pharmaceutical programming. INTRODUCTION As a long-time user of base SAS, I am generally a reluctant user of SQL. The energy required to learn how to do what I want in SQL always seemed high in comparison to the outcome. But I have found some things that are so easy to do in SQL and so hard to do in base SAS that I m gradually being converted. I d like to quickly present a few gems that I wouldn t do any other way. You may not need these often. You may not need them at all. But file them away in case you do, because you could spend inordinate amounts of time trying to do them the base SAS way. Cartesian Product Merge We had a request for an Adverse Event analysis file. For obscure reasons known only to those who practice the black art of Statistics, it needed to have 1 record for every possible combination of AE Term and ID. For example, if there were 5 unique subjects and 10 unique Adverse Events among them, we needed a file with 50 records 10 records for each ID. We would then create variables indicating whether the subject suffered that particular AE. (The statisticians rethought this approach when we moved on to a bigger study!). This kind of file, which is called the Cartesian product in set theory, is the first step in EVERY merge done by SQL. And the code is simple: * get a file of unique AE terms; proc sort data=aexp nodupkey out=aexp_term (keep=preferred_term ) * get a file of unique IDs ; proc sort data=aexp nodupkey out=aexp_ids (keep=subjectid); by subjectid; * SQL Cartesian merge; proc sql; create table aexp_shell as select * from aexp_term, aexp_ids; quit; Those of you familiar with SQL will notice that this SQL code is lacking the usual WHERE clause that subsets the Cartesian merge to select only those records where variables match (like the BY statements in SAS MERGE). There are no common variables, so SQL creates a file containing two variables, Preferred_Term and SubjectID, with records for all possible pairs. This can be done without too much trouble in base SAS, but it involves the use of the POINT option in the data step, a command that many programmers tend to avoid. I don t think I ve ever used it before, but in the interest of equal time, I did my research and found this code on the SAS tech support web search. Here is the final step in base SAS, starting with the same two datasets created above:

2 data aexp_shell; set aexp_ids; do i=1 to n; set aexp_term point=i nobs=n; output; end; The SQL approach is more efficient, as the following tests show: Test 1 Test 2 Number of AE records Number of Unique Ids Number of Unique Terms Number of Records in Cartesian Product Real Time SQL 0.29 sec sec Real Time Base SAS 0.29 sec sec CPU Time SQL 0.39 sec 6.81 sec CPU Time Base SAS 0.39 sec 7.08 sec Figure 1 Fuzzy Merge: I do a lot of work requiring the querying of laboratory data using normal ranges specified elsewhere (usually in a spreadsheet). Lab normal ranges are often gender specific, and tests done at local site laboratories will be site specific. So I receive a spreadsheet that looks like this: Site Lab Test Test Code SITE1 Glucose GLUC SITE1 Creatinine CREAT SITE1 AST/SGOT AST SITE1 ALT/SGPT ALT SITE1 Calcium CALC SITE2 Glucose GLUC SITE2 Creatinine CREAT SITE2 AST/SGOT AST SITE2 ALT/SGPT ALT SITE2 Calcium CALC Figure 2 Once this data is imported (using PROC IMPORT or DDE), it s a simple base SAS task to read this and put out a SAS file with one record for each combination of Test Code, Site and Gender. Each record contains the appropriate upper and lower limits. Merge that with your lab records by Test Code, Site and Gender and do the comparison. The file would look something like this: Site LabTest TestCode Gender SITE1 Glucose GLUC M SITE1 Glucose GLUC F SITE1 Creatinine CREAT M SITE1 Creatinine CREAT F Figure 3

3 But wait! Some lab tests also have age-specific ranges. And what if the normal ranges changed on a given date due to a procedural modification? Now your lab normal specification table is a lot more complicated: Site Lab Test Test Code From Age: At Least To Age: But Not yet Range Effective Start Date Range Effective End Date SITE1 Glucose GLUC /1/2000 1/1/2010 SITE1 Glucose GLUC /1/2000 1/1/2010 SITE1 Creatinine CREAT /1/2000 1/1/2010 SITE1 AST/SGOT AST /1/2000 1/1/2010 SITE1 ALT/SGPT ALT /1/2000 1/1/2010 SITE1 Calcium CALC /1/2000 1/1/2010 SITE2 Glucose GLUC /10/2003 1/1/2010 SITE2 Glucose GLUC /10/2003 1/1/2010 SITE2 Creatinine CREAT /10/2003 2/8/2004 SITE2 Creatinine CREAT /9/2004 1/1/2010 SITE2 AST/SGOT AST /10/2003 1/1/2010 SITE2 ALT/SGPT ALT /10/2003 1/1/2010 SITE2 Calcium CALC /10/2003 2/8/2004 SITE2 Calcium CALC /9/2004 1/1/2010 Figure 4 You now need to compare the date of the test with an effective date range, and the age of the subject with an age range. Doing one of the two is not too awful in base SAS, although it would require a significant restructure of the spreadsheet data. But trying to merge using both age and date ranges is quite a difficult problem. SQL to the rescue! We restructure the file in the same way, creating separate records for male and female: Site LabTest TestCode Gender Age Age Date Date SITE2 Glucose GLUC M /1/2000 1/1/2010 SITE2 Glucose GLUC F /1/2000 1/1/2010 SITE2 Glucose GLUC M /1/2000 1/1/2010 SITE2 Glucose GLUC F /1/2000 1/1/2010 SITE2 Creatinine CREAT M /10/2003 2/8/2004 SITE2 Creatinine CREAT F /10/2003 2/8/2004 SITE2 Creatinine CREAT M /9/2004 1/1/2010 SITE2 Creatinine CREAT F /9/2004 1/1/2010 Figure 5

4 Then we use an SQL merge to match on Site, Gender, age and effective dates: PROC sql; create table _labcks as select a.*, b.* from labtests a, labnorm b where ( a.testcode = b.testcode and a.site = b.site and a.sex = b.gender and (b.lowage <= a.age <= b.highage ) and (b.lowdate < a.labdt <= b.highdate ) ) ; QUIT; In SQL terminology, file A is my laboratory result file LABTESTS, in which SEX is the variable that contains the patient gender code, AGE is the age (in the same unit as the normals file) and LABDT is the lab test date. B is the lab normals file LABNORM which is illustrated above (Fig. 5). SQL Summary Functions: In pharmaceutical reporting, we always need to display the population totals for each group being analyzed the N value if the statistics are to be interpreted properly. In some of our table generation procedures, the N value for each column needs to be in a SAS macro variable. Using base SAS, this would be accomplished with a procedure to compute the totals by group (PROC SUMMARY, PROC UNIVARIATE), using an OUTPUT statement to write the values to a dataset. Then a second data step using the SYMPUT function would write each value to a macro variable. Placebo (Group 1) Drug Group 2 Total Characteristic N=14 N=16 N=30 Weight (kg) Mean (SD) Median 7.4 (1.04) (0.43) (0.94) 7.2 Range (7,9) (8,9) (7,9) This can all be done in one easy SQL step: PROC sql noprint; select count(distinct subjid) into: tot1 from all where treatment=1; select count(distinct subjid) into: tot2 from all where treatment=2; select count(distinct subjid) into: tot3 from all; QUIT; %PUT &TOT1 &TOT2 &TOT3; The log will show that the three macro variables equal: 14, 16 and 30, our column N values. The INTO: tell SQL that the variable being named is a macro variable. By using the summary function COUNT(DISTINCT), you can use a source file that contains multiple records for each ID, providing all possible IDs are represented at least once.

5 CONCLUSION There are some tasks that are so much easier to do in SQL than in base SAS that it is worth learning a little SQL to do them. I hope this paper has whet your appetite and that you ll explore SQL further to find other gems that will make your programming job easier. REFERENCES Foley, Malachy J. Fuzzy Merges: Examples and Techniques Proceedings of the Twenty-Fourth Annual SAS Users Group International Conference (2002), Cary NC: SAS Institute Inc. pp Foley, Malachy J. Merging: Comparing the DATA Step with SQL Proceedings of the Tenth Annual Southeast SAS Users Group Conference (2002), Cary NC: SAS Institute Inc. pp Hermansen, Sigurd W. Structured Query Language: Logic, Structure and Syntax Proceedings of the Tenth Annual Southeast SAS Users Group Conference (2002), Cary NC: SAS Institute Inc. pp CONTACT INFORMATION You can contact the author at: Pamela Reading Rho, Inc Quadrangle Drive Chapel Hill, NC Work Phone: Pamela_Reading@rhoworld.com SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. indicates USA registration.