Making Sense of Proc Tabulate Jonas V. Bilenas, JP Morgan Chase, New York, NY

Size: px

Start display at page:

Download "Making Sense of Proc Tabulate Jonas V. Bilenas, JP Morgan Chase, New York, NY"

Pamela Houston
5 years ago
Views:

1 Makig Sese of Proc Tabulate Joas V. Bileas JP Morga Chase New York NY ABSTRACT The TABULATE procedure i SAS provides a flexible platform to geerate tabular reports. May begiig SAS programmers have a difficult time uderstadig the sytax of PROC TABULATE ad as a result ted to avoid usig the procedure. This tutorial will explai the sytax of PROC TABULATE for the begiig programmer ad offer a framework for geeratig a report usig PROC TABULATE. The data used i this paper represets simulated cosumer credit card usage data ad the code was developed usig versio 6.12 of SAS. INTRODUCTION PROC TABULATE is based o a table geeratio code developed by the U.S. Departmet of Labor. The sytax does ot resemble other SAS PROCS ad will appear to be cryptic to the begiig SAS programmer. To make matters worse lookig at sample TABULATE code provides o clue for the ovice TABULATE user. Sample code ofte looks like log mathematical equatios that make o sese to the idividual learig TABULATE. Frustratio ofte leads to dismissal of the procedure. The TABULATE procedure is so flexible that there is o sigle correct way of geeratig tabular reports. The method I propose i this paper is the way I have leared TABULATE ad still use it to geerate custom tabular reports. I teach this method to all ew users of TABULATE ad fid it helps them grasp the procedure. Here are my 7 steps to geerate tabular reports usig PROC TABULATE: 1. Do t paic. Relax ad take a few meditative breaths. 2. Desig the report o paper. The desig of the TABULATE report should be first specified o paper. The report desig that is developed o paper will guide the code geereatio. 3. Geerate the iitial test code. Retur to the computer ad start codig the desig that you have formulated o paper. There are a few rules of TABULATE sytax that will geerate the report ad we will review these here. 4. Test retest ad verify usig a small sample. Test the iitial code usig a small umber of observatios. Use a radom sample or use a OBS= optio to test the sytax. TABULATE is tricky ad you may have to ru a few versios of your code ad possibly iclude some DATA preparatio. Verify that the results make sese. Look at your data ad report! Do t assume the report is correct just because you have o sytax errors. At this step do t be too cocered with report appearace. Verify that the results are correct. 5. Clea up the appearace of the report. Oce the code has geerated a report that makes sese clea up the output of the report usig a few tricks we will review here. Also cosider addig additioal summaries ad or statistics to make the report eve more useful. This may be difficult to do if the report has already bee specified but you ca recommed chages to the specificatios. 6. Ru code with OBS=MAX. 7. Sit back smile ad be proud of your report. Your maager may be cofused at how you were able to produce the report without sortig the data or without ay spreadsheet cruchig. He/She may be cofused at your code but the you ca take him/her through the 7 steps of TABULATE harmoy. STEP 1 DON T PANIC SAS code geeratio ca be frustratig to begiig SAS users. To begi a TABULATE code geeratio get ito a relaxed ad calm mode. The desig ad code may take a few iteratios but that should be expected with TABULATE. Table geeratio ad formattig will take a bit of code work ad re-work. It is to be expected that the first TABULATE code geerated is ot the fial versio. STEP 2 DESIGN THE REPORT ON PAPER The structure of the report must first be specified i order to geerate the TABULATE code that will produce the report. Desigig the report o paper will help you idetify which variables to use how variables are defied i TABULATE ad what summary iformatio is required. As a illustratio for this paper take a simple example from a accout maagemet experimet i cosumer credit. A treatmet is applied to existig accouts (test group with a hold out sample (cotrol group set aside to compare agaist. The desig sample also icludes a factor for credit score levels (low risk medium risk high risk that are recorded at time of experimetatio. After the test is completed ad results are available maagemet is lookig for some quick reports to read results. The first report specified is a simple report that could be geerated usig PROC FREQ but we will code it up i TABULATE sice maagemet still has ot fialized report desig. Usig PROC TABULATE will allow us to modify the code to accommodate further desig specificatios. The iitial desig o paper look as follows: Treatmet Group Total Accouts Deliquet Accouts Test Cotrol All Variables we have i our SAS DATASET are: : 1 if accout is deliquet 0 if ot. : 1 if i test group 0 if i cotrol SCORE: Iteger credit score with values specifyig risk level: = high risk = medium risk = low risk To proceed to the ext step we have to familiarize ourselves with some sytax rules of PROC TABULATE. STEP 3 GENERATE THE INITAIL CODE There are few sytax rules that oe has to get familiar with i order to use PROC TABULATE. The first step is to idetify if variables are classified as CLASS variables or VAR variables. CLASS variables are those that are categorical ad/or have levels that you wish to geerate summaries for each level. CLASS variables ca be either umeric or character.

2 VAR variables idetify aalysis variables that you wish to report statistics for. A variable caot be listed as both a CLASS ad a VAR variable i the TABULATE procedure. A VAR statemet is ot required i the TABULATE procedure. Table dimesios are specified i the TABLE statemet of PROC TABULATE. Table dimesios are separated by commas. This is differet from PROC FREQ where a asterisk is used to separate table dimesios. If the TABLE statemet does ot iclude ay commas the resultig report will be a sigle row with multiple colum output. A TABLE statemet with oe comma produces a row by colum report with the row dimesio specified first. The maximum umber of commas is 3 producig a page by row by colum report. The restrictio o the umber of commas does ot limit the umber of CLASS variables you ca iclude i your report. Further crossig of CLASS variables ca be itroduced with a asterisk. The asterisk is also used to specify VAR variables statistical summaries ad/or format of output. Statistical summaries are similar to the oes available i PROC MEANS. These iclude sums meas umber of records etc. that oe sees i other PROCS. Versio 6.12 does ot iclude percetile statistics but these are available i Versio 8. The TABULATE procedure also icludes a PCTN ad a PCTSUM statistical summary. Sytax may be difficult to grasp for these summaries providig more reaso to highlight the importace of STEP 4 (test ad retest. PCTN calculates the percet of a frequecy ad PCTSUM calculates the percet of a SUM. These require deomiator defiitios that are specified betwee brackets <>. So ow we have to worry about CLASS ad VAR variables o top of cosiderig where to use commas asterisks ad brackets. Ad to make it eve more cofusig we ca eve iclude a = withi the body of the TABLE statemet to specify labels to be prited. Wow o woder TABULATE code ca look cryptic. After much practice these will become secod ature. I the meatime follow these guidelies: CLASS: Used for categorical variables or variables for which you wat to see summaries for each value of the variable. Default statistic applied to a class variable if oe provided is N (umber of observatios. A PCTN statistic ca also be specified to produce frequecy percets. VAR : Used to specify aalysis variables that you wat geerate statistics for each level of CLASS variables. The default statistic if oe specifies is SUM. : Used for dimesio splits page row colum. Dimesio splits do ot have to be limited to CLASS variables. You ca for example have a CLASS dimesio as your row dimesio ad a VAR variable as your colum dimesio. *: Use to specify ay of the followig: o Aother CLASS variable Split o A VAR variable o A statistic o A Format <>: Specify the deomiator CLASS dimesios for PCTN or to specify deomiator VAR variable whe usig the PCTSUM statistic. = For labels specificatio that will improve the readability of the report. Let s try some code usig our hypothetical example. Both variables ca be specified as CLASS variables. However we may wat to report some additioal statistics o our variable such as a bad rate or maybe eve a good to bad odds ratio. Iitial code is icluded here: proc tabulate data=esug01.data formchar= class test var bad table test bad ru Note that the table statemet ca be listed o oe lie but listig variables ad dimesios o separate lies ca aide i debuggig of code. Output form this code ad modificatios are reviewed i Step 4 below (ote: the formchar= (11 spaces optio removes boxes aroud table cells makig it readable for those without SAS MONOSPACE fots. STEP 4 RE AND VERIFY USING A SMALL SAMPLE This step ofte requires a umber of iteratios ad tests util the table geerates results that we expect. It may take a umber of rus to get a table that has all the statistics that oe requires. Durig this step we ca make chages i the table output ad we may wish to redraw each modificatio o paper to get a clear picture of how to proceed i code geeratio. Output from the ru is: N SUM Not bad but we would like row totals o the report as well. PROC TABULATE has a ALL keyword that we ca provide totals for all levels of the CLASS variable. The code is modified alog with the output: proc tabulate data=esug01.data formchar= class test var bad table test all bad ru N SUM ALL Lookig better but ca we add a row percet for accout totals ad a bad rate statistic (bad total/n?

3 The PCTN statistic geerates the required row percet. This statistic also requires a deomiator defiitio that is placed iside the <>. Settig up this defiitio ca be complicated ad may require a umber of iteratios to get the correct output. The values etered iside the <> are class variables that make up the dimesio that oe wishes to see a percetage. The row dimesio is required to see row percets ad colum dimesios are required to see a colum percets. Souds cofusig so keep testig util the umbers look correct. If you expect row percets verify that the umbers reported are ideed row percets. Use the PCTSUM statistic to get the bad rates (MEAN ca also geerate bad rates as well for this example but I have always used the PCTSUM calculatio. We eed to specify the VAR variable that idetifies the deomiator. This may require some DATA step preparatio if the deomiator variable is ot icluded i the DATA. Bad Rate is defied as: (sum of bads/(sum of all accouts. The code required for these chages are: data stuff set esug01.data oa=1 proc tabulate data=stuff oseps formchar= class test var bad oa table test all pct<test all> bad bad*pctsum<oa> /rts=10 ru This code also icludes a NOSEPS optio. This optio prevets separate lies (or lie breaks betwee levels of row CLASS variables. I also icluded a RTS= optio for the TABLE statemet. TABULATE reserves a quarter of the page for the row dimesio labels. The RTS=10 specifies that oly 10 spaces are to be used to geerate the width of the row dimesio (icludig vertical separatio lies. Output is listed: N PCTN SUM PCTSUM ALL STEP 5 CLEAN UP THE APPEARANCE OF THE REPORT After the report has bee validated we ca clea up the report presetatio. There are a umber of tricks to make the report look professioal ad they are described with examples. Thigs we would like to chage i the above report are: 1. More descriptive labels that will make the report easier to read. 2. Specific formats for the umeric output. 3. Chage the levels of the CLASS variable to read ad CONTROL rather tha umeric values of 0 ad List the Test level before the Cotrol level. These chages are easily added to the code. Cotiue to work with small sample datasets to miimize ru time ad computer costs. The code ad output is listed followed by a explaatio of the methods used to format the output: proc format value test 0 = 'Cotrol' 1 = ' Test' picture p7r (roud = '009.00%' picture p6r (roud = '09.99%' proc tabulate data=stuff oseps formchar= order=formated class test var bad oa format test test. table test='' bad='number of Deliquet Accouts'*sum=' '*f=comma10. bad='bad Rate '* pctsum<oa>=' '*f=p6r. /rts=17 ru Number of Row Number of Bad Accouts Percet Deliquet Rate Test % % Cotrol % % Total % % The output looks much eater ad is ready to be placed i productio. The codig tricks used are summarized: 1. Use PROC FORMAT to assig values to CLASS variables if required. Some of the format labels ca iclude spaces so that results ca be ordered by FORMAT labels. Specify ORDER=FORMATED optio i TABULATE to order output by FORMATED labels as opposed to ordered by data values. 2. Set up a PICTURE FORMAT to prit TABULATE PCTN ad PCTSUM s with trailig % sigs. Usig a percet format withi TABULATE will ot work sice the values are ot outputted as proportios. 3. Specify specific formats for statistics withi the TABLE statemet usig *F=format_ame. 4. Specify formats for CLASS variables with a FORMAT assigmet statemet i PROC TABULATE. 5. Assig labels after variables ad statistics i the TABLE statemet with the use of = specificatio. If you do ot wat a default label prited (i.e. SUM PCTSUM us a ull label (=. 6. Adjust label spaces ad format widths to geerate a report that is presetable. Tabulate will split labels depedig o width of format. If format is too short the words cotaied i your labels may split before the word ed. Keep testig with small datasets by adjustig format widths ad TABLE labels util the desired output is obtaied. Addig additioal CLASS variables ca be added as page breaks or cross breaks withi other CLASS variables. Here is some code ad output for differet possibilities we have available whe we add aother CLASS variable. For this example we have the credit score that we ca set up levels for by use of PROC FORMAT ad specify the variable as a CLASS variable i the TABLE statemet of PROC TABULATE. Use of PROC FORMAT is more desirable tha

4 creatig levels i a DATA step sice it is faster requires o additioal DATA geeratio or modificatio ad is less likely to result i assigmet errors. The first example rus risk level as a page dimesio. Output is show after the code. proc format value test 0 = 'Cotrol' 1 = ' Test' value score = ' High ' = ' Medium' 701-high = 'Low' picture p7r (roud = '009.00%' picture p6r (roud = '09.99%' proc tabulate data=stuff oseps order=formatted formchar= class test score var bad oa format test test. score score. table score='risk = ' all='all Risk Levels' test='' bad='number of Deliquet Accouts' *sum=' '*f=comma10. bad='bad Rate '*pctsum<oa>=' '*f=p6r. /box=_page_ rts=17 misstext=' ' codese ru OUTPUT: 1. BOX=_PAGE_ prits out the value of the page dimesio i the upper left had corer box of the table. 2. MISSTEXT= sets missig text to the text specified i the label. I this case the ull label was specified so that if the report results i a missig value a blak is prited rather tha SAS MISSING values. 3. CONDENSE optio causes multiple tables to be fit o oe page if space permittig. Without the optio each page dimesio prits out a separate report. The risk CLASS variable ca be specified as a row dimesio ad the cross each level of risk by the levels of the test CLASS variable. The TABULATE code that geerates this table desig is provided alog with output. proc tabulate data=stuff oseps order=formatted formchar=' ' class test score var bad oa format test test. score score. table (score='risk' all='all Risk Levels' * (test='group ' all=' ' bad='number of Deliquet Accouts' *sum=' '*f=comma10. bad='bad Rate '*pctsum<oa>=' '*f=p6r. /box=_page_ rts=21 misstext=' ' codese ru Risk = High Number Number of Test % % Cotrol % % Total % % Risk = Medium Number Number of Test % % Cotrol % % Total % % Risk = Low Number Number of Test % % Cotrol % % Total % % All Risk Levels Number Number of Test % % Cotrol % % Total % % For the output geerated a umber of TABLE optios were also itroduced. The TABLE OPTIONS icluded are: Number Number of Risk Group High Test % % Cotrol % % % % Medium Group Test % % Cotrol % % % % Low Group Test % % Cotrol % % % % All Risk Group Levels Test % % Cotrol % % % % For the above table we eeded to place paretheses aroud SCORE ALL ad ALL specificatios sice we wated to see each level of ALL for each level of SCORE ad ALL. For more complicated desigs remember to always desig the table o paper first. This will ease the codig up of complicated reports ad give you more cofidece i desigig reports with PROC TABULATE. Let us look at oe more table desig ad aother DATA step trick. A umber of statisticias look at odds ratios rather the percetages. Versio 6.12 does ot have a ratio statistic. Here is code that fools TABULATE to provide a

5 odds ratio i the output. The defiitio of the odds ratio here is: total good accouts/total bad accouts For this example the defiitio of good is a accout that has ot become deliquet over the test evaluatio period. The code ad output follow (ote that FORMAT code is ot icluded here sice it is assumed the FORMATS have bee created: data sushi set stuff if bad=0 the good=1/100 else good=0 proc tabulate data=sushi oseps order=formated formchar=' ' class test score var bad oa good format test test. score score. table (score='risk LEVEL' (test='group ' * good=' '*pctsum<bad>=' '*f=8.2 /box='good/bad Odds' rts=21 misstext=' ' ru OUTPUT: CONCLUSION The TABULATE procedure is a complex PROC that geerates tabular reports. There is o sigle correct way to geerate reports i TABULATE. This paper provided a framework that will help you uderstad the workigs of TABULATE ad will provide a startig framework o how to geerate table reports usig PROC TABULATE. Ofte reports ca be geerated without additioal DATA steps. The use of PROC FORMAT ca assist i labelig output. DATA steps may be required to geerate additioal variables for TABULATE output ad/or modify variables to geerate statistics required i your output. Versio 8+ of SAS has may more optios ad statistical reportig capabilities. These iclude ODS OUTPUT of SAS DATASETS ad percetile statistics. CONTACT INFORMATION Your commets ad questios are valued ad ecouraged. Cotact the author at: Joas V. Bileas JP Morga Chase Decisio Scieces 55 Water Street 1618 New York NY Joas.Bileas@Chase.com Good/Bad Odds GROUP Test Cotrol Total RISK LEVEL High Medium Low Total The trick above was to divide the umerator i the PCTSUM calculatio by 100 i the DATA step. Sice TABULATE always multiplies the result of a PCTSUM by 100 the resultig output is a ratio. Now if I ca oly figure out how to report the log of odds I will become a TABULATE master. Note that the test CLASS variable is ow i the colum dimesio. We put paretheses aroud ad ALL sice we wated the PCTSUM summary for all levels of as well as ALL. The paretheses aroud SCORE ad ALL are ot required here but they do t impact the output. STEP 6 - RUN CODE WITH OBS=MAX Ru productio code o full datasets. Always test TABULATE code o smaller datasets. This is more efficiet ad time savig especially for large datasets. STEP 7 - SIT BACK SMILE AND BE PROUD OF YOUR REPORT Oce your TABULATE report is geerated feel cofidet that you have begu your trip to uderstadig the mystery of PROC TABULATE. Your trip will be a log oe grasshopper but it will be a full of elightemet as you become a TABULATE Master.

Python Programming: An Introduction to Computer Science

Python Programming: An Introduction to Computer Science Pytho Programmig: A Itroductio to Computer Sciece Chapter 6 Defiig Fuctios Pytho Programmig, 2/e 1 Objectives To uderstad why programmers divide programs up ito sets of cooperatig fuctios. To be able to