PROC SUMMARY AND PROC FORMAT: A WINNING COMBINATION

Transcription

1 PROC SUMMARY AND PROC FORMAT: A WINNING COMBINATION Alan Dickson - Consultant Introduction: Is this scenario at all familiar to you? Your users want a report with multiple levels of subtotals and grand totals - BUT they don t need the lowest level of detail and/or they want some wrinkle (such as calculated values on the subtotal lines) that good old PROC PRINT can t provide. Furthermore, they want the final report in a certain sequence that bears no resemblance to any collating sequence (EBCDIC, ASCII, ascending/descending ) that you have available. As PROC REPORT matures, it will perhaps address all these problems, but another viable approach is to use PROC SUMMARY to tackle the totaling end and PROC FORMAT to address the order. How can FORMAT be used to effectively sort a dataset? Well, of course it can t, but one of the beauties of SUMMARY is that the CLASS statement does not require sorted input to do its stuff. Essentially, the technique involves what could be called a 2-phase format. The steps are: 1)- Use FORMAT to superimpose your own simple sequence over the user s requirements for grouping and ordering. 2)- Use SUMMARY to generate the appropriate totals. 3)- Use FORMAT in conjunction with PROC PRINT or DATA-step reporting techniques to produce the report, substituting the user s values back over your interim values. If you use a DATA step, you can also address any subtotal calculation needs. Consider this example: PROC FORMAT; VALUE IN_AGE = = = = = = 6 ; VALUE $OUT_AGE 1 = = = = = = ; Here, the general trend of increased age equating to poorer health and thus to greater risk is distorted by one group, including the very young (infant illnesses) and young adults (wild & crazy lifestyles). It is a straightforward task to summarize the data based on the intermediate, grouped values, and then report them with the for-show-only values of the second format Clearly, it would be difficult, if not impossible, to reproduce this kind of user sequence without using some kind of recoding. However, while the technique outlined above solves many problems, it should not be applied without careful consideration. Since an entire DATA step pass of the data is required to set up the interim values, the cost could be substantial for large files. Of course, if you are already passing the data for other reasons, this setup can turn out to be almost a free ride. Regular use of this technique has made me a life-long fan, but a cautious one. The degree of control it gives you is impressive, but there are a few pitfalls for the unwary. The remainder of this paper will walk you through an example which will show how to apply the technique, what PROC SUMMARY can do for you, and what it can sometimes do TO you. The Demonstration Data (See Figure 1) This is a handful of selected records from a fictitious database of participants in a weight-loss clinic in New England which draws its clientele from the Northeast and Canada. Each individual has an indicator (Y/N) for each of the three programs (exercise, weights, aerobics) and the number of pounds lost on each.

2 Note that, while the combination of an N-flag and a non-zero loss is invalid, a Y-flag with no attendant loss is valid. A participant may be merely toning up, for instance. The males and females are banded by initial weight into the appropriate weight class for their sex, using the PUT function. The generated variable is associated with the output format for printing. The Basic SUMMARY First, we run a simple SUMMARY, using the NWAY option to suppress all subtotal and grand total records. Later, we ll look at the output when all records are included. For now, we will use the SUM= option on the analysis variables. This allows us to keep the same variable names and would also retain any LABEL or FORMAT attributes associated with the analysis variables. However, to protect you from overflow, SAS will always use a length of 8 for the summed variables. As always, SUMMARY generates the special variables _TYPE_ and _FREQ_. We will cover _TYPE_ in more depth later. _FREQ_ is simply a count of the observations included in each CLASS cell. PROC SUMMARY CLASS SEX COUNTRY STATE; OUTPUT OUT=TEST1 SUM=; PROC VAR PRINT; SEX COUNTRY STATE _FREQ_ EXERLOSS WEITLOSS AEROLOSS; Due to space constraints, the output is not shown but the resulting report, containing the summed totals for each loss variable, is ordered as expected - alphabetically by STATE within COUNTRY within SEX. PROC SUMMARY also has options for different orders. For example, adding ORDER=FREQ to the PROC statement orders the output based on frequency of occurrence. However, close examination of the results shows that the Female Canadians from the Province of Quebec (F/CN/PQ) come out after the sole lady from Ontario (Figure 2). What is happening here? It seems that SAS remembers the Male from Ontario and takes him into account when processing the Females. Since there are now two each from PQ and ON, and ON was encountered first, it remains first in the Female category, leading to the unexpected output. Similarly, if you try ORDER=DATA, SUMMARY will order the output based strictly on the order in which it encounters values in the input dataset, with similar nonintuitive results in the output. Because ON is encountered first in the Male group, it remains first in the Female group, despite the fact that there is a PQ Female before the ON Female in the data. Clearly, these unexpected results are dependent on the actual data. You could run a job several times using these options and then suddenly start running into problems when you least expect it. It should be noted that these results were obtained under CMS Version 6.09E and Win95 Version Other operating systems environments have produced different results in the past. Caveat emptor! The underlying reason for these results is that SUMMARY is evaluating the dataset in its entirety relative to the CLASS statement. You can cause it to treat categories in isolation by using the BY statement instead at the highest level or levels. This, of course, requires either that you know your data is sorted or you incur the cost of sorting it. In our example, based on our knowledge of the data, we could use a BY SEX NOTSORTED; statement to cause SUMMARY to deal with the two groups independently of each other. This will also result in less memory being used since SAS can essentially re-use the space when the SEX changes rather than holding all combinations separate for the entire step. An even more surprising result (at least it was to me) occurs when we add our derived, formatted variable WT_CLASS to the CLASS breakdown. Obviously, with so few observations, the results are pretty meaningless, but look closely at the data values when we run this program (Figure 3): PROC SUMMARY CLASS SEX COUNTRY STATE WT_CLASS; OUTPUT OUT=TEST1 SUM=; PROC PRINT; VAR SEX COUNTRY STATE WT_CLASS _FREQ_ EXERLOSS WEITLOSS AEROLOSS; FORMAT WT_CLASS $CHAR1.; Without the final FORMAT statement, everything actually looked fine. However, when we examine the underlying data values, we find that there are no WT_CLASS 6 values any more - in fact, they ve all been turned into value 3. Why?? The first time I encountered this phenomenon, I called SAS Institute to report it as a bug. I was politely in-

3 formed that SUMMARY is supposed to work this way and is furthermore fully documented on page 376 of the Version 6 Procedures manual. Essentially, because both values 3 and 6 are formatted to , SUMMARY will force them all to 3. Again, using a BY SEX NOTSORTED; statement will eliminate this effect since only Females can be 3. However, if BY is not an option for you, it s best to be aware of this subtle possibility. The ID and MAXID Statements Although not directly related to the technique of combining formats with SUMMARY, the ID statement can sometimes prove useful when using SUMMARY. ID gives you the highest value encountered for a variable in each cell. This often overlooked option is helpful when you want to carry a value from the detail records forward to the summarized output dataset. It is most applicable when the value remains the same across all detail records in each CLASS cell. For example, if you had hospital identifier codes and hospital names on the detail data and you wanted to summarize admissions or anything else by hospital and produce the report in code order, but also show the name on the report, ID fits the bill perfectly. PROC SUMMARY- DATA=HOSP.DATA NWAY; CLASS HOSPCODE; ID VAR HOSPNAME; ADMISSNS; etc If the variable you want to carry forward is a numeric variable, another option is to use the MAX statistic on that variable. However, you can only request statistics on analysis variables, which by definition must be numeric. Hospital name is clearly alphabetic. I have frequently seen hospital name used as a final CLASS variable to achieve this result, but that causes a lot of redundant summarization. To return to the weight-loss example, let s say that you wanted to produce a report showing total weight losses for each program on a separate line along with the indicators to show participation in a given cell. Simply ignoring zero values for the losses may be invalid (remember the guy just toning up). Unfortunately, this code will NOT work. PROC CLASS SUMMARY SEX COUNTRY STATE; ID EXER WEIT AERO; OUTPUT OUT=TEST1 SUM=; etc As the manual makes quite clear, when more than 1 ID variable is specified, the result represents the highest SINGLE OBSERVATION combination of values within each cell. In the test data, the Males in MA give a result of YYN - not YYY - because no single Male participated in all three programs, although all three had some MA Male participation. In Version 5, there was no easy way around this, short of generating numeric flag variables to replace the Y/N flags and taking the MAX. However, Version 6 added the MAXID (and MINID) statements which address this need. Essentially, MAXID returns the value of a partner variable when the other partner reaches its maximum value within a cell. MINID, obviously, returns the minimum. PROC SUMMARY CLASS SEX COUNTRY STATE; OUTPUT OUT=TEST1 SUM= MAXID( EXERLOSS(EXER) WEITLOSS(WEIT) AEROLOSS(AERO) )= MAX_EXER MAX_WEIT MAX_AERO; etc. This code will now give us the desired result of YYY for MA Males. Finally, let s look at the output from SUMMARY when we remove the NWAY option and all the intermediate level totals are produced. This is when the _TYPE_ special variable really comes into play. The number of values which _TYPE_ will take is a function of the number of variables in the CLASS statement - in fact, 2 raised to that power. So, in our example of CLASS SEX COUNTRY STATE;, _TYPE_ will have 8 values (2 cubed) ranging from 0 to 7. The _TYPE_ 0 record is always the grand total, representing all observations and so is always a single observations. The number of observations generated for each other _TYPE_ value depends on how many discrete combinations occur in the dataset.

4 Refer to the final output (Figure 4) as we walk through the rest of the _TYPE_s and you should see how SUMMARY builds them. Essentially, SAS moves from right to left along the CLASS statement in a binary fashion, getting progressively more detailed in the contents of each cell. So, _TYPE_ 1 represents a STATE breakdown, irrespective of SEX and COUNTRY. _TYPE_ 2 represents a COUNTRY breakdown, irrespective of SEX and STATE. _TYPE_ 3 represents a COUNTRY and STATE break down, irrespective of SEX. _TYPE_ 4 represents a SEX breakdown, irrespective of COUNTRY and STATE. _TYPE_ 5 represents a SEX and STATE breakdown, irrespective of COUNTRY. _TYPE_ 6 represents a SEX and COUNTRY breakdown, irrespective of STATE. _TYPE_ 7 represents a SEX, COUNTRY and STATE breakdown, the finest level of detail possible with this CLASS statement. Within each _TYPE_ value, the total of the _FREQ_ values adds up to that of the grand total, i.e. the whole dataset is counted in each _TYPE_. There is a good diagram in the V6 Procedures manual (p.377) which outlines how the _TYPE_s are built up in this binary fashion. However, it is helpful if you note that the diagram works up to the results of the statement CLASS C B A rather than CLASS A B C which you might be tempted to assume. With 3 CLASS variables, it is quite possible that the user will find all combinations valid and meaningful. With only 2, it s almost certain. However, when you move up to 4 or 5 CLASS variables (16 or 32 combinations), some of them are bound to be redundant or meaningless. Two solutions are possible at this point. The first is to let SUMMARY generate them all, and then discard the unwanted ones based on their _TYPE_ values. One way to do this would be a WHERE= dataset option on the output dataset. The second is to run an NWAY SUMMARY on the finest level of detail required and then subsequent SUMMARY step(s) on the resulting dataset. This may be one more non-nway step, or you could repeat the process of NWAY steps at increasingly higher levels of totaling. Finally, SET all the resulting datasets together to give you the master data for printing. As usual, the choice depends on your particular data and application but, in general, the first option will use more space (at least until you discard), while the second will use more CPU (multiple steps). I hope this brief explanation has given you some new insight into using SUMMARY and another valuable addition to your SAS toolkit. Acknowledgments: SAS is a registered trademark of SAS Institute Inc., Cary NC, USA.

5 Figure 1: E X H I B I T S PROC FORMAT LIBRARY=LIBRARY; VALUE FWT_FMT = = = HIGH = 4 ; VALUE MWT_FMT = = = HIGH = 8 ; VALUE $WT_FMT 1 = = = = = = = = 250+ ; *** FEMALE WEIGHT CLASSES MALE WEIGHT CLASSES ***; DATA TEST.DATA; LENGTH SEX $ 1 COUNTRY $ 2 STATE $ 2 WT 3 WT_CLASS $ 1 EXER $ 1 EXERLOSS 2 WEIT $ 1 WEITLOSS 2 AERO $ 1 AEROLOSS 2; LABEL WT_CLASS = WEIGHT CLASS ; FORMAT WT_CLASS $WT_FMT.; INPUT SEX $ COUNTRY $ STATE $ WT EXER $ EXERLOSS WEIT $ WEITLOSS AERO $ AEROLOSS; IF SEX = F THEN WT_CLASS = PUT(WT,FWT_FMT.); ELSE WT_CLASS = PUT(WT,MWT_FMT.); CARDS; M US MA 148 Y 03 Y 00 N 00 M US MA 195 N 00 Y 12 N 00 M US MA 262 Y 11 N 00 N 00 M US MA 172 Y 08 N 00 Y 05 M US MA 211 N 00 Y 16 Y 05 M CN ON 188 Y 08 N 00 N 00 M CN PE 222 Y 07 Y 07 Y 07 F CN PQ 202 Y 11 N 00 Y 11 F CN ON 177 N 00 N 00 Y 09 F CN PQ 169 Y 10 N 00 Y 04 F US MA 210 N 00 N 00 Y 08 F US MA 081 Y 02 N 00 Y 03 F US MA 111 Y 02 N 00 Y 02 ; PROC PRINT; VAR SEX COUNTRY STATE WT WT_CLASS EXER EXERLOSS WEIT WEITLOSS AERO AEROLOSS; TITLE PRINT OF RAW INPUT DATA - FORMATTED WEIGHT CLASS ; PROC PRINT; VAR SEX COUNTRY STATE WT WT_CLASS EXER EXERLOSS WEIT WEITLOSS AERO AEROLOSS; FORMAT WT_CLASS $CHAR1.; TITLE PRINT OF RAW INPUT DATA - UNFORMATTED WEIGHT CLASS ;

6 PRINT OF RAW INPUT DATA - FORMATTED WEIGHT CLASS OBS SEX COUNTRY STATE WT WT_CLASS EXER EXERLOSS WEIT WEITLOSS AERO AEROLOSS 1 M US MA Y 3 Y 0 N 0 2 M US MA N 0 Y 12 N 0 3 M US MA Y 11 N 0 N 0 4 M US MA Y 8 N 0 Y 5 5 M US MA N 0 Y 16 Y 5 6 M CN ON Y 8 N 0 N 0 7 M CN PE Y 7 Y 7 Y 7 8 F CN PQ Y 11 N 0 Y 11 9 F CN ON N 0 N 0 Y 9 10 F CN PQ Y 10 N 0 Y 4 11 F US MA N 0 N 0 Y 8 12 F US MA Y 2 N 0 Y 3 13 F US MA Y 2 N 0 Y 2 PRINT OF RAW INPUT DATA - UNFORMATTED WEIGHT CLASS OBS SEX COUNTRY STATE WT WT_CLASS EXER EXERLOSS WEIT WEITLOSS AERO AEROLOSS 1 M US MA Y 3 Y 0 N 0 2 M US MA N 0 Y 12 N 0 3 M US MA Y 11 N 0 N 0 4 M US MA Y 8 N 0 Y 5 5 M US MA N 0 Y 16 Y 5 6 M CN ON Y 8 N 0 N 0 7 M CN PE Y 7 Y 7 Y 7 8 F CN PQ Y 11 N 0 Y 11 9 F CN ON N 0 N 0 Y 9 10 F CN PQ Y 10 N 0 Y 4 11 F US MA N 0 N 0 Y 8 12 F US MA 81 1 Y 2 N 0 Y 3 13 F US MA Y 2 N 0 Y 2 Figure 2: PRINT OF SUMMARY DATA - FREQUENCY ORDERING OBS SEX COUNTRY STATE _FREQ_ EXERLOSS WEITLOSS AEROLOSS 1 M US MA M CN ON M CN PE F US MA F CN ON F CN PQ

7 Figure 3: PRINT OF SUMMARY DATA - UNFORMATTED WEIGHT CLASS OBS SEX COUNTRY STATE WT_CLASS _FREQ_ EXERLOSS WEITLOSS AEROLOSS 1 F CN ON F CN PQ F CN PQ F US MA F US MA F US MA M CN ON M CN PE M US MA M US MA M US MA M US MA Figure 4: PRINT OF SUMMARY DATA - NO NWAY - ALL SUMMARIES OBS _TYPE_ SEX COUNTRY STATE _FREQ_ EXERLOSS WEITLOSS AEROLOSS MA ON PE PQ CN US CN ON CN PE CN PQ US MA F M F MA F ON F PQ M MA M ON M PE F CN F US M CN M US F CN ON F CN PQ F US MA M CN ON M CN PE M US MA