SAS Talk: LAG Lead. The SESUG Informant Volume 9, Issue 2

Size: px

Start display at page:

Download "SAS Talk: LAG Lead. The SESUG Informant Volume 9, Issue 2"

Horace Fields
6 years ago
Views:

SAS Talk: Dr. Ian Whitlock has been programming largely in SAS since 1981. His main interest is in computer languages, particularly SAS, and how the language affects the programming process.

1 SAS Talk: Dr. Ian Whitlock has been programming largely in SAS since His main interest is in computer languages, particularly SAS, and how the language affects the programming process. He has participated since 1986 in many international, regional, and local SAS User Group meetings. He is also active on SAS-L with some insightful answers to perplexed SAS programmers. Let's talk about the LAG function - its traps (leading to bad code) and pitfalls (preventing good code). When first discovered by the neophyte SAS programmer, they are ecstatic; it is seen as the most useful function because it remembers a variable from the previous observation and conquers. That is the first pitfall - the LAG function has nothing to do with the previous observation. So what is the LAG function? You should picture it as a box, whenever you call the function, a value is removed from the box and returned while the argument is then put into the box. The following example shows that LAG does not mean previous observation. /* Pitfall #1 */ The log then shows: seq=1 l_seq=. seq=2 l_seq=1 seq=3 l_seq=2 The first time in the loop L_SEQ is assigned the value in the box. What is it? Missing, there is no value. What value goes into the box? The value is 1 because that is the value of SEQ. The next time in the loop L_SEQ is assigned 1 because that is the value that was put into the box on the previous iteration of the loop. When we get to the third iteration then L_SEQ is assigned 2 from the last time we put something in the box. Note that there are no observations read and no observations written, so LAG has nothing to do with the previous observation. Now suppose we have a data set W with the variable having the values 1, then 2, then 3. Now consider: set w ( keep = seq ) ; We see the same log as above. Why? The observations of W are read in the standard implicit DO-loop of the SAS DATA step. Now we see where the misconception comes from. The neophyte SAS programmer has missed the fact that it is an artifact of how the data is read in the implicit loop. (Continued on page 2)

2 Page 2 (Continued from page 1) So what? Well it's bad because... By misleading you, it prevents you from doing other useful things with this function By misleading you, it leaves you open to getting caught in various traps By its enchantment, it prevents you from using other tools when they are needed. For the first trap let's return to the first loop but unroll it. /* trap #1 */ seq = 1 ; seq = 2 ; seq = 3 ; Now the log shows: seq=1 l_seq=. seq=2 l_seq=. seq=3 l_seq=. OOPS, why is L_SEQ always missing? If you are in the mode of thinking previous observation, then you have no handle for debugging this problem. If you think, box and last value in the box, then the question to ask is, "How many boxes are there in the code?" Three calls to LAG then three boxes, three first missing values. Each physical statement involving LAG means a new box. But now you are set up for another trap. Consider: The log shows: /* trap #2 */ array a (3) v1 - v3 (1 2 3) ; do i = 1 to dim(a) ; l_a = lag(a[i]) ; put a[i]= l_a= ; v1=1 l_a=. v2=2 l_a=. v3=3 l_a=. What happened? There is only one physical call to LAG, but we still have a problem. On the other hand, we feed three different variables into the one LAG. SAS sets up a box for each physical LAG and each different variable name. And from that you should conclude that you will never see useful SAS code where LAG is called with a literal (Continued on page 3)

3 Page 3 (Continued from page 2) argument. We have explored various traps that become hard to debug unless one is willing to let go of the idea that the LAG is driven by the previous observation. Suppose we have the following code. /* trap or feature */ data w ; do group = 10 to 30 by 10 ; group_seq = group + seq ; output ; if first.group then l_group_seq = lag(group_seq) ; put group= seq= group_seq= l_group_seq= ; Now the log shows: group=10 seq=1 group_seq=11 l_group_seq=. group=10 seq=2 group_seq=12 l_group_seq=. group=10 seq=3 group_seq=13 l_group_seq=. group=20 seq=1 group_seq=21 l_group_seq=11 group=20 seq=2 group_seq=22 l_group_seq=. group=20 seq=3 group_seq=23 l_group_seq=. group=30 seq=1 group_seq=31 l_group_seq=21 group=30 seq=2 group_seq=32 l_group_seq=. group=30 seq=3 group_seq=33 l_group_seq=. It's a trap if you think previous observation, but a feature if you think box (queue) or remember last value loaded. Note that we only pushed something into the box at FIRST.GROUP, hence the lag is returning the value of GROUP_SEQ at the beginning of the previous group. In the first group there was no previous group, so L_GROUP_SEQ is missing. Now what about groups 10 and 20, why did L_GROUP_SEQ become missing? Ah, yes, we forgot to RETAIN L_GROUP. Note that the LAG function remembers, but this does not mean that the variable remembers. One often sees the advice, "Never make LAG conditional." This advice is based on the surprised or trapped feeling one gets by hanging onto the previous observation concept. However, as the above shows, you might have legitimate times when you really do want the effect shown. It is much better to realize you need to understand and control rather than make arbitrary rules to avoid surprises. On the other hand, consider: retain hold_group_seq ; if first.group then l_group_seq = hold_group_seq ; (Continued on page 4)

4 Page 4 (Continued from page 3) Here s the log: if first.group then hold_group_seq = group_seq ; put group= seq= group_seq= hold_group_seq= l_group_seq= ; group=10 seq=1 group_seq=11 hold_group_seq=11 l_group_seq=. group=10 seq=2 group_seq=12 hold_group_seq=11 l_group_seq=. group=10 seq=3 group_seq=13 hold_group_seq=11 l_group_seq=. group=20 seq=1 group_seq=21 hold_group_seq=21 l_group_seq=11 group=20 seq=2 group_seq=22 hold_group_seq=21 l_group_seq=. group=20 seq=3 group_seq=23 hold_group_seq=21 l_group_seq=. group=30 seq=1 group_seq=31 hold_group_seq=31 l_group_seq=21 group=30 seq=2 group_seq=32 hold_group_seq=31 l_group_seq=. group=30 seq=3 group_seq=33 hold_group_seq=31 l_group_seq=. So we have achieved the same thing as we did using the conditional LAG. It is largely a matter of taste and style as to whether you prefer LAG method or the extra retained variable method of achieving this result. On the other hand, it is best to have both patterns available. LAG has its uses, but often the introduction of a second variable declared in a RETAIN statement will be easier to work with. This is the second pitfall to LAG. Now, you should be wiser about LAG than when you began reading this article. If not, you probably lived through these examples in some other form at some point in your career. There are times when you need information from the next observation to be read instead of the previous one. There is no LEAD function. However, there are code patterns that allow you to get information from the next observation. To investigate the possibilities, let's look at the data set W created by the following code. data w ( drop = seq ) ; do group = 1 to 3 ; info + 1 ; output ; Suppose we want the next observation's value of INFO and do not care about crossing GROUP boundaries. The simplest solution is: merge w w ( firstobs = 2 keep = info rename = (info = next_info) ) ; /* no by statement, i.e. a 1-1 merge */ put (_all_) (=) ; (Continued on page 5)

5 Page 5 (Continued from page 4) The log shows: group=1 info=1 next_info=2 group=1 info=2 next_info=3 group=1 info=3 next_info=4 group=2 info=4 next_info=5 group=2 info=5 next_info=6 group=2 info=6 next_info=7 group=3 info=7 next_info=8 group=3 info=8 next_info=9 group=3 info=9 next_info=. Extending this solution to handling BY groups is messy. However, SQL makes it easy to do self joins with different variable names. First, we need a variable like SEQ. If it is not there, it is easy to create. data w ; if first.group then seq = 0 ; seq + 1 ; Now here is the SQL code. proc sql ; select w.*, wnext.info as next_info from w left join w as wnext on w.group = wnext.group and w.seq + 1 = wnext.seq ; quit ; The resulting listing is: group info seq next_info The idea is simple, but the first time I saw it used was by Howard Schreier in one of his fabulous messages on SAS-L at many years ago. To see other points of view, search for LAG or LEAD at In addition to considering LAG, I have had another objective. I have tried to demonstrate how one can get a better understanding of SAS (or any computer language) by looking at very simple easily manufactured test data and asking the question, "What happens with this code?" If you also absorbed that lesson, then you have learned something even more significant than the usage of LAG.

SAS Styles ODS, Right? No Programming! Discover a Professional SAS Programming Style That Will Last a Career

SAS Styles ODS, Right? No Programming! Discover a Professional SAS Programming Style That Will Last a Career Joe Perry, Perry & Associates Consulting, Oceanside, CA The typical, new SAS programmer has