Playing With String Data David Fickbobm, Fireman's Fund Insurance Company, Novato, CA

Transcription

1 Playing With String Data David Fickbobm, Fireman's Fund Insurance Company, Novato, CA Abstract This talk will introduce beginning SAS users to the idea of using string functions. Users will be introduced to several STRING functions that come with BASE SAS. Attendees will learn functions provide quick easy ways to work with STRING data. They will learn functions provide the building blocks for effective, more flexible SAS programs. Users will be introduced to the LENGTH, SUBSTR, COMPRESS, VERIFY, INPUT, PUT, TRANSLATE, SCAN, TRIM, UPCASE, REPEAT, II (concatenation), INDEX, and INDEXC functions. How Lengths of Character Variables are Set in a SAS Data Step Before discussing these functions, it would be helpful to understand bow base SAS assigns storage lengths to character variables and what the LENGTH function does for us. Look at the following program: DATA EXAMPLE!; INPUT STRING $3.; LEFT= 'X'; *X AND 4 BLANKS; RIGHT=' X'; *4 BLANKS AND X; Cl = SUBSTR(GROUP,l,2); C2 = REPEAT(GROUP,l); LGROUP = LENGTH(GROUP); LSTRING = LENGTH(STRING); LLEFT = LENGTH(LEFT); LRIGHT = LENGTH(RIGHT); LCJ = LENGTH(Cl); LC2 = LENGTH(C2); ABCDEFGH 123 XXX 4 y 5 PROC CONTENTS DATA=EXAMPLEI POSITION; TITLE 'OUTPUT FROM PROC CONTENTS'; PROC PRINT DATA=EXAMPLEI NOOBS; TITLE 'LISTING OF EXAMPLE I'; One purpose of this example is to explain the term LENGTH. If you look at the output from PROC CONTENTS, each of the variables is listed, along with a TYPE and LENGTH. Take a moment and look at the output from PROC CONTENTS below: Output From Proc Contents ----Variables Ordered by Position- # Variable Type Len Pos I GROUP Char STRING Char LEFT Char RIGHT Char Cl Char C2 Char LGROUP Num LSTRING Num LLEFT Num LRIGHT Num LCI Num LC2 Num The column labeled LEN (Length) is the number of storage positions needed to store the values for each variables listed. By default, all the numeric variables are stored in 8 bytes. But, what about the storage lengths of the character variables. First look at the two variables listed in the INPUT statement; GROUP and STRING. Since this is the first mention of these variables in this data step, their lengths will be assigned by the rules governing INPUT statements. No columns or IN FORMATS were associated with the variable GROUP, so its length will be set to 8 bytes, the default length for character variables in this situation. The variable STRING uses a $3. INFORMAT so its length is set to 3 bytes. The length oflleft and LRIGHT are controlled by the assignment statement. Most statements in SAS begin with a systatic name. The name defines what kind of instruction this is. The assignment statement begins with a variable name followed by the "=" and then a valid SAS statement. The storage lengths of C I and C2 are a bit more difficult to understand. The variable C 1 is defined to be a substring of the variable GROUP. The SUBSTR function takes the form: SUBSTR(char_var,start,length); which means to take a substring from char_var starting at the position indicated by the start argument for a length indicated by the length argument. Why then, is the length of C 1 equal to 8 and not 2? The SAS compiler determines 75

2 lengths at compile time. Since the starting position and length arguments of the SUBSTR function can be a variable, the compiler must set the length of C I equal to the largest possible value it can ever attaln, the length of GROUP. The same logic determines the length of C2, defined by the REPEAT function. Since the number of additional replications is defined by the second argument of the REPEAT function, and this argument can be a variable expression, the compiler sets the length of C2 to the largest possible value, 200; the maximum length of a character variable in the SAS system. Therefore, always use a LENGTH statement for any character variable that does not derive its length elsewhere. For example, to set the length of C2 to 16, you would write: LENGTH C2 $ 16; The LENGTH function returns the length of a character string, not including trailing blanks. The value of LEFT and RIGHT are one and five respectively, for every observation. This demonstrates that the trailing blanks in LEFT are not counted by the LENGTH function while the leading blanks in RIGHT are. The table below summarizes the values returned by the LENGTH function for the remaining variables: OBS group Lgroup string Lstring I abcdefgh XXX y 1 5 OBS Cl LC1 C2 1 ab 2 ABCDEFGHABCDEFGH 2 XX 2 XXX XXX 3 Y I Y Y The values oflgroup and LSTRING are very understandable. The value of LC 1 is 1 for the third observation since C I is only I byte in length in the third observation. The values for LC2 are not as easy to understand. The REPEAT function says to take the original value and repeat it n times. As a result, the first observation, LC2 is 16 (2 times 8). Looking at observations 2 and 3, the trailing blanks must be accounted for. In observation 2, the value of GROUP is 'XXXbbbbb' (where the b's stand for blanks). When we repeat this string one time, we get: 'XXXbbbbbXXXbbbbb'. We do not count the trailing blanks, so the length is II == I 1 The same logic applies to the third observation; we have a 'Y' followed by 7 blanks repeated once. We do not count the last 7 trailing blanks, so we have a length of9 = 8 + I. The foundation being set, we now start to get our string tools out of our toolbox. Using the proper tools makes building anything much easier, makes us more efficient, and more effective. Let us look at some of the string functions tools available in SAS software toolbox. How to Remove Charaeters from a String A common problem is removing selected characters from a string. For example, suppose you want to remove blanks, parentheses, and dashes from a telephone number or any other variable that has been stored as a character value. We take our COMPRESS function tool outof our toolbox. The COMPRESS function can remove any number of specified characters from a character variable. The program below uses the COMPRESS function twice: To remove blanks from the string; and to remove blanks, paoontheses, and dashes. Here is the code: DATA EXAMPLE2; INPUT PHONE$ 1-15; APHONE = COMPRESS(PHONE); BPHONE = COMPRESS(PHONE,'(-) '); (415) (510) PROC PRINT DATA=EXAMPLE2; TITLE 'LISTING OF EXAMPLE 2'; The blanks have been removed from APHONE. Please note we did not use a second argument in this situation. When a second argument is omitted, the COMPRESS function removes only blanks. For the variable BPHONE, the second argument contains a list of the characters to remove: left parenthesis, blank, right parenthesis, and blank. This string can be placed in single or double quotes. Cbaracter Data Verification Validation is a common job in information technology. As an example you need to be sure that only certain values are in a character variable. Below, only 'A', '8', 'C', 'D', and 'E' are valid data values. An easy way to test ifthere are any invalid characters present is shown next: 76

3 DATA EXAMPLE3; INPUT ID $ 1-3 ANSWER$ 5-9; P = VERIFY(ANSWER,'ABCDE'); OK=PEQO; 001 ACBED 002ABXDE CCE 004ABC E PROC PRINT DATA=EXAMPLE3 NOOBS; TITLE 'LISTING OF EXAMPLE 3'; The function tool we use in this program is the VERIFY function. It iuspects every character in the first argument and, if it finds a value not in the verity string (the second argument), it will return the position of the first erroneous value. If all values in the string are located in the verity string. a value of 0 is returned. In the first observation P = 0 and OK = 1. The P in the second observation-will be 3 (the position of the 'X') and OK will be 0. The P in the third observation is a 1 and OK will be 0. The P in the fourth observation will be a 5 and OK will be 0. The Substring Tool The substring function works with part of a longer string. In our example, we want to create two variables from our ID code input. Our data contains a two-position state abbreviation code in positions 1 and 2. Positions 7, 8, and 9 are a county code. We want to create two variables from this data; one containing the two digit state codes and the other, a numeric variable representing the county within the state. The county code will be from positions 7,8, and 9. We can use the SUBSTR tool very nicely here. The code follows: DATA EXAMPLE4; INPUT ID $ 1-9; LENGTHSTATE$2; LENGTH NUM $ 3.; STATE= SUBSTR(ID,l,2); NUM = INPUT(SUBSTR(ID,7,3),3.); CAXXXXI23 NV PROC PRINT DATA=EXAMPLE4 NOOBS; TITLE 'LISTING OF EXAMPLE 4'; Creating the state code is easy. We use the SUBSTR function. The first argument is the variable from which we want to extract the substring, the second argument is the starting position of the substring. and the last argument is the length of the substring. Note it is the length, NOT the last position of the substring. Also, note the use of the LENGTH statement to set the length of STATE to 2 bytes. Extracting the three digit number code is more work. First, we use a length statement to set the length ofnum to 3. Next, we use the SUBSTR function to pull out the three numbers. However, the result of a SUBSTR function is always a character value. We want numeric output, so we use our INPUT tool to convert the character value to a number. If we had not used a length statement, num would default to 8 when it was converted from character to numeric.. The INPUT function takes the first argument and "reads" it as if it were coming from a file, according to the informat listed as the second argument. So, the SUBSTR function would return the string '123' for the first observation and the INPUT function would convert this to the number 123. Here is a little tip, you may use a longer informat as the second argument and SAS will NOT object. For example, the INPUT statement could have been written as: INPUT (SUBSTR{ID,7,3),8.); and everything would have been fine. If you do not know the length of the string ahead of time, using a longer informat is very useful. Using the SUBSTR Function on the Left Hand Side of the Equal Sign The SUBSTR function, placed on the left-hand side of the equal sign, can place characters in specific locations within a string. Suppose you have some resting heart rate measurements (RHR) and some stress heart rate measurements (SHR) in a SAS data set. You want to print out these values and star high values with an asterisk Here is a program that uses the SUBSTR function on the left of the equals sign to do that: DATA EXAMPLES; INPUT RHR LENGTH RHR_CHK SHR_CHK $4; RHR_CHK = PUT(RHR,3.); SHR_CHK = PUT(SHR,3.); IF SHR GT 130 THEN SUBSTR(SHR_CHK,4,1) = '*'; IF RHR GT 90 THEN 77

4 SUBSTR(RHR _ CHK,4, 1) = '*'; PROC PRINT DATA=EXAMPLES NOOBS; TITLE 'LISTING OF EXAMPLE 5'; First I set the lengths of RHR _ CHK and SHR_ CHK to 4 (three spaces for the value plus one for the possible asterisk). Then, we use the PUT function to perform a numeric to character conversion. The PUT function is similar to the INPUT function. It "creates" the value of the first argument, according to the FORMAT specified in the second argument. In other words "creates" actually assigns the value to the variable on the left of the equal sign. The SUBSTR function then places an asterisk in the fourth position when a value ofwhr is greater than 130 or a value ofrhr is greater than 80. Doing the Previous Example tbe Hard Way It is possible and informative to create the above results without using the SUBSTR function. This solution uses a number of character functions that can be useful. Therefore, this is a great way to demonstrate them. Here is the program: DATA EXAMPLE6; INPUT SHR LENGTH SHR_CHK RHR_CHK $ 4; SHR_CHK = PUT(SHR,3.); RHR_CHK = PUT(RHR,3.); IF SHR GT 160 THEN SHR_CHK = SUBSTR(SHR_CHK,l,3) II'*'; IF RHR GT 90 THEN RHR_CHK = TRIM(RHR_CHK) II'*'; ' PROC PRINT DATA=EXAMPLE6 NOOBS; TITLE 'LISTING OF EXAMPLE 6'; Actually example6 is not much more complex but not as elegant as examples. This program uses the concatenation operator (II) to join the 3 character heart rate values with an asterisk. Since SHR _ CHK and RHR _ CHK were.both assigned a length of 4, we wanted to be sure to concatenate at most the first 3 positions with the asterisk. I did this two ways to show more uses of functions. For the SHR_ CHK variable, we used a SUBSTR function to extract the first 3 bytes. For the RHR_CHK variable, the TRIM function was used. The TRIM function removes trailing blanks from a character string. Unpacking a String To save disk space, I sometimes store several single digit numbers in a longer character string. As an example, storing four numbers as numeric variables with the default 8 bytes each would take up 32 positions of disk space for each observation. Even reducing this to 3 positions each would result in 12 positions ofspace. Instead, store the four digits as a single character value, you will need only 4 positions. This is fine, but at some point, you will want to use the numbers for computational purposes. Here is an easy way to do this: DATA EXAMPLE7; INPUT STRING $ 1-4; DATA UNPACK; SET EXAMPLE7; ARRAY X[4];. DOJ =I T04; X[J] =INPUT( SUBSTR(STRING,J,l),l.); END; DROPJ; PROC PRINT DATA=UNPACK NOOBS; TITLE 'LISTING OF UNPACK'; First, I created an array to hold the four numbers, XI to X4. Do NOT worry if you do not see any variables listed on the ARRAY statement. ARRAY X[4]; is the same as ARRAY X[4] Xl X4; We use a DO loop to cycle through each of the 4 starting positions corresponding to the four numbers we want. As we mentioned before, since the result of the SUBSTR function is a character value, we need to use the INPUT function to perform the character to numeric conversion Parsing a String Parsing means to take something apart based on some rules. The example that follows shows how five separate character values were placed together on a line with either a space, a comma, a semi-colon, a period, or an explanation mark 78

5 between them. Our job, if we should accept it, is to extract the five values and assign them to five separate variables. Fortunately, the SCAN function makes this job easy: DATA EXAMPLES; INPUT LONG_STR $ 1-80; ARRAY PARTS[5] $ 10 PARTS1-PARTS5; DO I= I TO 5; PARTS[!]= SCAN(LONG_STR,I,',;.! '); END; DROP LONG_STR I; DATALINES4; THIS LINE,CONTAINS!FIVE.WORDS ABCDEFGHIJKL AAA;BBB,, PROC PRINT DATA=EXAMPLE8 NOOBS; TITLE 'LISTING OF EXAMPLE 8'; Before our discussion of the SCAN function, I want to explain OAT ALINES4 and the four semi-colons ending the data. If you have data values that can include semicolons,.you cannot use a simple OAT ALINES (or CARDS) statement since the semicolon would signal the end of your data. Instead, the statement DATALINES4 (or CARDS4) is used. This causes the program to continue reading data values until four semicolons are read. The SCAN (char var,n,'list-of-delimiters') function returns the nth "word" from the char_var, where a "word" is defmed as anything between two delimiters. If there are fewer than n words in the character variable, the SCAN function will return a blank. Placing the SCAN function in a DO loop will let us pick out the nth word in the character string. Loeating tbe Position of One String witbin Anotber String Two somewhat similar functions, INDEX and INDEXC, can be used to locate a string, or one of several strings within a longer string. For instance, if you have a string ABCDEFG and want the location of the letters DEF (starting position 4), the following INDEX function could be used: INDEX('ABCDEFG','DEF') this would return a value of 4. If you want to know the starting position of any one of several strings, the INDEXC function can be used. As an example, if you want to find the starting position of either 'BC', or 'FG' in the string 'ABCDEFG', you would code: INDEXC('ABCDEFG','BC','FG'); The function would return a value of2, the starting position of'bc'. A short program which demonstrate these two functions follows: DATAEX_9; INPUT STRING $ 1-10; FIRST= INDEX(STRING,'XYZ'); FIRST_C= INDEXC(STRING,'X','Y','Z'); ABCXYZI ABCX1Y2Z39 ABCZZZXYZ3 PROC PRINTDATA=EX_9NOOBS; TITLE 'LISTING OF EXAMPLE 9'; FIRST and FIRST_ C for each of the the 4 observations are: OBS FIRST FIRST_ C I When the search fails, both functions return a zero. Cbanging Lower Case to Upper Case The UPCASE function does exactly what you expect it to do. This function is useful when a mixture of upper and lower cases values are entered for the same variable. The answer is to convert them all to uppercase. An easy way to handle this is by placing the character variables in an array and UPCASE them all. Here is an example of such a program: DATAEX_IO; LENGTHABCDE$1; INPUT A B C DE X Y; NGRr'E23 OmnFN45 DATA UPWEGO; SETEX_lO; ARRAY ALL_C[*]_CHARACTER_; DO I= I TO DIM(ALL_C); 79

6 ALL_C[I] ""UPCASE(ALL_C[I]); END; DROP I; PROC PRINT DATA=UPWEGO NOOBS; TITLE 'LISTING OF UPWEGO'; This program uses the_ CHARACTER_ keyword to select all the character variables. Running this job converts the values of the variables A, B, C, D and E to upper case. Substituting One Character for Another TRANSLATE is a very nice character function. This tool allows us to convert one character to another in a string. As an example, multiple choices survey answers have been entered as I, 2, 3, 4, 5, these numbers represent the letters 'A' through 'E' respectively. When the answers are printed out, we want to see the characters A' through 'E' rather than' I' through '5'. While formats would accomplish this nicely, this method provides an example for the TRANSLATE function. Here is the code: DATAEX II; INPUT ANS : ANS= TRANSLATE(ANS,'ABCDE','12345'); PROC PRINT DATA=EX_ll NOOBS; TITLE 'LISTING OF EXAMPLE II'; The TRANSLATE function syntax is: TRANSLATE( char_ var,to _string, from_ string) Each value in the from_ string will be translated to the corresponding value in the to_ string. One more use of the TRANSLATE function is changing character variables to numeric variables depending on the value of the character variable. Maybe you want to set values of'n' to 0 and values of 'Y' to I. This is easily done with IF-THEN/ELSE statements but we can also do it using the TRANSLATE function. Here gbes: UPCASE(CHAR),'OI','NY'),l.); NYnyABOI ' PROC PRINT DATA=EX_12 NOOBS; TITLE 'LISTING OF EXAMPLE 12'; The UPCASE function sets all values to upper case. Next, the TRANSLATE function converts values of'n' to '0' and 'Y' to 'I'. Finally, the INPUT function converts the numerals '0' and 'I' to the numbers 0 and I respectively. Conclusions Well, that is all of the character string tools we have time and space to discuss. This hopefully has gotten you interested in using these tools. So go out there and have a ball with strings! Acknowledgements Thanks to Virginia "Gigi" Gruber for reviewing the topics and content of this paper for clarity, completeness, and usefulness. Thanks to Francis "Fran" Larson for reviewing the paper for SAS accuracy. The paper and I are both better for it. SAS is a registered trademark of the SAS Institute Inc., Cary, NC, USA Any comments, questions, or thoughts on your experiences with these functions are most welcome. David can be contacted atthe following address: Fireman's Fund Insurance Company 777 San Marin Drive Novato, CA David Fickbohm@FFIC.COM (415) DATAEX_l2; LENGTH CHAR$ I; INPUT X=INPUT( TRANSLATE( 80