An Introduction to Compressing Data Sets J. Meimei Ma, Quintiles

Transcription

1 An Introduction to Compressing Data Sets J. Meimei Ma, Quintiles r:, INTRODUCTION This tutorial introduces compressed data sets. The SAS system compression algorithm is described along with basic syntax. The efficiency trade offs between saving space and CPU (machine "thinking" time) are explored. Examples of the level of possible space savings are presented. The goal is to provide fundamental knowledge in order to encourage deliberate consideration of the COMPRESS: option. The topic is limited to the COMPRESS= option for SAS data sets. No discussion is included about operating system compression utilities such as ZIP. See Bardsley (1993) for more information about other compression alternatives. Note that if you use tape storage, compression is not an option. Do not confuse the COMPRESS= option with the COMPRESS function for character variables. The primary target audience for this topic includes programmers, data managers, statisticians, and other people who make decisions related to storing large data sets or saving space in general. Only basic knowledge of DATA step programming in the SAS system is assumed. The focus is on Release 6.07, with brief discussion of differences from Essentially all hardware platforms are covered. TERMINOLOGY Efficiency Elements There are two primary categories of elements to consider when evaluating efficiency: machine and human. Machine efficiency elements include computer processing time (CPU) processing time for reading or writing computer data (I/O for Input/Output) storage space. Storage space becomes a matter of efficiency when you consider the cost of storage hardware. Therefore, a technique that requires less space may be thought of as more cost efficient. Efficiency for humans is related to programmer time, level of expertise required, or clarity of final logic. Major components of programmer time include planning programming strategy, designing database structures, writing or revising code, testing programs, running production programs, and writing documentation. Depending on the type of research involved, database design and storage space planning may also be major programming activities. Always consider both machine and human elements when choosing between options for efficiency reasons. The choice may be difficult, or at least ambiguous, since the more machine efficient option can require additional human effort or vice-versa. In the case of compression, a conflict may also occur between the machine elements of CPU and I/O. Large File Environment In large file environments choosing an efficient programming strategy tends to be important. A large file can be defined as a file for which processing all records is a Significant event. This may apply to files that are short and wide, with relatively few observations but a large number of variables, or long and thin, with only a few variables for many observations. The exact size that qualifies as large depends on the computing environment. In a mainframe environment a file may need to have ten thousand records before being considered large. For a microcomputer, even one thousand records may seem to take a long time to process. Batch processing is used more frequently when dealing with large files.. Compression Terms The terms associated with the COMPRES5= option include uncompressed or non-compressed, also called fixed length compressed, also called variable length decompress. For example: records in a compressed data set are decompressed automatically before use by a SAS procedure. 1412

2 COMPRESSION ALGORITHM The compression algorithm has evolved since the first introduction in Release It is a straightforward algorithm designed for general use. Although more efficient compression algorithms could be applied, an advantage is that essentially no differences exist across different hardware platforms. In order to compress a data set, each observation (record) of a SAS data set is evaluated separately. The record is treated as a sequence of bytes. Distinctions between variables, both type and boundaries, are ignored. The algorithm compresses identical consecutive bytes into two or three bytes. Repeated blanks (3 to 129) or binary zeros (3 to 66) are compressed into two bytes. For other types of repeated values (3 to 63), the compressed result takes three bytes. Once you understand the way the algorithm works, predicting the potential level of savings becomes simpler. For character variables, if many blanks (embedded) or missing values (also blanks) occur, then compression could potentially save a great deal of space. On the other hand, character variables that contain few repeated blanks will compress much less. For numeric variables, savings are most likely for integer values stored in the default LENGTH of 8 bytes. For example, if ANSWER1=1 then seven of the bytes are zeros, which would reduce to two bytes. If ANSWER1=0 and ANSWER2=O and they are stored consecutively then those 16 bytes would shrink to two. For real numbers, essentially those with decimal places, the likelihood of repeating bytes is smaller. Note that no precision is lost when an observation is decompressed for use. Release 6.07 includes a change that substantially improves the time required for decompression. According to tests reported by Beatrous et al. (1992), the improvement can be as much as 60%. The decompression time decreases 41% even for an "average" data set. The process differs in that every field, whether compressed or uncompressed, is prefixed with an indication of field length. The 6.06 method for storing compressed fields requires that all fields in a compressed record be searched for an escape code to decide whether or not decompression is needed. You can create the 6.06 compressed format using Release 6.07 by using the ENGINE= or FILEFMT= option. You can read or modify 6.06 compressed data sets using 6.07 directly. COMPRESSION USAGE Syntax There are two options associated with compressed data sets: COMPRESS = YES I NO REUSE = YES I NO Both can be used as a system option or a data set option. Specifying a data set option overrides the system option setting. The COMPRESS: option applies only to output data sets. You cannot change an existing uncompressed SAS data set into a compressed data set without creating a new data set. After a system option statement OPTIONS COMPRESS: YES; is executed, all created data sets will be compressed. This applies to permanent or temporary data sets. For a program that includes a large number of WORK data sets, the extra CPU required could be substantial. Alternatively, you can compress specific data sets. For example: DATA libname.filename (COMPRESS=YES); SETinname; r Data Creation Statements' / If you are planning to insert, delete, or update observations using a compressed data set and space usage is critical, then the REUSE= option may help. Free space is tracked and reused when you specify REUSE=YES when creating a compressed data set. Using Compressed Data Sets Since decompression is automatic, you can almost forget whether or not a data set is compressed or uncompressed. This applies whether you are using a SET statement in a DATA step or a SAS procedure. To keep track, output from the CONTENTS procedure includes an indication of compression status. Using the POSmON option to review the order of variables is helpful if you want to rearrange variables to maximize repeated bytes for optimal compression of large permanent data sets. Release 6.07 includes two features related to compressed data sets not in The SAS Log contains NOTEs about the amount of savings for a data set as 1413

3 Program code Figure 1. SAS Log Notes (VMS) DATA file.newdata (COMPRESS=YES); SET file.olddata; r DATA CREATION STATEMENTS"' Compress note I RUN; NOTE: The data set FILE.NEWDATA has 1000 observations and 200 variables. NOTE: Compressing data set FILE.COMPDATA decreased size by percent. Compressed is 18 pages; un-compressed would required 36 pages. I t ~ t shown in Figure 1. With the reduction in size of 6.07 uncompressed data sets compared to 6.06 (Beatrous et al.), the percentage savings for 6.07 compressed data sets may not be as large. When the COPY procedure is used, the copy has the same attributes as the original. Thus a copy of a compressed data set is also compressed. In 6.06, whether or not the copy was compressed depended on the system default setting. DECISION FACTORS Disadvantages The primary disadvantage of using compressed data sets is the extra au required for decompression. Muller et al. (1992) compared au usage for compressed and uncompressed versions of a data set under MVS. They found that using the SORT, FREQ, MEANS, or CORR procedures on the compressed data set could take twice as much au. Realling in the compressed observations using SET took almost four times as much au. Since compressed observations are variable length, instead of fixed length, certain standard access ~ethods are not as effective. No direct access using POINT= is available although a work around is possible. You could create and index a variable with the value of _N_. In general, using FIRST. and LAST. processing will be slower. The additional complexity associated with using compressed data sets is a factor to remember. With variable length records, predicting space needs becomes more difficult although estimating an upper limit is possible. Since a l2-byte overhead per compressed record is required, a 6.07 compressed data set could be larger than its '1ean" counterpart, which only has a l-bit overhead. If au usage is also a concern, then a balancing act is required between using compressed and uncompressed data 1414 sets. One approach may be to compress only permanent data sets. For.end-user!,pplications, compression may cause problems if not tested thoroughly. Advantages There are other advantages to compression besides the obvious one of saving storage space. Note that specifying COMPRESS=YES adds no extra human effort when using a compressed data set since decompression is automatic. In addition,.no loss of precision occurs, which guarantees that results are identical to those based on an uncompressed version of the data. Compared to other efficiency techniques, making practical use of compressed data sets requires a relatively smail amount of programmer effort, either for decision making or execution. Compression can enhance other efficiency strategies such as minimizing storage length for variables dropping unneeded variables using indexes when appropriate. You can minimize the space required for specific variables by using LENGTH<8 or storing categorical values as character. Note that using a shorter LENGTH can result in precision problems, especially when data must be transferred across hardware. platforms. Storing categorical. variables such as SEX. as character may be better t~ just decreasing the LENGTH. Always consider using DROP/KEEP to assure that only required variables remain in temporary or permanent data sets. When you use an index to access observations in a compressed data set, only the required observations are decompressed. So by indexing a compressed data set, the amount of extra CPU required for decompression may decrease substantially. Combining any or all of these techniques with compression can efficiently save the most storage space.

4 Figure 2. Thin Versus Wide Space Savings 40 Percent 30 Improvement Short Long --Thin--- Short Long ---Wide-- EXAMPLES Example! As shown in Figure 2, Muller et al. confinned that wide data sets benefit more from compression. They created test data sets under MVS. Short meant 10,000 observations compared to 100,000 in the long data sets. Thin data sets included 20 variables while wide ones contained 110 variables. Regardless of the number of observations, about 45% savings in space occurred, for the wide data sets, which was more than twice as much as the 20% savings for thin data sets. Example 2 Bardsley demonstrated the effect of the compression algorithm by creating data sets emphasizing specific characteristics., The results under AI)( were, comparable to MVS. Wide data sets contained 120 variables. The results by type of value are: Type Numeric Real Single-digit Integer Numeric Missing Character Missing Percent Savings o As expected, data sets with a greater occurrence of repeated bytes benefit the most. Example 3 The table in Figure 3 shows space savings for actual clinical trial data sets under MVS. The data sets are shown sorted by observation length. In ADVERSE and MEDS, the character variables include a large number of blanks. ADVERSE is only half character variables, but they account for 83% of the storage space in fixed length records. Most of the values in EFFBASE are zeros or ones. In general, the wider data sets compressed more effectively with the Figure 3. Space Savings in Clinical Trials Number Record % Character % Space Data set of Variables Length Vars Length Saved 1) VITALS ) EFFSUMS ) ADVERSE ) MEDS' ) EFFBASE ) LABS

5 exception of LABS. However, this is not a surprise since lab data consists mostly of real numbers. CONCLUSION Generally, the trade offs are clear when evaluating whether or not to compress data sets. You must choose between the importance of saving space versus using the least amount of CPU, assuming both save money. The chacteristics of obvious compression situations include any or all of the following: not enough space available wide observations with repeated values "blank" character variables expected infrequent processing of large data sets subset processing of large indexed data sets >50% savings derr.onstrated. In any Release 6.07 computing environment, you can benefit from using the COMPRESS: option as long as you compress data sets deliberately. RECOMMENDED READING Compression Bardsley, P., (1993), "Space-saving Tools: Compression in SAS 6.07," Proc. of the First Annual Southeast SAS Users Group Conference, Cary, NC: SAS Institute Inc., Beatrous, S. and Stokes, J.T., (1992), "I/O Performance Improvements in Release 6.07 of the SAS System under MVS, CMS, and VMS," Proc. of the Seuenteenth Annual SAS Group Inti. Conference, Cary, NC: SAS Institute Inc., Clifford, W., Beatrous, S., Stokes, J.T., and Mosmon, K. (1989), "Using New SAS Database Features and Options," Proc. of the Fourteenth Annual SAS Group IntI. Conference, Cary, NC: SAS Institute Inc., Muller, S.M., Hardy, K.A., and Brown, K.J., (1992), "Getting It And Keeping It With SAS Software," Proc. of the Seuenteenth Annual SAS Group IntI. Conference, Cary,NC: SAS Institute Inc., SAS Institute Inc. (1990), SAS Companion for the VMS Environment, Version 6, First Edition, Cary, NC: SAS Institute Inc. (Compression related topics on pp. 137, 141,143,345,408.) SAS Institute Inc. (1990), SAS Language: Reference, Version 6, First Edition, Cary, NC: SAS Institute Inc. COMPRESS: option, pp ; REUSE= option, pp Efficiency Techniques Howard, N. (1991), "Efficiency Techniques for Improving I/O and Processing Time in the DATA Step," Proc. of the Fourteenth Annual SAS Group IntI. Conference, Cary, NC: SAS Institute Inc., Ma, J.M. (1991), "Effieency Revisited: Large Files and Release 6.06," Proc. of the Sixteenth Annual SAS Group IntI. Conference, Cary, NC: SAS Institute Inc., Mackiernan, Y.D. (1989), "Don't Do Anything You Don't Have To: Elementary Strategies For Processing Large Data Sets With SAS Software," Proc. of the Fourteenth Annual SAS Group IntI. Conference, Cary, NC: SAS Institute Inc., Muller, K.E., Smith, J., and Bass, J. (1982), "Managing 'not small' Datasets in a Research Environment:' Proc. of the Seuenth Annual SAS Users Group IntI. Conference, Cary, NC: SAS Institute Inc., ' SAS Institute Inc. (1990), SAS Programming Tips: A Guide to Efficient SAS Processing, Cary, NC: SAS' Institute Inc. Smith, u. (1991), "Effieent Use of Numeric and Character Data Types," Proc. of the Sixteenth Annual SAS Group Inti. Conference, Cary, NC: SAS Institute Inc., CONTACT ADDRESS Dr. J. Meimei Ma Quintiles P. O. Box Research Triangle Park, NC SAS is a registered trademark of SAS InstibJte Inc. in the USA and other countries. lid indicates USA registra~on. 1416