A measure of compression gain for new alphabet symbols in data-compression

Transcription

1 A measure of comression gain for new alhabet symbols in data-comression Richard Fredlund arxiv: v [cs.it] 9 Feb 204 Abstract In coding theory it is widely known that the otimal encoding for a given alhabet of symbol codes is the Shannon entroy times the number of symbols to be encoded. However, deending on the structure of the message to be encoded it is ossible to beat this otimal by including only frequently occurring aggregates of symbols from the base alhabet. We rove that the change in comressed message length by the introduction of a new aggregate symbol can be exressed as the difference of two entroies, deendent only on the robalities of the characters within the aggregate lus a correction term which involves only the robality and length of the introduced symbol. The exression is indeendent of the robality of all other symbols in the alhabet. This measure of information gain, for a new symbol, can be alied in data comression methods. I. Introduction The symbol alhabet used for most data comression schemes are either individual characters, syllables (for examle [, 2]), or words (for examle [3, 4]). With a few technicalities (e.g. unctuation in word encodings) these alhabets essentially form a artition of the text into more or less, similar arts. Saces have been added to words [5] and frequently occurring 2,3 and 4 grams have also been used to ad out the alhabet, [6]. However, adding sequences of artrary length to a base alhabet of characters, based on information gain, to be used by a variable length comression scheme such as Arithmetic coding [7] or Huffman encoding [8] has not been considered. Shannon s theorem [9] states that the best (er character) comression achievable for a message (and closely aroximated by Arithmetic encoding [7, 0]) is it s entroy relative to the alhabet. At first glance adding symbols to a base alhabet in order to further reduce this rate of comression may seem counter to Shannon s theorem. However, Shannon s theorem is true only for a fixed alhabet, and there is no reason why, deending on the structure of the message, adding aggregate symbols may not decrease the comression further. We ask the question at what oint is an aggregate symbol, comosed of two or more alhabet symbols worth adding to the model. For examle when encoding simle text, comosed of 26 alhabet symbols at what frequency is the comound symbol the comosed of three alhabet symbols worth adding to the model? To realize that for high enough frequencies adding the symbol may roduce a decrease in the exected message length notice that the number of symbols in the message will decrease by two times the number of occurrences of the word the. As each occurrence of the word the reresents a condensing of three alhabet symbols into just one new symbol. We define the gain of an new aggregate symbol to be the decrease in otimal encoding length with the introduction of the symbol. However, with the addition of a new aggregate symbol it is also ossible that the otimal encoding length of the message with resect to the alhabet will increase, rather than de-

2 crease. The gain therefore may be ositive or negative. In Section II we rove our main result, that the average gain, as measured in ts er character, of introducing a new symbol S can be exressed as the difference of two entroies, each exressed only in terms of the relative frequencies of the alhabet symbols, which are being comned to form S lus a correction term which involves only the robality and length of S. In Section III, we erform an exeriment to demonstrate the use of the information gain measure to create an aggregate alhabet for data comression. Starting with a base alhabet of just 72 distinct characters and an otimal comression rate of 4.5 ts er character, we discover an alhabet with 39 aggregate symbols which effectively reduces the rate of comression to under 3 ts er character. II. Main Result otation: start by considering a message M = m, m 2,..., m of character symbols from the alhabet A x = {a, a 2,..., a K } with known character frequencies F x = { f a, f a2,..., f ak }. The robalities of each character symbol are P x = { a, a2,..., ak } where ai = f a i for each i in {, 2,..., K}. II. Theorem We rove that when a new aggregate symbol S = b b 2...b r comosed of r (not necessarily distinct) symbols from the original alhabet A x is added to the alhabet and {S} reresents the set of distinct characters in S, then the er character information gain can be exressed as the difference of two entroies: log 2 λ log 2 () λ (Where λ reresents the robality of symbol b i after occurrences of b i in S have been removed) lus a correction term exressed only in terms of s the robality of S in the message, and r. Letting µ s = ( s (r )), the correction term is again the difference of two entroies and is given by: µ s log 2 µ s s log 2 s (2) We call µ s the rescale constant, and the length of the new message is µ s. II.2 Proof By Shannon s information theory [9] the otimal encoding for this message gives an average code-length for each encoded symbol of: L(C, X) H(X) = x A x x log 2 x (3) where H(X) is the entroy of the message M for this robality model, described by (A x, P x ). The otimal comressed code length for the whole message M is the message length times this entroy: L(M, A x ) = H(x). (4) This can be re-written in terms of the character frequencies as: L(M, A x ) = log 2 x A x f x log 2 f x (5) using the identity log 2 x = log 2 x and by noting that the sum of the frequencies x Ax f x is equal to. Let S = b b 2...b r be an aggregate symbol comosed of r symbols from the original alhabet A x so that b i A x for i in {, 2,..., K}, (ote that b i and b j may not be distinct as S may contain more than one coy of some characters in A x ), and let f S be the number of non-overlaing occurrences of S in the message M. Adding S to the alhabet gives a new alhabet of character symbols, A x = {S, a, a 2,..., a K }, a new set of robalities P k = { s, a,..., a K } and a new set of character frequencies F k = { f s, f a,..., f a K }. 2

3 The message length will also change. This is because, for each of the f S occurrences of S in the message, r symbols from the original character sequence M will be condensed into just one character in the new alhabet A x. This gives a new message length of: = f S (r ) (6) By counting the number of characters a i in the original message and subtracting occurrences which aear in S we get the udated character frequencies: f a i = f ai f S S(a i ) (7) where S(a i ) is the number of times the character symbol a i aears in S. ote for characters not in S the number of occurrences will remain unchanged and f a i = f a. Substituting 6 and 7 into 5 gives the new exected code length for the message, with the introduction of the new aggregate symbol S. (8) L(M, A x) = log 2 f S log 2 f S f ai log 2 f ai a i / S f a i log 2 f a i Here M is the message written in the new alhabet A x and L(M, A x) is the new exected coding length for the message. We have slit the summation of the log frequencies into three distinct arts. Firstly, the term containing f S the number of occurrences of the newly introduced character S. Secondly, those characters not in S whose frequency do not change with the introduction of S and finally, those characters which are in S and whose frequencies do change. The advantage of this formulation is that the terms involving frequencies not in S cancel and comarison of 5 and 8 leads to a formula, in terms of the symbol frequencies, for the exected coding gain gain(s) over the whole message of introducing S into the alhabet: gain(s) = log 2 log 2 + f S log 2 f S f ai log 2 f ai + f a i log 2 f a i (9) The addition of the aggregate symbol S to the alhabet A x will shorten the otimally encoded message if and only if gain(s) > 0. The disadvantage of this formulation is that the exected information gain er symbol, (which is gain(s) ), aears to be deendent on, due to the log 2 term. To see that this ought not to be the case consider the scenario of a new message M new which is recisely M reeated twice, such that all character frequencies are exactly doubled. We would exect to see the overall gain for this new message to be 2 gain(s). We now recast 9 in terms of robalities. Starting by relacing f s with s in 6 to give = µ s, where µ s = ( s (r )), which also give f a i = a i µ s. Substituting these in 9 gives: gain(s) = log 2 µ s log 2 µ s + S log 2 S ai log 2 ai + a i µ s log 2 a i µ s (0) This simlifies by noticing that all terms are of the form A log 2 A and using the identity A log 2 A = A log 2 A log 2 A. Suming u the log 2 terms everything cancels by noting that P s r = ai S s s(a i ). This gives a sum over terms all of the form A log 2 A, the first of which is log 2 = 0 which also cancels to give: 3

4 gain(s) =µ s log 2 s log µ 2 s s + ai log 2 ai a i µ s log 2 a i µ s () We finish by noting that a i µ s = f a f S S(a i ) = ai s s(a i ) and let λ ai = ai s s(a i ). The average information gain er character symbol can therefore be written as: char_gain(s) = log 2 λ log 2 λ + µ s log 2 µ s s log 2 s (2) which is indeendent of the robalities, of all other alhabet symbols excet those contained in S and the robality of S itself. In more concise notation this can be rewritten as: char_gain(s) =H(µ s, ai : a i {S}) III. H( s, λ ai : a i {S}) exeriment (3) Here we erform an exeriment to demonstrate the use of the information gain measure to create an aggregate symbol alhabet for lossless text comression by finding an alhabet of aggregate symbols for a single document, Alice in wonderland (alice29.txt) from the Canterbury text corus. While both encoder and decoder are assumed to have access to a small basic alhabet of characters, without the need for transmission with the message, all other characters, (from the character set, and from the aggregate symbols) must also be encoded in the tstream for transmission. We therefore make a distinction between the message encoding, and the alhabet encoding. For each new symbol the alhabet encoding will increase in size, so the message encoding must decrease by at least this amount to be worth adding to the alhabet. III. Alhabet encoding scheme We start with a very basic standard alhabet of just 64 characters, which are assumed to be known reviously, by both the encoding and decoding software. This character set is simly the uercase and lowercase letters, and 2 unctuation symbols. These are, sace, newline, colon, semi-colon, exclamation mark, oen and close brackets, hyhen, comma, fullsto and question mark. The first 5 ts of the alhabet encoding are used to encode, MXBITS, the number of ts required to encode a frequency count in this document. The maximum being 32 ts assigned to each frequency count, which would allow u to 4294 million occurrences of any character within the document. MXBITS is determined by the maximum number of occurrences of any character in the document. For Alice29.txt, MXBITS is set to 5 indicating that 5 ts should be used to record the frequency of each alhabet symbol. (ote that in ractice this could be considerably reduced by noting that this is only the maximum ever required, and in ractice most symbol frequencies for this document fall into the range and could be exressed using only 8 ts). ext each of the 64 characters from the standard alhabet is assigned 5 ts to record it s frequency in the document (960 ts). The next 8 ts in the encoding are assigned to UMCHAR, the number of nonstandard characters in the document. We kee things simle here and assume all other characters in the document are 8 t ascii codes (unicode symbols are not considered). In Alice.txt there are 8 characters not included in the standard alhabet. These are aostrohe, two forms of seech-mark,, oen and close, square bracket, and the nu- 4

5 merals, 2 and 9. These are encoded with 8 ts for ascii code, and 5 ts for character frequency. That is 23 8 = 84 ts. Before any aggregate symbols are added therefore, the number of ts, used by the alhabet encoding are = 57. The next 5 ts of the encoding are allocated to AGCOUT, the number of aggregate symbols to be included in the alhabet. At initialization, AGCOUT is set to 0 and the alhabet contains = 72 characters. Aggregate symbols in this imlementation consist of two ordered symbols, from the already existing alhabet. As new aggregates are added, they may consist of comnations of revious aggregates, or of characters from the base alhabet. As the aggregate alhabet increases in size, the number of ossible aggregates also increases. For an alhabet of length, the first and second symbol both have ossible values giving a total of 2 ossible new aggregate symbols. The number of ts required to encode each aggregate symbols deends on the number of ts required to encode a value in{,..., }. At initialization = 72, hence the number of ts to encode each sub-symbol in the aggregate is given by log 2 (AGCOUT+72). 5 ts must also be allocated to record the frequency count. This gives a total of 5+2 log 2 (AGCOUT+72) ts for each new aggregate symbol. This starts of at 29 ts and increases by 2 ts each time gets too large. III.2 Search algorithm Here we have ket the search for aggregate symbols with maximum, comression gain as simle as ossible. At each ste the symbol gain is calculated for each of the ossible 2 aggregates and the highest gain is selected. If this gain is higher than the number of ts necessary for the alhabet encoding, then the symbol is added. This aroach becomes infeasible as grows large, and it is exected that better search algorithms can be found to find aggregates with a high exected gain. gain symbol freq 6538 he the in ing ou [s] the , [s] [nl][nl] [nl][nl] ve nd and , [nl][nl] , [s] and [s]the[s] th Al Ali 395 III.3 Results The brute force algorithm described above was imlemented on Alice29.txt from the Canterbury Corus. The uncomressed document contains 48480, 8 t ascii characters. At initialisation there were 72 distinct characters in the base alhabet. The alhabet encoding occuied 57 ts and the otimal message encoding length which could be closely aroximated by arithmetic encoding was ts. So the total message encoding before any aggregates were introduced was 6724 ts. This gives a message encoding of 4.5 (2.d..) ts er character. and a total encoding of 4.52 (2.d..) ts er character. Subsequently 39 aggregate symbols were added to the alhabet. This increased the number of ts required by the alhabet encoding to ts. However, the otimal message encoding droed to ts, leaving a total encoding length of 4376 ts. This gives a message encoding of 2.63 (2.d..) ts er character and a total encoding of 2.95(2.d.) ts er character. The t rate could have droed further 5

6 however by this oint the exeriment had slowed down considerably, so the exeriment was stoed soon after the t rate of 3 ts er character was reached. The above table shows the first 20 aggregate symbols selected by the algorithm. otice that the first 8 symbols seem quite generic and might well be informative symbols for many text documents. It is interesting to note that the second symbol is the while th is not selected until the 8th symbol, instead he is selected as the first symbol, having high frequency and an information gain of 6538 ts. However, symbol 9 and 20 are Al and Ali and the next two symbols are Alic and Alice resectively. These symbols are secific to the document. Other aggregate symbols, which are recognizable as being secific to Alice29.txt and aear in the alhabet are, Dormouse, Hare, Mock, curious, remark, said Alice, little, the Queen and Duchess. otice that the information gain is not always strictly decreasing with each new aggregate symbol. The information gain of ing, and and Ali are each higher than their immediate redecessors, in, nd and Al, resectively. Meaning that where a longer aggregate symbol has high gain, the shorter aggregates which build u to the longer symbol, may have a lower gain. III.4 Discussion of Results The aggregate alhabet found, achieves a 4.7% (.d..) reduction in message encoding length, and a 34.7% (.d..) reduction in total encoding length relative to the Shannon otimal for the basic alhabet of single characters. Reaching a message encoding of 2.63 (2.d..) ts er character and a total encoding of 2.95 (2.d.) ts er character before the exeriment was stoed. Both the alhabet encoding, and the search algorithm used to find new aggregate symbols, are not ideal, and leave considerable scoe to be imroved uon, in develoing a ractical data comression scheme. However, the results obtained demonstrate the basic efficacy of using this measure of information gain to create an alhabet for data comression. The two drawbacks of this imlementation are: firstly, it can only find larger chunks which are informative by working u through smaller chunks first. e.g. al, ali, alic, alice. (which requires the smaller chunks to also be informative); secondly, once a larger chunk is found, the smaller chunks which lead u to it, will often become less informative in the alhabet. However it is interesting that the airwise-encoder is able to find some longer sequences such as the Queen, and said Alice, showing there is enough information gain in the airwise stes to lead u to these articular sequences. Better comression rates could be achieved by using a dictionary aroach and initially adding whole words before starting the airwise comnation of symbols, and also removing smaller chunks if they become uninformative. IV. Conclusion The contribution in Section II, has been to find a simle formulation which measures the change in comression length when adding a new aggregate symbol to an alhabet code for data comression. This formulation is concise and indeendent of the frequency of all other characters from the alhabet not in the new symbol. Comutation time for this measure is therefore indeendent of size of the alhabet. This measure is otentially alicable to any variable length coding scheme with an alhabet and could be used with either static or adative schemes such as PPM [0]. One advantage of using this measure to create alhabets for symbol codes is that no rior knowledge of the structure of the data is required, and like Lemel-Ziv [] it can be alied to any data which contains reetitions. 6

7 References [] J. Lansky and M. Zemlicka, Comression of small text files using syllables., in DCC,. 458, IEEE Comuter Society, [2] I. Akman, H. Bayindir, S. Ozleme, Z. Akin, and S. Misra, A lossless text comression technique using syllable based morhology., Int. Arab J. Inf. Technol., vol. 8, no., , 20. [3] A. Moffat and R. Y. K. Isal, Word-based text comression using the burrowswheeler transform., Inf. Process. Manage., vol. 4, no. 5, , [4] A. Farina, G. avarro, and J. R. Parama, Boosting text comression with wordbased statistical encoding., Comut. J., vol. 55, no.,. 3, 202. [5] R. Y. K. Isal, A. Moffat, and A. C. H. gai, Enhanced word-based block-sorting text comression., in ACSC (M. J. Oudshoorn, ed.), vol. 4 of CRPIT, , Australian Comuter Society, [6] S. Grabowski, Text rocessing for burrows-wheeler block sorting comression., in VII Konferencja Sieci i systemy Informatycne-Teorra, Projekty, Wdrozenia,Lodzkiej, 999. [7] F. Run, Arithmetic stream coding using fixed recision registers., IEEE Transactions on Information Theory, vol. 25, no. 6, , 979. [8] D. Huffman, A.: A method for the construction of minimumredundancy codes, in Proceedings of the IRE, 952. [9] D. MacKay, Information Theory, Inference, and Learning Algorithms. Cambridge University Press, [0] P. G. Howard and J. S. Vitter, Analysis of arithmetic coding for data comression., in Data Comression Conference (J. A. Storer and J. H. Reif, eds.),. 3 2, IEEE Comuter Society, 99. [] J. Ziv and A. Lemel, Comression of individual sequences via variable-rate coding, IEEE Trans. Inf. Theor., vol. 24, , Set