Arithmetic Coding. Prof. Ja-Ling Wu. Department of Computer Science and Information Engineering National Taiwan University

Transcription

1 Arithmetic Coding Prof. Ja-Ling Wu Department of Computer Science and Information Engineering Nationa Taiwan University

2 F(X) Shannon-Fano-Eias Coding W..o.g. we can take X={,,,m}. Assume p()>0 for a. The cumuative distribution function F() is defined as F a p a P() 0

3 Consider the modified cumuative distribution function F Pa P a where F denotes the sum of the probabiities of a symbos ess than pus haf the probabiity of the symbo. Since the r.v. is discrete, the cumuative distribution function consists of steps of size p(). The vaue of the Function F is the midpoint of the step corresponding to. Since a the probabiities are positive, F(a)F(b) if ab, and hence we can determine if we know F. Thus the vaue of F can be used as a code for. 3

4 But in genera F is a rea number epressibe ony by an infinite number of bits. So it is not efficient to use the eact vaue of Fas a code for. If we use an approimate vaue, what is the required accuracy? Assume that we round off F to () bits (denoted by F ). Thus we use the first () bits of F as a code for. By definition of rounding off, we have if F og F p p, F then F and therefore F ies within the step corresponding to. Thus () bits suffice to describe. 4

5 In addition to requiring that the codeword identify to corresponding symbo, we aso require the set of codewords to be prefi-free. Consider each codeword z, z,, z to represent not a point but the interva [0. z z z, 0. z z z +/ ]. The code is prefi-free iff the intervas corresponding to codewords are disjoint. The interva corresponding to any codeword has ength -(), which is ess than haf the height of the step corresponding to. The ower end of the interva is in the ower haf of the step. Thus the upper end of the interva ies beow the top of the step, and the interva corresponding to any codeword ies entirey within the step corresponding to that symbo in the cumuative distribution function. Therefore, the intervas corresponding to different codewords are disjoint and the code is prefi-free. Note that this procedure does not require the symbos to be ordered in terms of probabiity. 5

6 Since we use og p bits to represent, the epected ength of this code is L H p p og p if the probabiities are ordered as p p p m then Eampe: H L H p() F() in binary og codeword F F p 6

7 Arithmetic coding Huffman coding : a bottom-up procedure the cacuation of the probabiities of a source sequences of a particuar bock ength and the construction of the corresponding compete code tree. A better scheme is one which can be easiy etended to onger bock engths without having to redo a the cacuations. Arithmetic coding : a direct etension of Shannon- Fano-Eias coding cacuate the probabiity mass function p( n ) and the cumuative distribution function F( n ) for the source sequence n. 7

8 We can use a number in the interva (F( n ) P( n ),F( n )] as the code for n. Shannon-Fano-Eias code For eampe: epressing F( n ) to an accuracy of og code for the source. wi give us a the codeword corresponding the any sequences ies within the step in the cumuative distribution function. The codewords are different sequences of ength n. However, this procedure does not guarantee that the set of codewords is prefi-free. P n 8

9 We can construct a prefi-free set by using F rounded off to og P bits, as in the previous eampe. Arithmetic coding : Keep track of F( n ) and P( n ) We assume that we have a fied bock ength n that is known to both the encoder and the decoder. With a sma oss of generaity, we assume that the source aphabet is binary. We assume that we have a simpe procedure to cacuate P(,,, n ) for any string,,, n. We use the natura eicographic order on strings, so a string is greater than a string y if i = and y i =0 for the first i such that i y i. Equivaenty, 9

10 y i.e., if 0 if i i i i y i the corresponding binary numbers satisfy 0 y i, We can arrange the string as the eaves of a tree of depth n, where each eve of the tree corresponds to one bit. 0

11 0 T T T 3 0 In the above tree, the ordering >y corresponds to the fact that is to the right of y on the same eve of the tree.

12 The sum of the probabiities of a the eaves to the eft of n is the sum of the probabiities of a the subtrees to the eft of n. Let be a subtree starting with k- 0. The probabiity of this subtree is 0 k T 0 as can rewrite Therefore we 0 0 to theeft of :T is 0 n n T y n n n k y y n k k P P T y P F F P y y P P T n n n k k

13 E: if P(0)=, P()= in the above Binary tree, then F(00) = P(T ) + P(T ) + P(T 3 ) = P(00) + P(00) + P(00) = ( ) + ( ) + ( ). To encode the net bit of the source sequence, we need ony cacuate P( i i+ ) and update F( i i+ ) using the above scheme. Encoding can therefore be done sequentiay, by ooking at the bits as they come in. 3

14 To decode the sequence, we use the same procdeure to cacuate the cumuative distribution function and check whether it eceeds the vaue corresponding to the codeword. We then use the above binary tree as a decision tree. At the top node, we check to see if the received codeword F( n ) is greater than P(0). If it is, then the subtree starting with 0 is to the eft of n and hence =. Continuing this process down the tree, we can decode the bits in sequence. Thus we can compress and decompress a source sequence in a sequentia manner. 4

15 The above procedure depends on a mode for which we can easiy compute P( n ). Two eampes of such modes are i.i.d. source, where P n n P i and Markov source, where P n n P P i i i In both cases, we can easiy compute P( n n+ ) from P( n ). Note that it is not essentia that the probabiities used in the encoding be equa to the true distribution of the source. i 5

16 In some cases, such as in image compression, it is difficut to describe a true distribution for the source. Even then, it is possibe to appy the above arithmetic coding procedure. G.G. Langdon. An introduction to arithmetic coding, IBM Journa of Research and Deveopment, vo. 8, pp ,

17 7 Competitive Optimaity of the Shannon code Theorem: Let () be the codeword engths associated with the Shannon code and et () be the codeword engths associated with any other code. Then Proof: Since - () by the Kraft inequaity. No other code can do much better than the Shannon code most of the time. ' c r c P, ) ( ' og ' og ' ) ( ) ( ' ) ( : ' ) ( ) ( ' c c p c r p r p r r c p p P c P c P c P

18 Theorem: For a dyadic probabiity mass function p(), et og be the wordengths of the binary p Shannon code for the source, and et () be the ength of any other uniquey decodabe binary code for the source. Then P r ' P ' r with equaity iff ()=() for a. The codeength assignment og competitivey optima. p is uniquey 8

19 Proof: Define the function sgn(t) as foows:, if t>0 sgn(t) = 0, if t=0 -, if t<0 sgn() t - - Note: sgn(t) t - True for integer vaue of t for t=0,,, 9

20 0 0 ' sgn ' sgn ' ' ' ' ' ' ) ( ) '( : ) ( ) '( : r r p E p p p P P

21 Coroary: For non-dyadic probabiity mass function Esgn( ( ) '( ) ) ' og where p and is any other code for the source. 0

22 The Huffman coder generates a new codeword for each input symbo. the ower imit on compression for a Huffman coder is one bit per input symbo Higher compression ratio can be achieved by combining severa symbos into a singe unit ; however, the corresponding compeity for codeword construction wi be increased Another probem with Huffman coding is that the coding and modeing steps are combined into a singe process, and thus adaptive coding is difficut. If the probabiities of occurrence of the input symbos change, then one has to redesign the Huffman tabe from scratch.

23 Arithmetic coding is a ossess compression technique that benefits from treating mutipe symbos as a singe data unit but at the same time retains the incrementa symbo-by-symbo coding approach of Huffman coding. Arithmetic coding separates the coding from modeing. This aows for the dynamic adaptation of the probabiity mode without affecting the design of the coder. Encoding Process : AR coding : a singe codeword is assigned to each possibe data set. each codeword can be considered a haf-open subinterva in the interva [,0) 3

24 By assigning enough precision bits to each of the codewords, one can distinguish one subinterva from any other subintervas, and thus uniquey decode the corresponding data set. Like Huffman codewords, the more probabe data sets correspond to arger subintervas and thus require fewer bits of precision. E : symbo Probabiity Huffman codeword k u w e 0.3 r 0. 00? 0. 0 input string : u u r e? 0,0,00,00,00,,0 8 bits 4

25 Input ?? r r e e w u k u u

26 . At the start of the process, the message is assumed to be in the haf-open interva [0,). The interva is spit into severa subintervas, one subinterva for each symbo in our aphabet. The upper imit of each subinterva is the cumuative probabiity up to and incuding the corresponding symbo. The ower imit is the cumuative probabiity up to but not incuding the symbo. Si Pi subinterva end of message marker k 0.05 [0.00, 0.05) 0. [0.05, 0.5) u 0. [0.5, 0.35) w 0.05 [0.35, 0.40) e 0.3 [0.40, 0.70) r 0. [0.70, 0.90)? 0. [0.90,.00) 6

27 . When the first symbo, appears, we seect the corresponding subinterva and make it the new current interva. The intervas of the remaining symbos in the aphabet set are then scaed accordingy. Let Previous ow and Previous high be the ower and the upper imit for the od interva. Let Range = Previous ow - Previous high After input, the ower imit for the new interva is: Previous ow + Range subinterva ow of sumbo. the upper imit for the new interva is: Previous ow + Range subinterva high of symbo. 7

28 Previous ow = 0, Previous high = Range = 0 = subinterva ow of = 0.05 subinterva high of = 0.5 New interva ow = = 0.05 New interva high = = 0.5 After input, [ 0, ) [ 0.05, 0.5) 3. After the nd, we have New interva ow = [ ] 0.05 = 0.06 New interva high = [ ] 0.5 = 0.0 8

29 4. For the 3rd input u, we have New interva ow = [ ] 0.5 = 0.07 New interva high = [ ] 0.35 = The cacuations described in previous steps are repeated using the imits of the previous interva and subinterva ranges of the current symbo. This yieds the imits for the new interva. After the symbo?, the fina range is [ , ) 6. There is no need to transmit both vaues of the bounds in the ast interva. Instead, we transmit a vaue that is within the fina range. In the above eampe, any number such as , ,, coud be used. if is used = bits are required Arithmetic coding yieds better compression because encodes a message as a whoe new symbo instead of separate symbos. 9

30 Decoding Process Given the symbo probabiities, to each symbo in our aphabet we assign a unique number (i) and we associate a cumuative probabiity vaue cumprob i. Si i Cumprob i K u w e r? Given the fractiona representation of the input codeword vaue, the foowing agorithm outputs the corresponding decoded message. 30

31 DecodeSymbo (vaue): Begin Previous ow = 0 Previous high = Range = Previous high Previous ow. Repeat Find i such that vaue Previous Cumprob i Range Cumprob Output symbo corresponding to i from the decoding tabe. Update: Previous high = Previous ow + Range Cumprob i- Previous ow = Previous ow + Range Cumprob i Range = Previous high Previous ow Unti symbo decoded is? End ow i 3

32 E:. Initiay, Previous ow =0, Previous high =, and Range=. For i=6, vaue Previous Cumprob i Thus, the first decoded symbo is. Update: Previous high = 0.5 Previous ow = 0.5 Range = 0.0 Range ow Cumprob i. We repeat the decoding process and find that i=6 satisfies again the imits vaue 0.05 Cumprob i Cumprob 0.0 Thus the nd decoded symbo is. Update: Previous high = 0.0 Previous ow = 0.06 Range = 0.04 i 3

33 3. Repeating the decoding process yieds the i=5 satisfies the imits vaue 0.06 Cumprob i Cumprob 0.04 Thus the 3 rd decoded symbo is u Update: Previous high = Previous ow = Range = Repeat the decoding process unti? is decoded, then terminate the decoding agorithm, i 33

34 Impementation Issues : Incrementa Output The encoding method we have described geneates a compressed bit stream ony after reading the entire message. However, in most impementation, and particuary in image compression, we desire an incrementa transmission scheme. From the encoding figure, we observe that after encoding u, the subinterva range is [0.07, 0.074). In fact, it wi start with the vaue 0.07; hence we can transmit the first two digits 07. After the encoding of the net symbo, the fina representation wi begin with 0.07 since both the upper and the ower imits of this range contain Thus, we can transmit the digit. We repeat this process for the remaining symbos. Thus, incrementa encoding is achieved by transmitting to the decoder each digit in the finia representation as soon as it is known. The decoder can perform incrementa cacuation too. : encoding / decoding detais can be found in 我把電腦變大了 ( 黃鶴超 ) 數字編碼篇 34

35 High-precision arithmetic Most of the computations in arithmetic coding use foating-point arithmetic; however, most ow-cost hardware impementations support ony fied-point arithmetic. Furthermore, division (used by the decoder) is undesirabe in most impementations. Consider the probem of arithmetic precision in the encoder. During encoding, the subinterva range is narrowed as each new symbo is processed. Depending on the symbo probabiities, the precision required to represent this range may grow; thus, there is a potentia for overfow or underfow. For instance, in an integer impementation, if the fractiona vaues are not scaed appropriatey, different symbos may yied the same imits of Previous high and Previous ow ;that is, no subdivision of the previous subinterva takes pace. At this point, encoding woud have to be abnormay terminated. 35

36 Let subinterva imits Previous ow and Previous high be represented as integers with C bits of precision. The ength of a subinterva is equa to the product of the probabiities of the individua events. If we represent this probabiity with f bits of precision to avoid overfow or underfow, we required f c+ and f+c p, where p is the arithmetic precision for the computations. For AR coding on a 6-bit computer, p=6. If c=9, then f=7. If the message is composed of symbos from a k- symbo aphabet, then f k. Thus, a 56-symbo aphabet cannot be correcty encoded using 6-bit arithmetic. 36

37 Probabiity Modeing Thus far, we have assumed a priori knowedge of the symbo probabiities p i. In many practica impementations that use the arithmetic coder, symbo probabiities are estimated as the pies are processed. This aows the coder to adapt better to changes in the input stream. A typica eampe is a document that incudes both tet and images. Tet and images have quite different probabiity symbos. In this case, an adaptive arithmetic coder is epected to perform better than a nonadaptive entropy coder. 37

38 good references :. Witen, Nea, Ceary, Arithmetic coding for data compression, Communication ACM, 30(6), pp , June Mitche & Pennebaker, Optima hardware and software arithmetic coding procedures for the Q- coder, IBM J. of Research and Deveopment, 3(6), pp , Nov H.C. Huang, & Ja-Ling Wu, Windowed Huffman coding agorithm with size adaptation, IEE Proceeding-I, pp. 09-3, Apri Chia-Lun Yu & Ja-Ling Wu, Hierarchica dictionary mode and dictionary management poices for data compression, pp , Signa Processing,