Lecture 15. Error-free variable length schemes: Shannon-Fano code

Transcription

1 Lecture 15 Agenda for the lecture Bounds for L(X) Error-free variable length schemes: Shannon-Fano code 15.1 Optimal length nonsingular code While we do not know L(X), it is easy to specify a nonsingular which has the least average length possible. For convenience, we treat the empty sequence as a valid codeword. The optimal code is obtained by considering symbols in a nonincreasing order of probabilities and assigning an available codeword sequence of shortest length to each codeword. For instance, the highest probability symbol is assigned, the next symbol is assigned 0, the next 1, then 00, then 01, and so on. Note that this scheme must have average length smaller than any other scheme. Indeed, consider a code C which assigns codewords c 1,..., c m to symbols with probabilities p 1,..., p m, respectively, where p 1 p 2... p m. Let c 1,..., c n We provide a sequence of codes C 0, C 1..., C m such that C 0 = C, C n is equal to the optimal code described above, and the average length of C i is greater than that of C i+1 for every i. Specifically, for 1 i n, C i assigns the codeword c j to the symbol j for 1 j i. We c Himanshu Tyagi. Feel free to use with acknowledgement. 1

2 complete the construction iteratively. Suppose we have C i with desired property for an i. Then, for i + 1, if c i+1 / C i, replace (i + 1)th codeword of C i with c i+1. Else, assign the (i + 1)th codeword to the symbol which has been assigned c i+1 in C i and assign c i+1 to symbol i+1. Proceeding inductively, we can see that the length of the (i+1)th codeword in C i is at least as much as that of c i+1. Therefore, since the probabilities have been arranged in a nonincreasing order, the average length of C i is more than that of C i+1. Claim: Denoting by l(i) the length of the codeword assigned by the optimal code above to the symbol with probability p i, we have l(i) log p i, for all 1 i n. Proof. Suppose there exists i such that l(i) > log p i. Then, for all j i, p j p i > 2 l(i). Therefore, (by a packing argument) i 1 P j > i 2 l(i), j=1 i.e., i 2 l(i) 1. But there are 2 l(i) 1 binary sequences of length less than or equal to l(i) 1. Thus, all the symbols 1,..., i can be mapped to codewords of length less than l(i) 1 by the optimal code, which is contradiction since we assumed i was mapped to a codeword of length l(i). As a corollary of the simple observation above, we obtain the following upper bound for L(X). Lemma For every source X over a discrete alphabet X, L(X) H(X). 2

3 Proof. By the previous claim, L(X) m m p i l(i) p i log p i = H(X). i=1 i=1 Therefore, a nonsingular code without any additional structure can achieve a smaller average length, in general. However, the next result shows that the gain is rather negligible, only logarithmic in H(X) (no more than log log X ). Theorem For every source X over a discrete alphabet X, H(X) log eh(x) < L(X) H(X). Proof. It only remains to prove the lower bound. Our proof will also illustrate the power of chain rules in analyzing difficult counting problems. Specifically, consider a nonsingular code for source X. Denote by N the (random) length of the codeword used when X is stored N is random since X is random and by (Y 1,..., Y N ) {0, 1} N the random codeword used. Note that the average length of the code is equal to E [N]. Since the code is nonsingular, there is a one-to-one map between X and Y 1,..., Y N. Thus, H(X) = H(Y 1,..., Y N ) (why?) = H(N) + H(Y 1,..., Y N N). We bound each term on the right-side above separately. For H(N), recall from Section 12.2 (in lecture 12) that for every N-valued rv N, H(N) log ee [N]. 3

4 Note that the proof we gave can be easily extended to include rvs which also take the value 0; try it as an exercise. For the second term, since (Y 1,..., Y n ) cannot take more than 2 n values for each fixed realization N = n, H(Y 1,..., Y N N) = = P N (n) H(Y 1,..., Y N N = n) n=0 P N (n) H(Y 1,..., Y n N = n) n=0 P N (n) n n=0 = E [N]. Therefore, upon combining the bounds above, H(X) log(ee [N]) + E [N]. Thus, if E [N] H(X), then clearly the claimed lower bound for E [N] holds. On the other hand, if E [N] < H(X), then H(X) < log(eh(x)) + E [N], which yields the claimed lower bound for E [N]. In summary, we have established the following relationships between average lengths for various regimes in error-free codes: H(X) log(eh(x)) < L(X) H(X) L u (X) = L p (X) H(X)

5 Thus, the ease of implementation afforded by prefix-free codes comes at roughly no additional cost. For this very reason, we shall only focus on prefix-free coding schemes Error-free coding schemes We shall discuss four different error-free coding schemes: The Shannon-Fano code, the Shannon-Fano-Elias code, the Huffman code, and the Arithmetic code. All these codes are prefix-free. However, while Huffman code relies on the binary tree representation of prefixfree codes, the other three use the interval representation. Each scheme has its advantages and disadvantages, which we shall discuss in the next few sections The Shannon-Fano code The algorithm for code construction is as follows: Input: Source distribution P Output: Code C = {c 1,..., c m }. 1. Sort the probabilities of symbols in a descending order. Let p 1 p 2... p m be the sorted sequence of probabilities. 2. for i = 1,..., m (i) Let l(i) = log p i. (ii) Compute F i = j<i p j (where F 0 = 0). Let c denote the infinite sequence corresponding to the binary representation of F i. If F i has a terminating binary representation, append 0s at the end to make it an infinite sequence. (iii) The codeword c i is given by the first l(i) bits of c, i.e., by the approximation of c to l(i) bits. For illustration, consider the following example: Note that the code uses Shannon lengths log p i and, therefore, has average length less than H(X) + 1. It only remains to verify that the code is prefix-free. We make the following observation: 5

6 Alphabet P X Sorted rank F i in binary l(i) codeword a 1/ b 1/ c 1/ d 1/ Table 1: An illustration of Shannon-Fano code. For any two numbers x, y [0, 1] which have the same first l bits in binary representation, x y + 2 l, with equality if and only if the lth bit in both x and y is 0, and y has all (infinitely many) bits 1 starting from the (l + 1)th position. Claim: The Shannon-Fano code described above is a prefix-free code. Proof. Suppose for j > i the codeword c i is a prefix of the codeword c j. Note that l i < l j. Therefore the first l i bits of the binary representation for F i and F j are the same. By the observation above, F j < F i + 2 l i F i + p i F j, which is a contradiction. 6