392D: Coding for the AWGN Channel Wednesday, March 21, 2007 Stanford, Winter 2007 Handout #26. Final exam solutions

Transcription

1 92D: Coding for the AWGN Channel Wednesday, March 2, 27 Stanford, Winter 27 Handout #26 Problem F. (8 points) (Lexicodes) Final exam solutions In this problem, we will see that a simple greedy algorithm for choosing progressively longer binary block codes with a given minimum distance d produces some surprisingly good linear codes. A binary lexicographic code (or lexicode) C(n,d) of length n and minimum distance d is defined as follows [Conway and Sloane, 986]. First, the binary n-tuples {x (x,x,...,x n ) (F 2 ) n } are put into lexicographical order corresponding to the standard binary representation of the integers in the interval [, 2 n ); i.e., x > x if and only if x 2x 2 n x n > x 2x 2 n x n. Now C(n, d) is constructed by considering the binary n-tuples in lexicographical order, smallest to largest, starting with, and greedily including an n-tuple in C(n,d) if and only if it is at Hamming distance d or greater from all previously chosen n-tuples. For example, to construct the lexicode C(, 2) of length and minimum distance 2, we consider the binary -tuples in the following order and make the following choices: tuple integer include? choices so far yes no 2 no yes, 4 no, 5 yes,, 6 yes,,, 7 no,,, Thus C(, 2) is the (, 2, 2) single-parity-check code of length. (a) ( points) Construct the lexicode C(6, 4) of length 6 and minimum distance 4. Considering the 64 binary 6-tuples in lexicographical order, we first choose. The next 6-tuple with at least 4 ones is ; thus C(4, 4) is the (4,, 4) repetition code. We clearly can achieve a distance of at most 2 from and in the first four coordinates; therefore the next 6-tuple must be. Complementing the first four coordinates gives us the next and final word,. Thus C(6, 4) is the (6, 2, 4) binary linear block code generated by and. (b) ( points) What is the shortest nontrivial lexicode with minimum distance d? A code that contains only the all-zero word is trivial. The next word to be added in a distance-d lexicode must differ from the all-zero word in d places. Thus the shortest nontrivial lexicode with distance d is the code consisting of an all-zero d-tuple and an all-one d-tuple, namely the (d,,d) binary repetition code of length d.

2 (c) ( points) Assume that C(n,d) is a binary linear (n,k,d) block code. Show that C(n,d) is a binary linear (n,k,d) block code, where either k k or k k. Conclude that every lexicode C(n,d) is a binary linear block code. [Hint: let C (n,d) be the (n,k,d) block code obtained by extending all words in C(n,d) by a zero bit. Show that if there exists an (n )-tuple x (F 2 ) n at distance d from all words in C (n,d), then all words in the set C (n,d) x are at distance d from all words in C (n,d), and that there are no other such words.] Following the hint, it is obvious that if we consider the (n )-tuples in lexicographical order, then the words that we select for C(n,d) with x n will be the same as the words that we select for C(n,d), extended by a zero bit, x n. Therefore C (n,d) must be a subcode of C(n,d), and all words x C(n,d) that are not in C (n,d) must have x n. Since C(n,d) is linear (a group) by assumption, and addition of (n )-tuples is componentwise mod-2 addition, it follows that C (n,d) is also a linear code. Let us suppose that there exists a word x with x n that is at distance d from all words in C (n,d); if no such word exists, then C(n,d) C (n,d). Then the translate (coset) C (n,d) x must be a set of (n )-tuples of Hamming weight d or greater. Now let x x c be any other element of this translate, where c C (n,d); then C (n,d)x C (n,d)cx C (n,d)x, since C (n,d)c C (n,d), by the group property of C (n,d). Therefore the set of Hamming distances from x to all elements of C (n,d) is the same as that of x, so all such distances are greater than or equal to d. Finally, the difference between x and any other element of C (n,d) x is a nonzero codeword in C (n,d), and therefore the distance between x and any other element of C (n,d) x is also greater than or equal to d. Therefore all words in the translate C (n,d) x may be selected by the greedy lexicode construction algorithm. Finally, we should show that no other (n )-tuple x with x n can be selected. I confess that I don t know how to prove this; I was hoping that one of you would. Conway and Sloane use a general theorem from combinatorial game theory to prove that C(n, d) must be linear; given linearity, all words in C (n,d) x must be in C(n,d), and there can be no others with x n because only half the codewords c can have c n. Given that C(n,d) linear implies C(n,d) linear and that C(d,d) is linear, we may finally conclude that C(n,d) is a nontrivial linear code for all n d. (d) (2 points) Using the results of part (c), show that we can choose as generators for C(n,d) the generators of C(n,d), extended by a zero bit, plus (if dim C(n,d) > dim C(n,d)) the word in C(n,d) with x n that has the greatest number of low-order zeroes (i.e., the first index k for which x k is as great as possible). Show that such a set of generators for C(n,d) is trellis-oriented (minimum-span). Part (c) shows that we can add at most one generator each time we extend the code length. Therefore the stopping times of all generators are different. 2

3 Suppose that there is a generator g of C (n,d) that starts at the same time as the element of C (n,d)x that we have chosen as our new generator g. Then gg is an element of C (n,d)x that starts later than g, contradiction. Therefore no generator of C (n,d) starts at the same time as the new generator starts. By induction, the starting times of all generators are different. Since all starting and stopping times are different, this set of generators for C(n,d) is trellis-oriented (minimum-span). (e) ( points) Using this method, find a set of trellis-oriented generators for C(8, 4). From part (a), the first two generators (extended to length 8) are and. We can extend to length 7 with the generator x, which is at distance 4 from all words in C (7, 4). (Some of you guessed that would be the next generator, but is lexicographically least.) The latest starting time in the translate C (7, 4) x is attained by or, which may be taken as the next trellis-oriented generator. Finally, we can extend to length 8 with the (lexicographically least) generator x. The latest starting time in the translate C (8, 4) x is attained by, which may be taken as the final trellis-oriented generator. Thus C(8, 4) is generated by {,,, }; i.e., it is the (8,4,4) Reed-Muller code with the standard RM coordinate ordering. Problem F.2 (6 points) In this problem, we will analyze the performance of iterative decoding of a simple concatenated code C on a binary erasure channel (BEC) with erasure probability p, in the limit as the code becomes very long (n ). info bits (,, ) Π u(d) D y(d) Figure. Encoder for concatenated code C. The encoder for a repeat-accumulate (RA) code C is shown in Figure above; it works as follows. A sequence of information bits is first encoded by an encoder for a (,, ) repetition code, which simply repeats each information bit three times. The resulting sequence is then permuted by a large pseudo-random permutation Π. The permuted sequence u(d) is then encoded by a rate-/ 2-state convolutional encoder with input/output relation y(d) u(d)/(d); i.e., the input/output equation is y k u k y k, so the output bit is simply the accumulation of all previous input bits (mod 2). (a) What is the rate of this code C? The rate of the repetition code is. The rate of the convolutional code is ; i.e., it puts one bit out for every one bit in. Therefore the overall code rate is r. (b) Show that the code C is represented by the normal graph of Figure 2.

4 Π Figure 2. Normal graph of concatenated code C. The left-side nodes of Figure 2 represent the repetition code. Since the original information bits are not transmitted, they are regarded as hidden state variables, repeated three times. The repeated bits are permuted in the permutation Π. On the right side, the permuted bits u k are the input bits and the y k are the output bits of a 2-state trellis, whose states are the output bits y k. The trellis constraints are represented explicitly by zero-sum nodes that enforce the constraints y k u k y k. (c) Suppose that the encoded bits are sent over a BEC with erasure probability p. Explain how iterative decoding works in this case, using a schedule that alternates between the left constraints and the right constraints. The outputs of a BEC are either known with certainty or completely unknown (erased). The sum-product algorithm reduces to propagation of known variables through the code graph. If any variable incident on a repetition node becomes known, then all become known. On the other hand, for a zero-sum node, all but one incident variable must be known in order to determine the last incident variable; otherwise all unknown incident variables remain unknown. In detail, we see that initially an input bit u k becomes known if and only if the two adjoining received symbols, y k and y k, are unerased. After passage through Π, these known bits propagate through the repetition nodes to make all equal variables known. After passage through Π, the right-going known bits are propagated through the trellis, with additional input or output bits becoming known whenever two of the three bits in any set {y k,u k,y k } become known. Known input bits u k are then propagated back through Π, and so forth. 4

5 (d) Show that, as n, if the probability that a left-going iterative decoding message is erased is q r l, then the probability that a right-going messages is erased after a left-side update is given by q l r (q r l ) 2. As n, for any fixed number m of iterations, we may assume that all variables in the m-level computation tree are independent. A right-going message is erased if and only if both left-going messages that are incident on the same repetition node is erased. Thus if these two variables are independent, each with probability q r l of erasure, then q l r (q r l ) 2. (e) Similarly, show that if the probability that a right-going iterative decoding message is erased is q l r, then the probability that a left-going message is erased after a right-side update is given by ( p) 2 q r l ( p pq l r ) 2. [Hint: observe that as n, the right-side message probability distributions become invariant to a shift of one time unit.] For a particular input bit, the messages that contribute to the calculation of the outgoing message look like this: q in p q in p p q in p q in x y x y x x y x y x q out Here p denotes the probability that an output bit y k will be erased on the channel, q in denotes the probability q l r that an input bit u k will still be erased after the previous iteration, and q out denotes the probability q r l that the bit that we are interested in will be erased after this iteration. Again, as n, we may assume that all of these probabilities are independent. Following the hint, we use the symmetry and time-invariance of the trellises on either side of u k to assert that as n the probability of erasure x in all of the messages marked with x will be the same, and similarly that the probability of erasure y in all of the messages marked with y will be the same. The relations between these probabilities are then evidently as follows: x py; y ( q in )( x); q out ( x) 2. Solving the first two equations, we obtain y q in p pq in, x pq in p pq in, and thus q out ( p) 2 ( p pq in ) 2. 5

6 (f) Using a version of the area theorem that is appropriate for this scenario, show that iterative decoding cannot succeed if p 2. The area under the curve of part (d) is The area under the curve of part (e) is q 2 dq. ( p) 2 ( p)2 ( p pq) 2dq p [ p pq Iterative decoding will succeed if and only if the two curves do not cross. In order for the two curves not to cross, the sum of these two areas must be less than the area of the EXIT chart; i.e., p <, which is equivalent to p < 2 ; i.e., the capacity p of the BEC, namely p, must be greater than the rate of the RA code, namely. (For example, in Figure below, the area above the top curve is, whereas the area below the bottom curve is.) 2 (g) The two curves given in parts (d) and (e) are plotted in the EXIT chart below for p.5. Show that iterative decoding succeeds in this case. p.5 ] p. q l r.75 q r l Figure. EXIT chart for iterative decoding of C on a BEC with p.5. The two curves do not cross, so iterative decoding starting at (q r l,q l r ) (, ) must succeed (reach (q r l,q l r ) (, )). 6

7 (h) [Optional; extra credit.] Determine whether or not iterative decoding succeeds for p.6. The easiest way to determine whether iterative decoding succeeds for p.6 is to simulate it using the equations above. We obtain q l r q r l Thus iterative decoding succeeds fairly easily for p.6, even though the capacity of a BEC with p.6 is only.4, not much greater than the rate of the RA code. It is possible therefore that irregular RA codes may be capacity-approaching for the BEC. Even with regular codes, the two EXIT curves are already quite well matched. However, note that only the left degrees can be made irregular, which limits design flexibility. Scores on final exam: 56, 5, 47, 42, 42, 4,,, 7. 7