Yes, it is non singular. The two codewords are different.

Transcription

1 Yes, it is non singular. The two codewords are different. Yes, it is UD. Given a sequence of codewords, first isolate occurrences of 01 (i.e., find all the ones) and then parse the rest into 0's. Remark: This code is suffix-free. No codeword is a suffix of any other codeword. To see this, use backward decoding. For any received sequence/string, we work backward from the end, and look for the reversed codewords. Since the codewords satisfy the suffix-free condition, the reversed codewords satisfy the prefix-free condition, and therefore we can uniquely decode the reversed sequence/string. No, the code is not prefix-free. The first codeword (0) is a prefix of the second codeword (01). Reminder: prefix-free code is the same as prefix code. This is more accurate in terms of meaning. This is the name that has been traditionally used. Remark: Observe that we can specify whether a code is nonsingular, uniquely decodable, or prefix-free even when there is no information about the shared alphabet nor the pmf of the DMS symbols

2 Code tree: From the code tree, all the codewords are located at leaf nodes of the tree. So, yes,thecodeisprefix-free. Yes, any prefix-free code is uniquely decodable (UD). This is one of many reasons we consider prefix-free code odeideadie

3

4 Each entropy value can be calculated directly from the given row vectors, Here, we use a MATLAB script (entropy2.m) to do this task. See 2.44 in the lecture note

5 (a) The code {00,01,10,110} can be shortened to {00,01,10,11}. The prefix-free property is preserved; so the shortened code is still UD. Regardless of the probabilty associated with each source symbol (as long as the last symbol has non-zero probability), the shortened code will have smaller expected length. Because Huffman code is optimal, the original code can't be Huffman. (b) The code {01,10) can be shortened to {0,1}. The prefix-free property is preserved; so the shortened code is still UD. Regardless of the probabilty associated with each source symbol, the shortened code will have smaller expected length. Because Huffman code is optimal, the original code can't be Huffman. (c) The code {0,01} is not prefix-free. Any Huffman code must be prefix-free. Therefore, it cannot be a Huffman code. Alternatively, one can also use the same reasoning in part (a) and (b): The code {0,01) can be shortened to {0,1}. The new code is prefix-free and hence still UD. Regardless of the probabilty associated with each source symbol, the shortened code will have smaller expected length. Because Huffman code is optimal, the original code can't be Huffman.

6 Following the hint, we create a Huffman code from the suggested probability values This column is added for comparison Note that the answer for this question is not unique. You may check that This condition is here to guarantee that x = 2 and x = 3 are grouped frist.

7

8 Method 1 Method 2: Directly read the transition (conditional) probabilities from the channel diagram. To get the matrix P, we simply scale each row of matrix Q by the corresponding p(x) Method 1: Method 2:

9 In some of these parts, we will also try to derive the answer in a general form. To get the matrix P, we simply scale each row of matrix Q by the corresponding p(x). First, we find the Q matrix,

10 The convention for our class is that these numbers are ordered in the same way that they are specified in the support. We can get the P matrix by scaling each row of the Q matrix using the corresponding input probability p(x). Method 1: Method 2:

11 For naive decoder, look at each column of P and select (circle) the element whose corresponding x value is the same as y in that column. For DIY decoder, look at each column of P and select the element whose corresponding x value is the same as x(y) in the decoder table. Change all '2' to '1'. Change all '1' and '3' to '0'

12 We want to come up with some simple example. Therefore, we shall start by considering the Bernoulli RV X which has only two possible values: Because there are only two possible values, the pairing in the Huffman coding process must be between these two values: Therefore, Huffman coding (without extension) always needs 1 bit per symbol. You may recall that in Section 2.4, this expression is called the binary entropy function. The plot of this function is shown in the lecture notes; it is sketches below: Also note that we don't want to have p = 0 or p = 1 because they would make our RV X degenerated (deterministic). For degenerated RV, we don't have to waste any bit to convey its value. (The value is already pre-determined.) For example, if p = 1, we know that P[X = 1] = 1 and hence X 1 all the time. There is no uncertainty. Anyone can guess the value of X with 100% accuracy by simply guessing the value 1 every time. Alternatively, we may think of the situation here as sending the empty string ( ) to the receiver.

13 We first need to list all the probabilities for the (extended) source string (vector/block.) Idea 1: Ex when n = 4: We then put together these value in a pmf vector using for loop. Idea 2: Use MATLAB's "kron" command (a) (b)

14 From Q2c of HW2, From the P matrix, we select (circle) the maximum in each column and find the corresponding x value. Alternatively, we know that the MAP decoder must be optimal. Therefore, looking at the answers in Q2e of HW2, we see that the decoder that minimizes the error probability is (the first row) and the corresponding From Q2c of HW2, From the Q matrix, we select the maximum in each column. Alternatively, once we have determined the ML decoder, we see that, in this problem, it is exactly the same as the MAP decoder above and therefore the error probability must be the same. corresponding x value of the selected value in each column

15 From Q3b of HW2, From the P matrix, we select (circle) the maximum in each column and find the corresponding x value. Alternatively, we know that the MAP decoder must be optimal. Therefore, looking at the answers in Q3d of HW2, we see that the decoder that minimizes the error probability is (the third row) and the corresponding From Q3b of HW2, From the Q matrix, we select the maximum in each column. Alternatively, once we have determined the ML decoder, we see that and therefore, looking at the answers in Q3d of HW2, we see that (first row). corresponding x value of the selected value in each column From Q4a of HW2, For MAP detector, look at each column of P and select the element with max value. The decoder table can be read from the selected (circled) values simply by finding the corresponding x value. For ML decoder, look at each column of Q and select (circle) the element with max value From Q4a of HW2, For ML decoder, the elements selected in the P matrix are simply those corresponding to positions selected in the Q matrix. After the fact, we observe that the ML decoder here is the same as the naive decoder in Q4.c.vi of HW2. Therefore, it should have the same decoding error probability (of 0.5) as the naive decoder as well.

16 Codebook: Usually, for block coding, this should be a row vector because we work with k info-bit at a time treating the block as a row vector of length k. However, for repetition code, we have k = 1. Recall that the code rate of a repetition code is 1/n where n is the number of times that each info-bit is repeated. In class, we have shown that for repetition code over BSC with p < 0.5, the ML decoder is the same as a majority-voting decoder: Note that the real decoding table should have 32 rows to account for all possible 5-bit y. However, because only the number of 0s (or the number of 1s) is important, we can shorten the decoding table into the one given below: Error probability is found using the same technique discussed in ECS315 [See the lecture notes from last semester (Ch.6. p.65-67)] "Decoding Table":

17 Given an observed channel output, we can find the most likely codeword that was transmitted by using the MAP decoder. However, recall that, 1. When the codewords are equally-likely, ML decoder the same as the MAP decoder. 2) When the crossover probability of the BSC p is < 0.5, ML decoder is the same as the min distance decoder. Therefore, we can avoid complicated calculation and use minimum-distance decoder here: From the table, we see that the most likely transmitted codeword is The given expression for the joint pmf can be expressed using the joint pmf matrix as The sum of all elements in the P matrix should be 1:

18 The given expression for the joint pmf can be expressed using the joint pmf matrix as The sum of all elements in the P matrix should be 1: Recall that two random variables X and Y are indepepndent if and only if

19 These are the results of knowing that X and Y are independent. In class, we have shown that the MAPD for block coding over the BSC (with crossover probability p) is given by We have also whown that when the block coding is done simply by using repetition code, If this is set to be 0, Because we have exactly the same detector, the probability of decoding error should also be the same.

20 Additional Comments sum(binopdf(2:5,5,0.4)) 1-sum(binopdf(0:1,5,0.4)) sum(binopdf(0:1,5,1-0.4)) The "any" or "don't care" or "does not matter" in the table means that the MAP decoder can choose to guess 1 or 0 without changing the overall error-rate performance. However, some decoder may issue a "decoding error" warning and avoid making a guess. (We will not discuss this type of decoder.)

21 This is the same answer as majority voting.

22 When p = 0.4, the conditional probabilities are the same and the two cases are equally likely. Majority voting would always guess 0 because the #0s in is greater then the #1s. This would agree with our answer here when p > 0.4 [See additional remark at the end] In general, for transmission over BSC, we have

23 Additional Comments for Q8b,c It should be noted that parts (b.iii) and (c.iii) are almost the same as part (a). All of them consider MAP decoding. However, parts (b.iii) and (c.iii) consider only one case (one row): Alternative solution for Q From Q5.c

24

25 p Q q q P naive P naive y xmap y P MAP P MAP

26 y xml y P ML P ML

27 (ii) Note that this is a noisy channel with non-overlapping outputs. (Only one non-zero element in each column of Q.)

28

29 The given DMC belongs to a family of channels called Binary Erasure Channel (BEC). The general form of the channel diagram and the corresponding Q matrix are the case where the bit is "erased" One may say that the uniform pmf is "obvious" from the "symmetry" in the transition probabilities from the inputs.

30 Alternatively, we can solve this part by trying to apply properties of entropy (the same way that we solve the examples in class) and avoid using calculus.

31

32

33 close all; clear all; %% 1.a Q = [1/3 2/3; 2/3 1/3]; [ps C] = capacity_blahut(q) %% 1.b Q=[01/54/500;2/3001/30;00001]; [ps C] = capacity_blahut(q) %% 1.c Q = [1/3 2/3; 1/3 2/3]; [ps C] = capacity_blahut(q) %% 2.c Q=[1000;0100;0010]; [ps C] = capacity_blahut(q) %% 3.a Q=[1/32/30;02/31/3]; [ps C] = capacity_blahut(q) %% 3.b Q = [1 0; 1/2 1/2; 1/2 1/2]; [ps C] = capacity_blahut(q) %% 4 Q = [1 0; 0 1; 1/3 2/3]; [ps C] = capacity_blahut(q) >> Capacity_HW_blahut ps = C= ps = C= ps = C= 0 ps = C= ps = C= ps = C= ps = C=