Interval Algorithm for Homophonic Coding

Transcription

1 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL 47, NO 3, MARCH Interval Algorithm for Homophonic Coding Mamoru Hoshi, Member, IEEE, and Te Sun Han, Fellow, IEEE Abstract It is shown that the idea of the successive refinement of interval partitions, which plays the key role in the interval algorithm for random number generation by Han and Hoshi, is also applicable to the homophonic coding An interval algorithm for homophonic coding is introduced which produces an independent and identically distributed (iid) sequence with probability Lower and upper bounds for the expected codeword length are given Based on this, an interval algorithm for fixed-to-variable homophonic coding is established The expected codeword length per source letter converges to ( ) ( ) in probability as the block length tends to infinity, ( ) is the entropy rate of the source The algorithm is asymptotically optimal An algorithm for fixed-to-fixed homophonic coding is also established The decoding error probability tends to zero as the block length tends to infinity Homophonic coding with cost is generally considered The expected cost of the codeword per source letter converges to ( ) ( ) in probability as the block length tends to infinity, denotes the average cost of a source letter The main contribution of this paper can be regarded as a novel application of Elias coding technique to homophonic coding Intrinsic relations among these algorithms, the interval algorithm for random number generation and the arithmetic code are also discussed Index Terms Arithmetic coding, cost coding, Elias coding, homophonic coding, interval algorithm, random number generation I INTRODUCTION HOMOPHONIC coding (or substitution) is a coding technique that transforms a stream of source letters with an arbitrary distribution into an invertible random stream of symbols (random codeword), which all have the same frequency That is to guarantee security in source coding as much as possible The key idea of homophonic coding for randomization is to introduce a suitable number of representations (codewords) for each letter from the source alphabet and to randomly choose one of the representations (codewords) at each step The codewords assigned to the source letter are called the homophones for Simple examples are helpful to explain what is the homophonic coding Example 1: Let be a source taking values in the source alphabet Manuscript received August 30, 1999; revised September 7, 2000 The material in this paper was presented in part at the IEEE Information Theory and Communication Workshop, Kruger National Park, South Africa, June 20 25, 1999 The authors are with the Graduate School of Information Systems, The University of Electro-Communications, Chofu, Tokyo , Japan ( hoshi@isuecacjp; han@isuecacjp) Communicated by M Weinberger, Associate Editor for Source Coding Publisher Item Identifier S (01) Case 1 (Günther [1, p 407]): Let have the probability and The following mapping defines a homophonic code: with probability with probability with probability (1) The letter is encoded at random into, with probability respectively The codewords 0 and 10 are called the homophones for the source letter The probability of the codeword 0 (10) being produced is 1/2 (1/4) It is evident that this code is prefix free and, hence, uniquely decodable The code is represented by the tree of Fig 1, which we call the homophonic coding tree As is obvious from the figure, a leaf (denoted by a box) uniquely specifies a codeword (ie, the path from the root to the leaf) and vice versa Therefore, we also call the leaves with label the homophones for the source letter The expected length of the codewords (ie, the expected path length of the tree) is, as the entropy of the source is Case 2: Let have the probability and The following mapping defines a homophonic code: with probability with probability The homophonic coding tree of this code is shown in Fig 2 In this case, the set of the homophones (ie, codewords; leaves) is countably finite set The expected length of the codeword is, as the entropy of the source is Let us now formulate the homophonic coding Let be a random variable taking values in the source alphabet with probability Let be a finite set called the code alphabet and a prefix-free subset of, is the set of all nonnull finite sequences of elements of Let be a random variable taking values in We consider the conditional distribution of for given and define a stochastic multivalued mapping such that for each, there exists exactly one such that and ; is assigned to with probability and called a homophone for We refer to this mapping as the (prefix-free) homophonic coding The homophones (ie, codewords) for are those for which It is obvious from the above condition (2) /01$ IEEE

2 1022 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL 47, NO 3, MARCH 2001 Fig 1 Homophonic coding tree for the source X which have the probability p =3=4 and p =1=4 Code alphabet is f0; 1g Fig 2 Homophonic coding tree for the source X which have the probability p =2=3 and p =1=3 Code alphabet is f0; 1g that the inverse mapping of is unique (uniquely decodable) Note that homophonic coding, essentially, is interpreted as a channel with input set and output set Let be any variable-length random variable, is a positive integer-valued random variable such that the value of uniquely determines whether or not, (ie, is a stopping time) The variable-length random sequence is said to be completely -random, if each component variable is statistically independent of the preceding ones and obeys the same probability distribution In a word, the completely -random sequence may be rephrased to be a variablelength independent and identically distributed (iid) sequence subject to The homophonic coding is said to be -perfect if the homophone is completely -random When is uniform, we say the homophone and the homophonic coding, simply, completely random, and perfect, respectively [2] The perfect homophonic coding enables us to build what Shannon called the strongly ideal cipher system [2], [3] Günther [1] showed an algorithm for iid sources, called the variable-length homophonic substitution that makes the resulting -ary homophone completely random However, the required memory grows exponentially Jendal et al [2] have shown that for a binary homophonic coding, which is perfect and minimizes the expected length of the codeword among perfect homophonic codings, the expected length of the codeword is less than bits, is the entropy of the source (See also da Rocha et al [4]) Ryabko [5] suggested a perfect homophonic coding algorithm The expected length of the codeword generated by his algorithm is upper bounded by The required memory size is, is the redundancy defined as the difference between the average codeword length and the source entropy Homophonic coding proposed so far is required to be: 1) perfect, ie, 1a) iid and 1b) equiprobable; 2) error-free; 3) low redundant; and 4) equal symbol cost We consider the homophonic coding in more general situations, ie, generalize the requirement 1b) or 2) or 4) In this paper, we shall show that the idea of the successive refinement of interval partition, which was used in the interval algorithm for random number generation (cf Han and Hoshi [6]), is directly applicable also to the -perfect homophonic coding In Section II, we propose an interval algorithm for -ary homophonic coding, which produces an iid sequence with probability distribution, ie, completely -random sequence The expected length of the codeword generated by the algorithm is bounded as, depends only on In particular, when with, the produced sequence is completely random and is upper bounded by bits In Section III, we consider block coding for general stationary (and ergodic) source processes In Section III-A, we give an interval algorithm for fixed-to-variable -perfect homophonic coding which has the complexity linear in block length and is asymptotically optimal Furthermore, we show that the expected length of the codeword per source letter converges to in probability as the block length tends to infinity, is the entropy rate of the stationary ergodic source process (Theorem 4) Section III-B gives an asymptotically optimal linear complexity interval algorithm with stationary ergodic source processes for fixed-to-fixed -perfect homophonic coding and shows that, if the coding rate is lager than, the decoding error probability approaches to zero as the block length tends to infinity Section IV discusses -perfect homophonic coding by interval algorithm with cost and shows that the expected cost of the codeword per source letter converges to as the block length tends to infinity, denotes the average cost of the code letter The minimization of over all possible s is shown to be, which is attained by choosing case, it is also shown that (cf Section IV) In this converges in probability to

3 HOSHI AND HAN: INTERVAL ALGORITHM FOR HOMOPHONIC CODING 1023 if the source is stationary ergodic This property is used for the proof of the key theorem (Theorem 4) In Section V-A we discuss the relationship between the interval algorithms for random number generation and for homophonic coding In Section V-B we discuss the intrinsic relationship between the arithmetic coding algorithm and the interval algorithms In the Appendix, the proof of the convergence is given Finally, in closing this introduction, we would like to emphasize the following points First of all, it should be emphasized that the key idea of the present paper, ie, the key idea underlying the interval algorithm for homophonic coding is the successive refinement of interval partitions on the unit interval This key idea bas been earlier discovered first by Elias (cf Jelinek [7]), the origin of which may date back even to the Shannon Fano coding (cf Shannon [8]) Fixed-to-variable source codes based on this successive refinement of interval partitions are sometimes called the Elias code In other words, the standpoint of Elias code is to look at source sequences as the corresponding subintervals on the unit interval This standpoint can be made much clearer if we consider source coding with cost (cf Savari and Gallager [9]) Since, however, in order to construct Elias code the interval partitions with infinite precision were required, it was not implementable from the practical point of view, although the idea itself of Elias code remained very intriguing Subsequently, this practical implementation problem has been solved by Rissanen [10], Pasco [11], etc, by introducing floating point (instead of fixed point) arithmetic computations of interval partitions with finite precision at the expense of as small increase of the coding rate as desired Such codes have been called the arithmetic code Since there are many ways of modifying Elias code to practically implementable arithmetic codes by finite precision computations, the term arithmetic code sometimes denotes a family of those codes constructed on the basis of finite precision computations for interval partitions In a word, arithmetic code may reasonably be regarded as a kind of modification of Elias code, but not vice versa On the other hand, the interval algorithm for homophonic coding (IAHC) to be presented in this paper is properly constructed on the basis of the successive refinement of interval partitions with infinite precision, which is exactly along the original idea of Elias code rather than that of arithmetic code with finite precision This is because the interval algorithm for homophonic coding is required to be exactly (not approximately) -perfect In this sense, it is not right at all to say that the interval algorithm for homophonic coding may be viewed as a modification or direct application of arithmetic code: the consequence of any direct application of arithmetic code is far from -perfect homophonic coding It should be noted also that an auxiliary uniform random number (cf Section II), the use of which implies an intrinsic partition of a symbol s coding space in the Elias coding procedure among its homophones, intervenes and plays a crucial role in the interval algorithm for homophonic coding but does not at all in arithmetic code Nevertheless, the deep conceptual connection of the interval algorithm for homophonic coding as well as the interval algorithm for random number generation (cf Han and Hoshi [6]) to arithmetic code should not be inappropriately downplayed The main contribution of this paper can be regarded as a novel application of Elias coding technique to homophonic coding, it replaces the usual Huffman coding technique It is rather interesting also to observe that there exists an indeed intrinsic common structure underlying these algorithms, Elias code, and arithmetic codes, although they are, of course, respectively different at the detailed technical levels II INTERVAL ALGORITHM FOR HOMOPHONIC CODING In the previous paper [6], we have established an algorithm for random number generation The algorithm is based on the successive refinement of interval partition and is called the interval algorithm In the present paper, we propose the interval algorithm for -perfect homophonic coding Suppose that we are given a source taking values in the source alphabet with probability, Hereafter, we call this the source distribution Let be the code alphabet First, let us review the basic notation of the interval algorithm Partition the unit interval into subintervals according to probability distribution ; that is Hereafter, we sometimes call this distribution the partition ratio It is obvious that are mutually disjoint and Any interval may be partitioned into subintervals according to as follows: With this notation, the subinterval of length recursive manner as (3) (4) (5) with any sequence ) is defined in the for the null string It is then obvious that the size of the interval is given by Similarly to the above, we partition the unit interval into subintervals according to probability distribution ; that is (6) (7)

4 1024 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL 47, NO 3, MARCH 2001 Fig 3 Homophone 121 for X =1obtained by the algorithm IAHC [The source distribution q =(5=12; 1=3; 1=4), code alphabet Z = f1; 2g and partition ratio p =(1=2; 1=2)] (8) The interval algorithm for homophonic coding (denoted by IAHC) for the source is now described as follows Interval Algorithm for Homophonic Coding (IAHC) 0) Read a source output X = k 2f1; ; Ng, compute the interval J(k) and choose a random number r uniformly distributed on J(k); 1) Set s := (null string), I(s) :=[0; 1); 2) If I(s) J(k) then output the string s as a codeword for k, and terminate the algorithm; 3) Partition I(s) into M subintervals I(sj)(j =1; ; M); find the a such that r 2 I(sa); set s := sa, and go to step 2); This algorithm is summarized as, Given a source output, then take a uniform random number on the interval ; and find the largest interval (ie, the shortest string ), which contains the and is contained in It is easy to check that the set of codewords generated in this manner forms a prefix-free code Example 2 (Günther [1, p 409]): Consider the case with,, Fig 3 illustrates the homophonic coding by the interval algorithm Suppose that the source outputs a symbol We compute the interval and get a uniform random number, say Since is not included in, is partitioned with ratio to have intervals and Fig 4 Homophonic coding tree by the algorithm IAHC for the source with distribution q =(5=12; 1=3; 1=4), code alphabet Z = f1; 2g and partition ratio p =(1=2; 1=2) Since contains, set However, is not included in, and so is further partitioned to have intervals and The interval contains, but is not included in and, therefore, set is further partitioned to have intervals and Since contains and is included in, the algorithm outputs as a codeword for the source symbol and terminates In this manner, depending on the value of the uniform random number, we obtain one of the codewords for ; one of the codewords for ; the only codeword 22 for The homophonic coding tree of this code is shown in Fig 4 Noting that the probability that the algorithm reaches at a node of level is given as, we can see that the code produced is perfect The expected length of the codeword is 25, as the entropy of the source is When we use, instead of,we obtain a -perfect homophonic code in the same way as the above: the codewords for ; for ; for The -perfect homophonic code with such a nonuniform is closely related to the problem of homophonic coding with cost (cf Section IV)

5 HOSHI AND HAN: INTERVAL ALGORITHM FOR HOMOPHONIC CODING 1025 Just as in the case of interval algorithm for random number generation, we can equivalently describe the above algorithm IAHC in terms of an -ary tree (possibly of infinite size) as follows: 1) every internal node has children branches connecting the node to the children are labeled as in the order from left to right; 2) each node is associated with an interval as follows: a) the interval is associated with the root; b) the interval is associated in the one-to-one manner with the node if the path from the root to the node spells out the sequence of branches ; thus, we can use the associated sequence to denote the node ; 3) a node is a leaf and labeled if and (the same label may be assigned to several leaves) We call this tree the homophonic coding tree corresponding to the interval algorithm IAHC The tree thus defined works in the following way Starting with the root, we traverse the tree according to the source output and the uniform random number on Wegodown from a node to the son node such that the interval contains When we reach a leaf with label, output as a codeword (homophone) for, and terminate the algorithm The leaves s (or equivalently, the sequences s) with label are the homophones for Thereafter, we use the node, the associated interval, and the sequence, interchangeably Let be the expected length of the codeword generated by the homophonic coding tree Note that is equal to the expected level of the leaves, the level of the root is zero by convention Since the algorithm traverses the homophonic coding tree according to the source output and the uniform random number, the resulting variable-length output sequence (ie, the codeword) of the algorithm is also a random sequence is the random variable to denote the length of the sequence The key property of this sequence is that it is actually a variable-length iid sequence, which is stated as follows Theorem 1: When the algorithm IAHC with partition ratio is applied to the output of the source with probability distribution, the resulting -ary sequence is a variable-length iid sequence subject to probability distribution with stopping time, that is, is completely -random Thus, the homophonic coding by IAHC is -perfect Proof: Suppose that the source output took place with probability Let all the codewords (ie, all the homophones) for be listed as Then, it is obvious from the way of interval partition in the algorithm IAHC that (disjoint union) Since and the random number is uniformly distributed on, we see that On the other hand, from (6) it follows that: we have put Hence which implies that is a variable-length iid sequence subject to probability distribution with stopping time Corollary 1: Let be uniform, then the homophonic coding by IAHC is perfect, ie, the resulting -ary sequence is completely random Now that the performance of the homophonic coding tree is identical with that of generating tree for random number generation, the following theorem on the homophonic coding is a simple restatement of the theorem that has been already established with the interval algorithm for random number generation (cf [6, Theorem 3, pp ]) Theorem 2: For any and any, the expected length of the codeword generated by the interval algorithm IAHC is bounded as the lower bound must be satisfied by any tree algorithm for homophonic coding and is the binary entropy and Corollary 2: For uniform and any, the expected length of the codeword generated by the interval algorithm IAHC is bounded as (9) (10) In particular, if, the bound (10) reduces to (11) the logarithm of the entropy is to the base 2 The bound (11) is the same as that given by Ryabko [5] for this particular case III INTERVAL ALGORITHM FOR BLOCK HOMOPHONIC CODING In this section, let us consider the situation in which we want to encode a source block of length, instead of a one-shot source Let be any source

6 1026 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL 47, NO 3, MARCH 2001 process with component variables source alphabet taking values in the, and put Then, we can apply the interval algorithm IAHC in Section II with in place of, is the probability distribution of In order to do this, we first need to successively partition the unit interval according to the source distribution Let for the null string For any sequence, the subinterval is defined in the recursive manner as we have put (12) (13) (14) (15) It is here easy to see that (16) for any sequence and also that for In exactly the same manner as for, we associate a string over the alphabet with the interval A Fixed-to-Variable Homophonic Coding An interval algorithm for fixed-to-variable homophonic coding is obtained by slightly modifying the interval algorithm IAHC; that is, simply by replacing all the variable in steps 0) and 2) of the IAHC by Interval Algorithm for Fixed-to-Variable Homophonic Coding (v-iahc) 0) Read a source sequence x 1 xn; compute the interval J(x 1 xn); and choose a random number r uniformly distributed on J(x 1 xn); 1) Set s := (null string), I(s) :=[0; 1); 2) If I(s) J(x 1 xn), then output the string s as a codeword for x 1 xn and terminate the algorithm; 3) Partition I(s) into M subintervals I(sj)(j =1; ; M); find the a such that r 2 I(sa); set s := sa; and go to step 2); This algorithm is called the interval algorithm for fixed-tovariable homophonic coding (denoted by v-iahc) It is easy to see that this algorithm has the complexity linear in block length Since can be written as, it is evident from Theorem 2 that the algorithm thus defined satisfies the following bound: denotes the length of the codeword for and denotes the expectation Therefore, we have (17) which results in the following theorem Theorem 3: Let us here consider the case the source is stationary Then the sequence obtained by applying the algorithm v-iahc to the output of the source is completely -random and, hence, the homophonic coding by v-iahc is -perfect Furthermore, the algorithm v-iahc asymptotically minimizes the expected length of the codeword per source letter in the sense that (18) (19) is the entropy rate of the source This means that the interval algorithm v-iahc is asymptotically optimal with respect to the average code length per source letter (cf [6]) In particular, when is uniform, ie,, (18) becomes to (20) Thus, the algorithm v-iahc with uniform is not only perfect but also asymptotically optimal in terms of the average codeword length per source letter We are now in a position to state the following key theorem telling that the random number is actually constant as Theorem 4: Let the source be stationary ergodic and let denote the length of the codeword for generated by v-iahc with partition ratio Then, the length per source letter converges in probability as in prob (21) The proof of this theorem is given later in the Appendix because, for this proof, we need Theorem 8 to be stated in the next section B Fixed-to-Fixed Homophonic Coding So far we have discussed algorithms with the random stopping time that terminate with probability one However, these algorithms may not terminate within a prescribed finite time with probability one To ensure the finite time termination, one way is to restrict the number of interval parti-

7 HOSHI AND HAN: INTERVAL ALGORITHM FOR HOMOPHONIC CODING 1027 Fig 6 Homophonic coding tree by the algorithm f-iahc for the source with distribution q =(5=12; 1=3; 1=4), code alphabet Z = f1; 2g and partition ratio p =(1=2; 1=2) Fig 5 Homophone 121 for X =1obtained by the algorithm f-iahc (The source distribution q =(5=12; 1=3; 1=4), code alphabet Z = f1; 2g and partition ratio p =(1=2; 1=2)) tions (ie, the codeword length) to a prescribed fixed integer, with a positive number called the rate, at the cost of possible decoding error This is tantamount to considering the fixed-to-fixed coding It is easy to modify the algorithm v-iahc for this purpose To do so, we simply replace the step 2) in v-iahc by If then go to step 3); otherwise, output the string as a codeword for ; and terminate the algorithm; We call such an algorithm the interval algorithm for fixed-to-fixed homophonic coding (denoted by f-iahc), which is described as Interval Algorithm for Fixed-to-Fixed Homophonic Coding (f-iahc) 0) Read a source sequence x 1 xn; compute the interval J(x 1 xn); and choose a random number r uniformly distributed on J(x 1 xn); 1) Set ` := 0, s := (null string), I(s) :=[0; 1); 2) If jsj <L max (n) then go to step 3); otherwise, output the string s for x 1 xn; and terminate the algorithm; 3) Partition I(s) into M subintervals I(sj)(j =1; ; M); find the a such that r 2 I(sa); set s := sa; and go to step 2); Like the algorithm v-iahc, the complexity of this algorithm f-iahc is linear in block length However, with the code generated by the algorithm f-iahc, the decoding error is inevitable (ie, not uniquely decodable) in general, in contrast to the case of the algorithms IAHC and v-iahc Example 3: Consider Example 2 again ( ) Fig 5 illustrates the coding by f-iahc with Fig 6 shows the corresponding homophonic coding tree (See also the tree for v-iahc in Fig 4) In this example, the codewords for are and 122; the codewords for are and ; the codewords for are and These codewords are uniquely decodable except for the codeword Let be the decoding error probability by f-iahc with Let be a codeword generated by f-iahc and the homophonic coding tree of v-iahc If a prefix ( ) of the codeword corresponds to a leaf of the tree, can be uniquely decoded; in this case, the prefix is sufficient for correct decoding On the other hand, if corresponds to an internal node of, then decoding error occurs Therefore, the decoding error probability is equal to the probability that the algorithm f-iahc terminates at a leaf at level larger than in the tree This observation together with Theorem 4 leads to the following theorem Theorem 5: Let the source be stationary ergodic Then, the homophonic coding by f-iahc with partition ratio is -perfect Furthermore, with a fixed rate, the decoding error probability of the homophonic coding by f-iahc satisfies if (22) if An immediate consequence of Theorem 5 is as follows Corollary 3: Let the source be stationary ergodic Then, the homophonic coding by f-iahc with partition ratio is perfect Furthermore, with a fixed, the decoding error probability by f-iahc satisfies if (23) if Since the homophonic coding by f-iahc with uniform is perfect and all the codewords are of the same length, we can use f-iahc itself also as what Shannon called the strongly ideal cipher [3] IV HOMOPHONIC CODING WITH COST Suppose that each code symbol is assigned the cost and let The problem of source coding with such a general additive unequal

8 1028 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL 47, NO 3, MARCH 2001 cost has first been studied by Krause [12] and subsequently by Csiszár [13], etc Among others, the paper by Savari and Gallager [9] is one of a few papers treating the problem of arithmetic coding with unequal cost As for some different but related properties with cost, see, eg, Csiszár [14] In this section, we first study the problem of homophonic coding with unequal cost The cost of a codeword is defined by The expected cost of the codeword for a one-shot source that is generated by a homophonic coding tree is defined by (24) is the partition ratio and (25) is the probability that the codeword is generated by the tree We want here to minimize the expected cost by varying the partition ratio This is carried out as follows The weight of a node of is defined as the sum of the probabilities of leaves reachable from the node Then, we have the following relations (cf Ahlswede and Wegener [15]): (26) (27) is the random variable to denote the codeword generated by the tree These relations immediately yields the following key formula (eg, cf Ahlswede and Wegener [15]): (28) On the other hand, owing to (26), (24) is rewritten as follows: (29) we have put Let us now consider the homophonic coding with cost when the source subject to distribution on is replaced by the source block subject to distribution on Denote by and the length of the codeword for and the cost of the codeword for, respectively, then, in this case, (29) is written as (30) In -perfect homophonic coding with cost, we want to make the average cost [but not the average length ] of codewords as small as possible under the requirement that the output code sequence is -completely random Now, (30) together with Theorem 3 immediately yields (cf in Hoshi and Han [16, Theorem 1]): Theorem 6: Let the source be stationary, then for any partition ratio and cost function, the expected cost of the codeword per source letter generated by the algorithm v-iahc satisfies (31) Here the ratio can be interpreted as the local information gain per unit cost It is straightforward to check that attains the maximum over all possible s when, is the root of the equation, so that we have the following theorem (cf [16, Theorem 2]) Theorem 7: Consider the same source as in Theorem 6 For any and, we have the inequality under the algorithm v-iahc as (32) the logarithm of the entropy is to the base and the equality holds when and only when Theorems 6 and 7 mean that the algorithm v-iahc is asymptotically optimal (from the viewpoint of cost) with a given cost when In this connection, we notice that the uniform partition ratio is the best one only when all the costs are equal Remark 1: The term appears in the problem of minimizing the average cost for the stationary source using an optimal prefix code with unequal symbol costs (see [17] etc) A variant of Theorem 6 is the following theorem on the convergence of in probability, which is used to prove Theorem 4 in Section III-A Theorem 8: Let the source be stationary ergodic and given Let denote the cost of the codeword for generated by the algorithm v-iahc with Obviously, this homophonic coding is -perfect Then, the cost of the codeword per source letter satisfies as in prob (33) Proof: Let fix and consider any codeword Then, the probability of is given (34) (35) Since is a codeword, must be a leaf Hence, the interval is included in some, which is uniquely determined by Therefore, in view of and, we have (36)

9 HOSHI AND HAN: INTERVAL ALGORITHM FOR HOMOPHONIC CODING 1029 Since is stationary ergodic, we have the almost sure convergence (cf Barron [18]) This together with (36) implies that from which it follows that for any as (37) as (38) (39) On the other hand, it is clear that, for an arbitrary small constant X Then, a simple calculation gives We observe that and, hence, by (39) we have By virtue of Theorem 7, we have Thus, letting X in (44) it follows from (45) that (40) (41) (42) (43) (44) (45) (46) Since and are arbitrary constants, letting we obtain (47) In the light of (42), this together with (39) completes the proof V CONCLUDING REMARKS A Relation to Random Number Generation So far, we have shown that the idea of the successive refinement of interval partitions, which plays the key role also in the interval algorithm for random number generation (Han an Hoshi [6]), is directly applicable also to the homophonic coding Thus, as may have already been understood, -perfect homophonic coding with source distribution is identical with generating a random number with target distribution by repeatedly and independently tossing an -coin with probability Therefore, the length (cost) of the generated codeword exactly coincides with the number (cost) of coin tosses to generate a random number To help the reader s understanding on the intrinsic relationship between the interval algorithm for homophonic coding and that for random number generation, let us review here from [6] what is the interval algorithm for random number generation: Interval Algorithm for Random Number Generation 1) Set s = (null string), I(s) =[0; 1); 2) If I(s) J(k) for some k 2X, then set X = k and terminate the algorithm; 3) Toss the M -coin with distribution p to have a value a 2f1; ; Mg, then set s = sa, and go to step 2) The difference between the algorithms for homophonic coding and random number generation lies only in the ways of using the same tree As was stated, in the homophonic coding, the input to the tree is an output from the source with distribution We traverse with partition ratio the tree according to the source output and the uniform random number on to have a codeword for On the contrary, in the random number generation, the input to the tree is an output sequence of independent -coin tosses subject to distribution We traverse the tree according to the sequence to reach a leaf and get a random number with distribution Actually, on the basis of this observation, we have thus far derived Theorems 2 (expected code length), Theorem 3 (expected code length per source letter), Theorem 6 (expected cost per source letter), and Theorem 7 (lower bound for the expected cost per source letter) in correspondence with the theorems on the performance of the interval algorithms for random number generation ([6, Theorem 3, eq (52)], [16, Theorems 1, 2]) Inversely, we can easily see that Theorems 4 and 8 (the convergence in probability of the codeword length and of the cost per source letter) are valid also to evaluate the performance of the iterative random number generation, and are here understood to denote the number of coin tosses and the cost required to generate a random sequence of length, respectively As an illustrative case, let us consider the counterpart of Theorem 5, ie, the situation we want to produce a random sequence of length by tossing -coin times (ie, fixed-to-fixed random number generation) In this case, we cannot always produce a random sequence of length subject exactly to the prescribed target distribution In the fixed-to-fixed algorithm for random number generation by -coin with probability, we output a special sequence, say,, when we stay still at an internal node of the generating tree in the final step of coin tosses This causes a difference between the target distribution and the generated distribution It is straightforward to check by using Theorem 5 that the variational distance (48)

10 1030 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL 47, NO 3, MARCH 2001 between the target distribution satisfies and the generated distribution if (49) if is any stationary ergodic source with It should be noted here that the lower equality of (49) remains to be valid even if we invoke whatever random number generation algorithms that are not necessarily the interval algorithm used here (cf Han [19, Lemma 22]) B Relation to Arithmetic Coding Now, we should point out that there exists the remarkable conceptual analogy of the interval algorithm with the arithmetic code The encoding process of the algorithm IAHC corresponds to the encoding process of the arithmetic coding algorithm (cf [20], [9]), as the process of generating a random number by the interval algorithm in [6] corresponds to the decoding process of the arithmetic coding algorithm In the case of homophonic coding, as was described in Section II, we encode each source symbol into the several codewords with code alphabet The codewords for are, depending on the uniform random number, defined as the sequences of the smallest length such that the interval is entirely contained with infinite precision in the interval On the contrary to homophonic coding, in the case of arithmetic coding, we assign the only one (not several) codeword to each source symbol The codeword for is defined as the sequence of the smallest length for which the interval is approximately contained with finite precision in the interval The idea of the successive refinement of interval partitions has been earlier discovered first by Elias (cf [7]), and then applied with due modifications by many people, for example, in the fields of arithmetic coding, random number generation, homophonic coding, and so on It is rather surprising that the intrinsic relation among these applications remained not quite clear until recently This would be mainly because arithmetic coding, random number generation, and homophonic coding have been separately studied APPENDIX PROOF OF THEOREM 4 Let the v-iahc with partition ratio be as given in Theorem 4 Note that here is given and in accordance with this we introduce the cost function such that, is specified as in Theorem 8, ie, Then, clearly have put to the base in terms of Section IV Note here that we In this appendix, the logarithm is Let be the set of all the codewords such that (50) is an arbitrary small constant Then, from Theorem 8, under the algorithm v-iahc with partition ratio we have (51) Step 1) Consider the algorithm f-iahc with such that and (52) (53) Since the codeword is an iid sequence subject to, is also an iid sequence Therefore, by virtue of Chebyshev s inequality, we have Lemma 1: For any (54) Let be the set of all the codewords such that (55) is the code alphabet Then, by (52) and (53), can be expressed as (56) Consider a codeword of such that and write (57) so that If, then from (56) we have (58) which contradicts the fact that, owing to (50) Thus, must hold ( denotes the complement of a set), which together with (51) and the property (due to Lemma 1) yield (59) denotes the length of the codeword for generated by the v-iahc Step 2) Consider the algorithm f-iahc with and such that (60) Similarly to the step 1), since the codeword is an iid sequence subject to, again by means of Chebyshev s inequality, we have the following Lemma 2: For any (61)

11 HOSHI AND HAN: INTERVAL ALGORITHM FOR HOMOPHONIC CODING 1031 Let be the set of codewords such that Then, by (53) and (60), can be expressed as (62) (63) Consider a codeword of such that Then, there exist and such that (64) and, hence, If, then from (63) we have (65) which contradicts the fact that, owing to (50) Thus must hold, which together with (51) and the property (due to Lemma 2) conclude that (66) In the light of (53) and, (59) together with (66) completes the proof ACKNOWLEDGMENT The authors would like to thank Dr J Abrahams for having brought their attention to the homophonic coding They would also like to thank Prof K Kobayashi and Prof H Morita for their discussions REFERENCES [1] C G Günther, A universal algorithm for homophonic coding, in Advances in Cryptology Eurocrypt 88 (Lecture Notes in Computer Science) Berlin, Germany: Springer-Verlag, 1988, pp [2] H N Jendal, Y J B Kuhn, and J L Massey, An information theoretic treatment of homophonic substitution, in Advances in Cryptology Eurocrypt 89 (Lecture Notes in Computer Science) Berlin, Germany: Springer-Verlag, 1990, pp [3] C E Shannon, Communication theory of secrecy systems, Bell System Tech J, vol 28, pp , 1949 [4] V C da Rocha and J L Massey, On the entropy bound for optimum homophonic substitution, in Proc IEEE Int Symp Information Theory, Ulm, Germany, Sept 1997, p 93 [5] B Ryabko and A Fionov, A fast and efficient homophonic coding algorithm, in Algorithms and Computation, T Asano et al, Eds Berlin, Germany: Springer-Verlag, 1996, pp [6] T S Han and M Hoshi, Interval algorithm for random number generation, IEEE Trans Inform Theory, vol 43, pp , Mar 1997 [7] F Jelinek, Probabilistic Information Theory New York: Mc- Graw-Hill, 1968 [8] C E Shannon, A mathematical theory of communication, Bell System Tech J, vol 27, pp , 1948 [9] S A Savari and R G Gallager, Arithmetic coding for finite-state noiseless channels, IEEE Trans Inform Theory, vol 40, pp , Jan 1994 [10] J Rissanen, Generalized Kraft inequality and arithmetic coding, IBM J Res Devel, vol 20, pp , 1976 [11] R C Pasco, Some coding algorithms for fast data compression, PhD dissertation, Dept Elect Eng, Stanford Univ, Stanford, CA, 1976 [12] R M Krause, Channels which transmit letters of unequal duration, Inform Contr, vol 5, pp 13 24, 1962 [13] I Csiszár, Simple proofs of some theorems on noiseless channels, Inform Contr, vol 14, pp , 1969 [14] I Csiszár, G Katona, and G Túsnady, Information sources with different cost scales and the principle of the conservation of entropy, Z Wahrscheinlichkeitstheorie verw Gebiete, vol 12, pp , 1969 [15] R Ahlswede and I Wegener, Search Problems New York: Wiley, 1987 [16] M Hoshi and T S Han, Random number generation with cost by an interval algorithm, in Proc IEEE Int Symp Information Theory, Ulm, Germany, Sept 1997, p 158 [17] I Csiszár and J Körner, Information Theory: Coding Theorems for Discrete Memoryless Systems New York: Academic, 1981 [18] A R Barron, The strong ergodic theorem for densities: Generalized Shannon McMillan Breiman theorem, Ann Probab, vol 13, no 4, pp , 1985 [19] T S Han, Information-Spectrum Methods in Information Theory (in Japanese) Tokyo, Japan: Baifu-kan, 1998 [20] J Rissanen and G G Langdon, Arithmetic coding, IBM J Res Devel, vol 23, pp , 1979 [21] M Hoshi and T S Han, Interval algorithms for homophonic coding, in Proc IEEE Information Theory Workshop, Kruger National Park, South Africa, June 1999, pp [22] B Ryabko and A Fionov, Decreasing redundancy of homophonic coding, in Proc IEEE Int Symp Information Theory, Ulm, Germany, Sept 1997, p 94