Bounds on the Number of Iterations for Turbo-Like Ensembles over the Binary Erasure Channel

Transcription

1 ACCEPTED TO THE IEEE TRANSACTIONS ON INFORMATION THEORY, REVISED FEBRUARY 29. Bounds on the Numer of Iterations for Turo-Like Ensemles over the Binary Erasure Channel Igal Sason, Memer, IEEE and Gil Wiechman, Student Memer, IEEE Astract This aer rovides simle lower ounds on the numer of iterations which is required for successful messageassing decoding of some imortant families of grah-ased code ensemles including low-density arity-check codes and variations of reeat-accumulate codes). The transmission of the code ensemles is assumed to take lace over a inary erasure channel, and the ounds refer to the asymtotic case where we let the lock length tend to infinity. The simlicity of the ounds derived in this aer stems from the fact that they are easily evaluated and are exressed in terms of some asic arameters of the ensemle which include the fraction of degree-2 variale nodes, the target it erasure roaility and the ga etween the channel caacity and the design rate of the ensemle. This aer demonstrates that the numer of iterations which is required for successful message-assing decoding scales at least like the inverse of the ga in rate) to caacity, rovided that the fraction of degree-2 variale nodes of these turo-like ensemles does not vanish hence, the numer of iterations ecomes unounded as the ga to caacity vanishes). Index terms Accumulate-reeat-accumulate ARA) codes, area theorem, inary erasure channel BEC), density evolution DE), extrinsic information transfer EXIT) charts, iterative message-assing decoding, low-density aritycheck LDPC) codes, staility condition. I. INTRODUCTION During the last decade, there have een many develoments in the construction and analysis of low-comlexity error-correcting codes which closely aroach the Shannon caacity limit of many standard communication channels with feasile comlexity. These codes are understood to e codes defined on grahs, together with the associated iterative decoding algorithms. Grahs serve not only to descrie the codes themselves, ut more imortantly, they structure the oeration of their efficient su-otimal iterative decoding algorithms. Proer design of codes defined on grahs enales to asymtotically achieve the caacity of the inary erasure channel BEC) under iterative message-assing decoding. Caacity-achieving sequences of ensemles of low-density arity-check LDPC) codes were originally introduced y Shokrollahi [29] and y Luy et al. [3], and a systematic study of caacity-achieving sequences of LDPC ensemles was resented y Oswald and Shokrollahi [9] for the BEC. Analytical ounds on the maximal achievale rates of LDPC ensemles were derived y Barak et al. [6] for the asymtotic case where the lock length tends to infinity; this analysis rovides a lower ound on the ga etween the channel caacity and the achievale rates of LDPC ensemles under iterative decoding. The decoding comlexity of LDPC codes under iterative message-assing decoding scales linearly with the lock length, though their encoding comlexity may e suer-linear with the lock length. However, the class of reeat-accumulate codes and their more recent variants see, e.g., [], [] and [2]) exhiit the interleaver gain henomenon, and their encoding and decoding comlexities scale oth linearly with the lock length. Due to the simlicity of the density evolution analysis for the BEC, suitale constructions of caacity-achieving ensemles of variants of reeataccumulate codes were devised in [], [2], [2] and [26]. All these works rely on the density evolution analysis for the BEC, and rovide an asymtotic analysis which refers to the case where we let the lock length of these code ensemles tend to infinity. Rateless caacity-achieving codes for the BEC were introduced y Luy [], and later imroved y Shokrollahi [3]. The innovation of this aroach enales to achieve the caacity of the BEC without the knowledge of the channel arameter. This aer was sumitted to the IEEE Transactions on Information Theory in Novemer 27, and Revised in Feruary 29. The research work of was suorted y the Israel Science Foundation grant no. 7/7). The authors are with the Deartment of Electrical Engineering, Technion Israel Institute of Technology, Haifa 32, Israel {sason@ee, igillw@tx}.technion.ac.il).

2 2 ACCEPTED TO THE IEEE TRANSACTIONS ON INFORMATION THEORY, REVISED FEBRUARY 29. The erformance analysis of finite-length LDPC code ensemles whose transmission takes lace over the BEC was introduced y Di et al. [8]. This analysis considers su-otimal iterative message-assing decoding as well as otimal maximum-likelihood decoding. In [2], an efficient aroach to the design of LDPC codes of finite length was introduced y Amraoui et al.; this aroach is secialized for the BEC, and it enales to design such code ensemles which erform well under iterative decoding with a ractical constraint on the lock length. In [23], Richardson and Uranke initiated the analysis of the distriution of the numer of iterations needed for the decoding of LDPC ensemles of finite lock length which are communicated over the BEC. For general channels, the numer of iterations is an imortant factor in assessing the decoding comlexity of grah-ased codes under iterative message-assing decoding. The second factor determining the decoding comlexity of such codes is the comlexity of the Tanner grah which is used to reresent the code; this latter quantity, defined as the numer of edges in the grah er information it, serves as a measure for the decoding comlexity er iteration. The extrinsic information transfer EXIT) charts, ioneered y ten Brink [3], [32]), form a owerful tool for an efficient design of codes defined on grahs y tracing the convergence ehavior of their iterative decoders. EXIT charts rovide a good aroximative engineering tool for tracing the convergence ehavior of soft-inut soft-outut iterative decoders; they suggest a simlified visualization of the convergence of these decoding algorithms, ased on a single arameter which reresents the exchange of extrinsic information etween the constituent decoders. For the BEC, the EXIT charts coincide with the density evolution analysis see [22]) which is simlified in this case to a one-dimensional analysis. A numerical aroach for the joint otimization of the design rate and decoding comlexity of LDPC ensemles was rovided in []; it is assumed there that the transmission of these code ensemles takes lace over a memoryless inary-inut outut-symmetric MBIOS) channel, and the analysis refers to the asymtotic case where we let the lock length tend to infinity. For the simlification of the numerical otimization, a suitale aroximation of the numer of iterations was used in [] to formulate this joint otimization as a convex otimization rolem. Due to the efficient tools which currently exist for a numerical solution of convex otimization rolems, this aroach suggests an engineering tool for the design of good LDPC ensemles which ossess an attractive tradeoff etween the decoding comlexity and the asymtotic ga to caacity where the lock length of these code ensemles is large enough). This numerical aroach however is not amenale for drawing rigorous theoretical conclusions on the tradeoff etween the numer of iterations and the erformance of the code ensemles. A different numerical aroach for aroximating the numer of iterations for LDPC ensemles oerating over the BEC is addressed in [5]. A different aroach for characterizing the comlexity of iterative decoders was suggested y Khandekar and McEliece see [], [2], [6]). Their questions and conjectures were related to the tradeoff etween the asymtotic achievale rates and the comlexity under iterative message-assing decoding; they initiated a study of the encoding and decoding comlexity of grah-ased codes in terms of the achievale ga in rate) to caacity. It was conjectured there that for a large class of channels, if the design rate of a suitaly designed ensemle forms a fraction ε of the channel caacity, then the decoding comlexity scales like ε ln ε. The logarithmic term in this exression was attriuted to the grahical comlexity i.e., the decoding comlexity er iteration), and the numer of iterations was conjectured to scale like ε. There is one excetion: For the BEC, the comlexity under the iterative messageassing decoding algorithm ehaves like ln ε see [3], [25], [26] and [29]). This is true since the asolute reliaility rovided y the BEC allows every edge in the grah to e used only once during the iterative decoding. Hence, for the BEC, the numer of iterations erformed y the decoder serves mainly to measure the delay in the decoding rocess, while the decoding comlexity is closely related to the comlexity of the Tanner grah which is chosen to reresent the code. The grahical comlexity required for LDPC and systematic irregular reeat-accumulate IRA) code ensemles to achieve a fraction ε of the caacity of a BEC under iterative decoding was studied in [25] and [26]. It was shown in these aers that the grahical comlexity of these ensemles must scale at least like ln ε ; moreover, some exlicit constructions were shown to aroach the channel caacity with such a scaling of the grahical comlexity. An additional degree of freedom which is otained y introducing state nodes in the grah e.g., unctured its) was exloited in [2] and [2] to construct caacity-achieving ensemles of grah-ased codes which achieve an imroved tradeoff etween comlexity and achievale rates. Surrisingly, these caacity-achieving ensemles under iterative decoding were demonstrated to maintain a ounded grahical comlexity regardless of the erasure roaility of the BEC. A similar result of a ounded grahical comlexity for caacity-achieving

3 SASON AND WIECHMAN: BOUNDS ON THE NUMBER OF ITERATIONS FOR TURBO-LIKE ENSEMBLES OVER THE BINARY ERASURE CHANNEL3 ensemles over the BEC was also otained in [9]. This aer rovides simle lower ounds on the numer of iterations which is required for successful messageassing decoding of grah-ased code ensemles. The transmission of these ensemles is assumed to take lace over the BEC, and the ounds refer to the asymtotic case where the lock length tends to infinity. The simlicity of the ounds derived in this aer stems from the fact that they are easily evaluated and are exressed in terms of some asic arameters of the considered ensemle; these include the fraction of degree-2 variale nodes, the target it erasure roaility and the ga etween the channel caacity and the design rate of the ensemle. The ounds derived in this aer demonstrate that the numer of iterations which is required for successful message-assing decoding scales at least like the inverse of the ga in rate) to caacity, rovided that the fraction of degree-2 variale nodes of these turo-like ensemles does not vanish hence, the numer of iterations ecomes unounded as the ga to caacity vanishes). The ehavior of these lower ounds matches well with the exerimental results and the conjectures on the numer of iterations and comlexity, as rovided y Khandekar and McEliece see [], [2] and [6]). Note that lower ounds on the numer of iterations in terms of the target it erasure roaility can e alternatively viewed as lower ounds on the achievale it erasure roaility as a function of the numer of iterations erformed y the decoder. As a result of this, the simle ounds derived in this aer rovide some insight on the design of stoing criteria for iteratively decoded ensemles over the BEC for other stoing criteria see, e.g., [3], [27]). This aer is structured as follows: Section II resents some reliminary ackground, definitions and notation, Section III introduces the main results of this aer and discusses some of their imlications, the roofs of these statements and some further discussions are rovided in Section IV. Finally, Section V summarizes this aer. Proofs of some technical statements are relegated to the aendices. II. PRELIMINARIES This section rovides reliminary ackground and introduces notation for the rest of this aer. A. Grahical Comlexity of Codes Defined on Grahs As noted in Section I, the decoding comlexity of a grah-ased code under iterative message-assing decoding is closely related to its grahical comlexity, which we now define formally. Definition 2. Grahical Comlexity): Let C e a inary linear lock code of length n and rate R, and let G e an aritrary reresentation of C y a Tanner grah. Denote the numer of edges in G y E. The grahical comlexity of G is defined as the numer of edges in G er information it of the code C, i.e., G) E nr. Note that the grahical comlexity deends on the secific Tanner grah which is used to reresent the code. An analysis of the grahical comlexity for some families of grah-ased codes is rovided in [9], [2], [2], [25], [26]. B. Accumulate-Reeat-Accumulate Codes Accumulate-reeat-accumulate ARA) codes form an attractive coding scheme of turo-like codes due to the simlicity of their encoding and decoding where oth scale linearly with the lock length), and due to their remarkale erformance under iterative decoding []. By some suitale constructions of uncturing atterns, ARA codes with small maximal node degree are resented in []; these codes erform very well even for short to moderate lock lengths, and they suggest flexiility in the design of efficient rate-comatile codes oerating on the same ARA decoder. Ensemles of irregular and systematic ARA codes, which asymtotically achieve the caacity of the BEC with ounded grahical comlexity, are resented in [2]. This ounded comlexity result stays in contrast to LDPC ensemles, which have een shown to require unounded grahical comlexity in order to aroach channel caacity, even under maximum-likelihood decoding see [25]). In this section, we resent ensemles of irregular and systematic ARA codes, and give a short overview of their encoding and decoding algorithms; this overview is required for the later discussion. The material contained in this section is taken from [2, Section II], and is introduced here riefly in order to make the aer self-contained. From an encoding oint of view, ARA codes are viewed as interleaved and serially concatenated codes. The encoding of ARA codes is done as follows: first, the information its are accumulated i.e., differentially encoded),

4 ACCEPTED TO THE IEEE TRANSACTIONS ON INFORMATION THEORY, REVISED FEBRUARY 29. and then the its are reeated a varying numer of times y an irregular reetition code) and interleaved. The interleaved its are artitioned into disjoint sets whose size is not fixed in general), and the arity of each set of its is comuted i.e., the its are assed through an irregular single arity-check SPC) code). Finally, the its are accumulated a second time. A codeword of systematic ARA codes is comosed of the information its and the arity its at the outut of the second accumulator. Since the iterative decoding algorithm of ARA codes is erformed on the aroriate Tanner grah see Fig. ), this leads one to view them as sarse-grah codes from a decoding oint of view. Following the notation in [2], we refer to the three layers of it nodes in the Tanner grahs as systematic its which form the systematic art of the codeword, unctured its which corresond to the outut of the first accumulator and are not a art of the transmitted codeword, and code its which corresond to the outut of the second accumulator and form the arity-its of the codeword see Fig. ). Denoting the lock length of the code y n and its dimension y k, each codeword is comosed of k systematic its and n k code its. The two layers of check nodes are referred to as arity-check nodes and arity-check 2 nodes, which corresond to the first and the second accumulators of the encoder, resectively. An ensemle of irregular ARA codes is defined y the lock length n and the degree distriutions of the unctured it and arity-check 2 nodes. Following the notation in [2], the degree distriution of the unctured it nodes is given y the ower series Lx) L i x i ) where L i designates the fraction of unctured it nodes whose degree is i. Similarly, the degree distriution of the arity-check 2 nodes is given y Rx) R i x i 2) where R i designates the fraction of these nodes whose degree is i. In oth cases, degree of a node only refers to edges connecting the unctured it and the arity-check 2 layers, without the extra two edges which are connected to each of the unctured it nodes and arity-check 2 nodes from the accumulators see Fig. ). Considering the distriutions from the edge ersective, we let λx) λ i x i, ρx) ρ i x i 3) i= designate the degree distriutions from the edge ersective; here, λ i ρ i ) designates the fraction of edges connecting unctured it nodes to arity-check 2 nodes which are adjacent to unctured it arity-check 2 ) nodes of degree i. The design rate of a systematic ARA ensemle is given y R = where a L i a R i i= i= il i = L ) = ir i = R ) = i= λt)dt ρt)dt ar a L+a R designate the average degrees of the unctured it and arity-check 2 nodes, resectively. Iterative decoding of ARA codes is erformed y assing messages on the edges of the Tanner grah in a layery-layer aroach. Each decoding iteration starts with messages for the systematic it nodes to the arity-check nodes, the latter nodes then use this information to calculate new messages to the unctured it nodes and so the information asses through layers down the grah and ack u until the iteration ends with messages from the unctured it nodes to the arity-check nodes. The final hase of messages from the arity-check nodes to the systematic it nodes is omitted since the latter nodes are of degree one and so the outgoing message is not changed y incoming information. Assume that the code is transmitted over a BEC with erasure roaility. Since the systematic its receive inut from the channel, the roaility of erasure in messages from the systematic it nodes to the arity-check nodes is equal to throughout the decoding rocess. For other messages, we denote y x l) i where i =,,...,5 the roaility of erasure of the different message tyes at decoding iteration )

5 SASON AND WIECHMAN: BOUNDS ON THE NUMBER OF ITERATIONS FOR TURBO-LIKE ENSEMBLES OVER THE BINARY ERASURE CHANNEL5 DE x x 5 systematic its arity checks unctured its x x random ermutation x 2 x 3 arity checks 2 code its Fig.. Tanner grah of an irregular and systematic accumulate-reeat-accumulate code. This figure is reroduced from [2]. numer l where we start counting at zero). The variale x l) corresonds to the roaility of erasure in message from the arity-check nodes to the unctured it nodes, x l) tracks the erasure roaility of messages from the unctured it nodes to the arity-check 2 nodes and so on. The density evolution DE) equations for the decoder ased on the Tanner grah in Figure are given in [2], and we reeat them here: x l) = x l ) ) 5 ) x l) = x l) ) 2 λ x l ) ) x l) 2 = R x l) ) x l ) 3 ) l =,2,... x l) 3 = x l) 2 x l) = x l) ) 2 3 ρ x l) ) x l) 5 = x l) L x l) ). 5) The staility condition for systematic ARA ensemles is derived in [2, Section II.D] and states that the fixed oint x l) i = of the iterative decoding algorithm is stale if and only if ) 2 λ 2 ρ ) + 2R ). 6) C. Big-O notation The terms O, Ω and Θ are widely used in comuter science to descrie asymtotic relationshis etween functions for formal definitions see e.g., [3]). In our context, we refer to the ga in rate) to caacity, denoted y ε, and discuss in articular the case where ε i.e., sequences of caacity-aroaching ensemles). Accordingly, we define fε) = O gε) ) means that there are ositive constants c and δ, such that fε) c gε) for all ε δ. fε) = Ω gε) ) means that there are ositive constants c and δ, such that c gε) fε) for all ε δ. fε) = Θ gε) ) means that there are ositive constants c, c 2 and δ, such that c gε) fε) c 2 gε) for all ε δ. Note that for all the aove definitions, the values of c, c, c 2 and δ must e fixed for the function f and should not deend on ε.

6 6 ACCEPTED TO THE IEEE TRANSACTIONS ON INFORMATION THEORY, REVISED FEBRUARY 29. III. MAIN RESULTS In this section, we resent lower ounds on the required numer of iterations used y a message-assing decoder for code ensemles defined on grahs. The communication is assumed to take lace over a BEC, and we consider the asymtotic case where the lock length of these code ensemles tends to infinity. Definition 3.: Let { C m }m N e a sequence of code ensemles. Assume a common lock length n m) of the codes in C m which tends to infinity as m grows. Let the transmission of this sequence take lace over a BEC with caacity C. The sequence { } C m is said to achieve a fraction ε of the channel caacity under some given decoding algorithm if the asymtotic rate of the codes in C m satisfies R ε)c and the achievale it erasure roaility under the considered algorithm vanishes as m ecomes large. In the continuation, we consider a standard iterative message-assing decoder for the BEC, and address the numer of iterations which is required in terms of the achievale fraction of the channel caacity under this decoding algorithm. Theorem 3.: [Lower ound on the numer of iterations for LDPC ensemles transmitted over the BEC]. Let { n m,λ,ρ) } e a sequence of LDPC ensemles whose transmission takes lace over a BEC with erasure m N roaility. Assume that this sequence achieves a fraction ε of the channel caacity under message-assing decoding. Let L 2 = L 2 ε) e the fraction of variale nodes of degree 2 for this sequence. In the asymtotic case where the lock length tends to infinity, let l = lε,,p ) denote the numer of iterations which is required to achieve an average it erasure roaility P over the ensemle. Under the mild condition that P < L 2 ε), the required numer of iterations satisfies the lower ound lε,,p ) 2 L 2 ε) P ) 2 ε. 7) Corollary 3.: Under the assumtions of Theorem 3., if the fraction of degree-2 variale nodes stays strictly ositive as the ga in rate) to caacity vanishes, i.e., if lim L 2ε) > ε then the numer of iterations which is required in order to achieve an average it erasure roaility P < L 2 ε) under iterative message-assing decoding scales at least like the inverse of this ga to caacity, i.e., lε,,p ) = Ω. ε) Discussion 3.: [Effect of messages scheduling on the numer of iterations] The lower ound on the numer of iterations as rovided in Theorem 3. refers to the flooding schedule where in each iteration, all the variale nodes and susequently all the arity-check nodes send messages to their neighors. Though it is the commonly used scheduling used y iterative message-assing decoding algorithms, an alternative scheduling of the messages may rovide a faster convergence rate for the iterative decoder. As an examle, [28] considers the convergence rate of a serial scheduling where instead of sending all the messages from the variale nodes to arity-check nodes and then all the messages from check nodes to variale nodes, as done in the flooding schedule, these two hases are interleaved. Based on the density evolution analysis which alies to the asymtotic case of an infinite lock length, it is demonstrated in [28] that under some assumtions, the required numer of iterations for LDPC decoding over the BEC with serial scheduling is reduced y a factor of two as comared to the flooding scheduling). It is noted that the main result of Theorem 3. is the introduction of a rigorous and simle lower ound on the numer of iterations for LDPC ensemles which scales like the recirocal of the ga etween the channel caacity and the design rate of the ensemle. Though such a scaling of this ound is roved for the commonly used aroach of flooding scheduling, it is likely to hold also for other efficient aroaches of scheduling. It is also noted that this asymtotic scaling of the lower ound on the numer of iterations suorts the conjecture of Khandekar and McEliece []. Discussion 3.2: [On the deendence of the ounds on the fraction of degree-2 variale nodes] The lower ound on the numer of iterations in Theorem 3. ecomes trivial when the fraction of variale nodes of degree 2 vanishes. Let us focus our attention on sequences of ensemles which aroach the channel caacity under iterative

7 SASON AND WIECHMAN: BOUNDS ON THE NUMBER OF ITERATIONS FOR TURBO-LIKE ENSEMBLES OVER THE BINARY ERASURE CHANNEL7 message-assing decoding i.e., ε ). For the BEC, several such sequences have een constructed see e.g. [3], [29]). Asymtotically, as the ga to caacity vanishes, all of these sequences known to date satisfy the staility condition with equality; this roerty is known as the flatness condition [29]. In [2, Lemma 7], the asymtotic fraction of degree 2 variale nodes for caacity-aroaching sequences of LDPC ensemles over the BEC is calculated. This lemma states that for such sequences which satisfy the following two conditions as the ga to caacity vanishes: The staility condition is satisfied with equality i.e., the flatness condition holds) The limit of the ratio etween the standard deviation and the exectation of the right degree exists and is finite the asymtotic fraction of degree 2 variale nodes does not vanish. In fact, for various sequences of caacity aroaching LDPC ensemles known to date see [3], [9], [29]), the ratio etween the standard deviation and the exectation of the right degree-distriution tends to zero; in this case, [2, Lemma 7] imlies that the fraction of degree-2 variale nodes tends to 2 irresectively of the erasure roaility of the BEC, as can e verified directly for these code ensemles. Discussion 3.3: [Concentration of the lower ound] Theorem 3. alies to the required numer of iterations for achieving an average it erasure roaility P where this average is taken over the LDPC ensemle whose lock length tends to infinity. Although we consider an exectation over the LDPC ensemle, note that l is deterministic as it is the smallest integer for which the average it erasure roaility does not exceed a fixed value. As shown in the roof see Section IV), the derivation of this lower ound relies on the density evolution technique which addresses the average erformance of the ensemle. Based on concentration inequalities, it is roved that the erformance of individual codes from the ensemle concentrates around the average erformance over the ensemle as we let the lock length tend to infinity [22, Aendix C]. In light of this concentration result and the use of density evolution in Section IV which alies to the case of an infinite lock length), it follows that the lower ound on the numer of iterations in Theorem 3. is valid with roaility for individual codes from the ensemle. This also holds for the ensemles of codes defined on grahs considered in Theorems 3.2 and 3.3. Discussion 3.: [On the numer of required iterations for showing a mild imrovement in the erasure roaility during the iterative rocess] Note that for caacity-aroaching LDPC ensemles, the lower ound on the numer of iterations tells us that even for successfully starting the iteration rocess and reducing the it erasure roaility y a factor which is elow the fraction of degree-2 variale nodes, the required numer of iterations already scales like ε. This is also the ehavior of the lower ound on the numer of iterations even when the it erasure roaility should e made aritrarily small; this lower ound therefore indicates that for caacity-aroaching LDPC ensemles, a significant numer of the iterations is erformed for the starting rocess of the iterative decoding where the it erasure roaility is merely reduced y a factor of 2 as comared to the erasure roaility of the channel see Discussion 3.2 as a justification for the one-half factor). This conclusion is also well interreted y the area theorem and the asymtotic ehavior of the two EXIT curves for the variale nodes and the arity-check nodes) in the limit where ε ; as the ga to caacity vanishes, oth curves tend to e a ste function juming from to at the origin, so the iterations rogress very slowly at the initial stages of the decoding rocess. In the asymtotic case where we let the lock length tend to infinity and the transmission takes lace over the BEC, suitale constructions of caacity-achieving systematic ARA ensemles enale a fundamentally imroved tradeoff etween their grahical comlexity and their achievale ga in rate) to caacity under iterative decoding see [2]). The grahical comlexity of these systematic ARA ensemles remains ounded and quite small) as the ga to caacity for these ensemles vanishes under iterative decoding; this stays in contrast to un-unctured LDPC code ensemles [25] and systematic irregular reeat-accumulate IRA) ensemles [26] whose grahical comlexity necessarily ecomes unounded as the ga to caacity vanishes see [2, Tale I]). This oservation raises the question whether the numer of iterations which is required to achieve a desired it erasure roaility under iterative decoding, can e reduced y using systematic ARA ensemles. The following theorem rovides a lower ound on the numer of iterations required to achieve a desired it erasure roaility under iterative messageassing decoding; it shows that similarly to the arallel result for LDPC ensemles see Theorem 3.), the required numer of iterations for systematic ARA codes scales at least like the inverse of the ga to caacity. Theorem 3.2: [Lower ound on the numer of iterations for systematic ARA ensemles transmitted over the BEC]. Let { n m,λ,ρ) } e a sequence of systematic ARA ensemles whose transmission takes lace over m N a BEC with erasure roaility. Assume that this sequence achieves a fraction ε of the channel caacity under

8 8 ACCEPTED TO THE IEEE TRANSACTIONS ON INFORMATION THEORY, REVISED FEBRUARY 29. message-assing decoding. Let L 2 = L 2 ε) e the fraction of unctured it nodes of degree 2 for this sequence where the two edges related to the accumulator are not taken into account). In the asymtotic case where the lock length tends to infinity, let l = lε,,p ) designate the required numer of iterations to achieve an average it erasure roaility P of the systematic its. Under the mild condition that P < L 2ε), the numer of iterations satisfies the lower ound lε,,p ) 2 ε) L 2 ε) P 2 ε. 8) As noted in Section II-B, systematic ARA codes can e viewed as serially concatenated codes where the systematic its are associated with the outer code. These codes can e therefore decoded iteratively y using a turo-like decoder for interleaved and serially concatenated codes. The following roosition states that the lower ound on the numer of iterations in Theorem 3.2 is also valid for such an iterative decoder. Proosition 3.: [Lower ound on the numer of iterations for systematic ARA codes under turo-like decoding]. Under the assumtions and notation of Theorem 3.2, the lower ound on the numer of iterations in 8) is valid also when the decoding is erformed y a turo-like decoder for uniformly interleaved and serially concatenated codes. The reader is referred to Aendix I for a detailed roof. The following theorem which refers to irregular reeataccumulate IRA) ensemles is roved in a concetually similar way to the roof of Theorem 3.2. Theorem 3.3: [Lower ound on the numer of iterations for IRA ensemles transmitted over the BEC]. Let { n m,λ,ρ) } e a sequence of systematic or non-systematic) IRA ensemles whose transmission takes m N lace over a BEC with erasure roaility. Assume that this sequence achieves a fraction ε of the channel caacity under message-assing decoding. Let L 2 = L 2 ε) e the fraction of information it nodes of degree 2 for this sequence. In the asymtotic case where the lock length tends to infinity, let l = lε,,p ) designate the required numer of iterations to achieve an average it erasure roaility P of the information its. For systematic codes, if P < L 2 ε), then the numer of iterations satisfies the lower ound For non-systematic codes, if P < L 2 ε), then lε,,p ) 2 ε) L 2 ε) ) 2 P ε. 9) lε,,p ) 2 ε) L 2 ε) ) 2 P ε. ) A. Proof of Theorem 3. IV. DERIVATION OF THE BOUNDS ON THE NUMBER OF ITERATIONS Let { x l)} designate the exected fraction of erasures in messages from the variale nodes to the check nodes l N at the l th iteration of the message-assing decoding algorithm where we start counting at l = ). From density evolution, in the asymtotic case where the lock length tends to infinity, x l) is given y the recursive equation x l+) = λ ρ x l))), l N ) with the initial condition x ) = 2) where designates the erasure roaility of the BEC. Considering a sequence of {n m,λ,ρ)} LDPC ensemles where we let the lock length n m tend to infinity, the average it erasure roaility after the l th iteration is given y P l) = L ρ x l) ) ) 3)

9 SASON AND WIECHMAN: BOUNDS ON THE NUMBER OF ITERATIONS FOR TURBO-LIKE ENSEMBLES OVER THE BINARY ERASURE CHANNEL9 sloe=λ 2 ) sloe=λ 2 ) v Right to Left Message Erasure Proaility vx) = λ x/) x [,] sloe=λ 2 ) sloe=λ 2 ) sloe=λ 2 ) v 3... h cx) = ρ x) v 2 h 3 sloe = ρ ) v h 2 sloe = ρ ) h sloe = ρ ) sloe = ρ ) h sloe = ρ ) Left to Right Message Erasure Proaility Fig. 2. Plot of the functions cx) and vx) for an ensemle of LDPC codes which achieves vanishing it erasure roaility under iterative message-assing decoding when communicated over a BEC whose erasure roaility is equal to. The horizontal and vertical lines track the evolution of the exected fraction of erasure messages from the variale nodes to the check nodes at each iteration of the message-assing decoding algorithm. where L designates the common left degree distriution of the ensemles from the node ersective. Since the function fx) = λ ρ x) ) is monotonically increasing, Eqs. ) 3) imly that an average it erasure roaility of P is attainale under iterative message-assing decoding if and only if λ ρ x) ) < x, x x,] ) where x is the unique solution of P = L ρ x ) ). Let us define the functions cx) ρ x), vx) = { ) λ x x < x. 5) From the condition in ), an average it erasure roaility of P is attained if and only if cx) < vx) for all x x,]. Since we assume that vanishing it erasure roaility is achievale under message-assing decoding, it follows that cx) < vx) for all x,]. Figure 2 shows a lot of the functions cx) and vx) for an ensemle of LDPC codes which achieves vanishing it erasure roaility under iterative decoding as the lock length tends to infinity. The horizontal and vertical lines, laeled { } h l l N and { } v l, resectively, are used to track the exected l N fraction of erased messages from the variale nodes to the arity-check nodes at each iteration of the messageassing decoding algorithm. From ) and 2), the exected fraction of erased left to right messages in the l th

10 ACCEPTED TO THE IEEE TRANSACTIONS ON INFORMATION THEORY, REVISED FEBRUARY 29. decoding iteration where we start counting at zero) is equal to the x value at the left ti of the horizontal line h l. The right-angled triangles shaded in gray will e used later in the roof. The first ste in the roof of Theorem 3. is calculating the area ounded y the curves cx) and vx). This is done in the following lemma which is ased on the area theorem for the BEC [5]. Lemma.: ) C R vx) cx) dx = 6) a L where C = is the caacity of the BEC, R is the design rate of the ensemle, and a L is the average left degree of the ensemle. Proof: The roof of this equality is straightforward. Alternatively, the reader is referred to the matching condition in [22, Section 3..] which is justified via the area theorem in [5]. Let us consider the two sets of right-angled triangles shown in two shades of gray in Figure 2. The set of triangles which are shaded in dark gray are defined so that one of the legs of triangle numer i counting from right to left and starting at zero) is the vertical line v i, and the sloe of the hyotenuse is equal to c ) = ρ ). Since cx) is concave for all x [,], these triangles are guaranteed to e aove the curve of the function c. Since the sloe of the hyotenuse is ρ ), the area of the i th triangle in this set is A i = 2 v i ) vi ρ = v i 2 ) 2ρ ) 7) where v i is the length of v i. We now turn to consider the second set of triangles, which are shaded in light gray. Note that the function λx) is monotonically increasing and convex in [,] and also that λ) = and λ) =. This imlies that λ is concave in [,] and therefore vx) is concave in [,]. The triangles shaded in light gray are defined so that one of the legs of triangle numer i again, counting from the right and starting at zero) is the vertical line v i and the sloe of the hyotenuse is given y v ) = λ ) ) = λ ) = λ 2 where the second equality follows since λ) =. The concavity of vx) in [, ] guarantees that these triangles are elow the curve of the of function v. The area of the i th triangle in this second set of triangles is given y B i = 2 v i v i λ 2 ) = λ 2 v i 2. 8) 2 Since vx) is monotonically increasing with x, the dark-shaded triangles lie elow the curve of the function v. Similarly, the monotonicity of cx) imlies that the light-shaded triangles are aove the curve of the function c. Hence, oth sets of triangles form a suset of the domain ounded y the curves of cx) and vx). By their definitions, the i th dark triangle is on the right of v i, and the i th light triangle lies to the left of v i ; therefore, the triangles do not overla. Comining 7), 8) and the fact that the triangles do not overla, and alying Lemma., we get C R a L = 2 i= vx) cx) ) dx A i + B i ) ρ ) + λ 2 ) l i= v i 2 9) where l is an aritrary natural numer. Since we assume that the it erasure roaility vanishes under iterative message-assing decoding, the staility condition imlies that ρ ) λ 2. 2)

11 SASON AND WIECHMAN: BOUNDS ON THE NUMBER OF ITERATIONS FOR TURBO-LIKE ENSEMBLES OVER THE BINARY ERASURE CHANNEL Sustituting 2) and R = ε)c in 9) gives The definition of h l and v l in Figure 2 imlies that for an aritrary iteration l l Cε a L λ 2 v i 2. 2) i= l ρ x l) ) = cx l) ) = v i. i= Sustituting the last equality in 3) yields that the average it erasure roaility after iteration numer l can e exressed as ) l = L v i. 22) P l ) Let l designate the numer of iterations required to achieve an average it erasure roaility P over the ensemle where we let the lock length tend to infinity), i.e., l is the smallest integer which satisfies P l ) P since we start counting at l =. Although we consider an exectation over the LDPC ensemle, note that l is deterministic as it is the smallest integer for which the average it erasure roaility does not exceed P. Since L is monotonically increasing, 22) rovides a lower ound on l i= v i of the form l ) v i L P. 23) i= From the Cauchy-Schwartz inequality, we get l 2 v i ) i= l i= l i= v i 2 = l v i 2. 2) i= Comining the aove inequality with 2) and 23) gives the inequality a L λ 2 L )) 2 P Cε l which rovides the following lower ound on the numer of iterations l: a L λ 2 L )) 2 P l. 25) )ε To continue the roof, we derive a lower ound on L x) for x,). Since the fraction of variale nodes of degree i is non-negative for all i = 2,3,..., we have Sustituting t = Lx) gives Lx) = i l i= L i x i L 2 x 2, x. t L 2 L t) ) 2, t,) which is transformed into the following lower ound on L x): x L x), x,). 26) L2 Under the assumtion P < L 2, sustituting 26) in 25) gives l a ) 2 L λ 2 L2 P L 2 )ε = a L λ 2 L2 ) 2 P. 27) L 2 )ε

12 2 ACCEPTED TO THE IEEE TRANSACTIONS ON INFORMATION THEORY, REVISED FEBRUARY 29. λ2 al The lower ound in 7) is otained y sustituting the equality L 2 = 2 into 27). Taking the limit where the average it erasure roaility tends to zero on oth sides of 7) gives the following lower ound on the numer of iterations: lε,,p ) 2 L 2 ε). ε B. Proof of Theorem 3.2 We egin the roof y considering the exected fraction of erasure messages from the unctured it nodes to the arity-check 2 nodes see Fig. ). The following lemma rovides a lower ound on the exected fraction of erasures in the l th decoding iteration in terms of this exected fraction at the receding iteration. Lemma.2: Let n,λ,ρ) e an ensemle of systematic ARA codes whose transmission takes lace over a BEC with erasure roaility. Then, in the limit where the lock length tends to infinity, the exected fraction of erasure messages from the unctured it nodes to the arity-check 2 nodes at the l th iteration satisfies x l) λ ρ x l ) ) ), l =,2,... 28) where the tilted degree distriutions λ and ρ are given as follows see [2]): ) 2 λx) λx) 29) )Lx) ) 2 ρx) ρx) 3) Rx) and L and R designate the degree distriutions of the ARA ensemle from the node ersective. Proof: See Aendix II.A. From Fig., it can e readily verified that the roailities x and x for erasure messages at iteration numer zero are equal to, i.e., x ) = x ) =. 3) Let us look at the RHS of 28) as a function of x, and oserve that it is monotonically increasing over the interval [, ]. Let us comare the erformance of a systematic ARA ensemle whose degree distriutions are λ, ρ) with an LDPC ensemle whose degree distriutions are given y λ, ρ) see 29) and 3)) under iterative messageassing decoding. Given the initial condition x ) =, the following conclusion is otained y recursively alying Lemma.2: For any iteration, the erasure roaility for messages delivered from unctured it nodes to aritycheck 2 nodes of the ARA ensemle see Fig. ) is lower ounded y the erasure roaility of the left-to-right messages of the LDPC ensemle; this holds even if the a-riori information from the BEC is not used y the iterative decoder of the LDPC ensemle note that the coefficient of λ in the RHS of 28) is equal to one). Note that unless the fraction of arity-check 2 nodes of degree is strictly ositive i.e., R > ), the iterative decoding cannot e initiated for oth ensemles unless some the values of some unctured its of the systematic ARA ensemle are known, as in [2]). Hence, the comarison aove etween the ARA and LDPC ensemles is of interest under the assumtion that R > ; this roerty is imlied y the assumtion of vanishing it erasure roaility for the systematic ARA ensemle under iterative message-assing decoding. In [2, Section II.C.2], a technique called grah reduction is introduced. This technique transforms the Tanner grah of a systematic ARA ensemle, transmitted over a BEC whose erasure roaility is, into a Tanner grah of an equivalent LDPC ensemle where this equivalence holds in the asymtotic case where the lock length tends to infinity). The variale and arity-check nodes of the equivalent LDPC code evolve from the unctured it and arity-check 2 nodes of the ARA ensemle, resectively, and their degree distriutions from the edge ersective) are given y λ and ρ, resectively. It is also shown in [2] that λ and ρ are legitimate degree distriution functions, i.e., all the derivatives at zero are non-negative and λ) = ρ) =. As shown in [2, Eqs. 9) 2)], the left and right degree distriutions of the equivalent LDPC ensemle from the node ersective are given, resectively, y Lx) = x λt)dt = λt)dt Lx) )Lx) 32)

13 SASON AND WIECHMAN: BOUNDS ON THE NUMBER OF ITERATIONS FOR TURBO-LIKE ENSEMBLES OVER THE BINARY ERASURE CHANNEL3 and Rx) = x ρt)dt = ρt)dt )Rx) Rx). 33) Let P l) designate the average erasure roaility of the systematic its after the l th decoding iteration where we start counting at l = ). For LDPC ensemles, a simle relationshi etween the erasure roaility of the code its and the erasure roaility of the left-to-right messages at the l th decoding iteration is given in 3). For systematic ARA ensemles, a similar, though less direct, relationshi exists etween the erasure roaility of the systematic its after the l th decoding iteration and x l) ; this relationshi is resented in the following lemma. Lemma.3: Let n,λ,ρ) e an ensemle of systematic ARA codes whose transmission takes lace over a BEC with erasure roaility. Then, in the asymtotic case where the lock length tends to infinity, the average erasure roaility of the systematic its after the l th decoding iteration, P l), satisfies the inequality P l) L ρ x l) where ρ and L are defined in 3) and 32), resectively similarly to their definitions in [2]). Proof: See Aendix II.B. Remark.: We note that when P l) is very small, the LHS of 3) satisfies P l) P l) 2, so 3) takes a similar form to 3) which refers to the erasure roaility of LDPC ensemles. Consider the numer of iterations required for the message-assing decoder, oerating on the Tanner grahs of the systematic ARA ensemle, to achieve a desired it erasure roaility P. Comining Lemmas.2 and.3, and the initial condition in 3), a lower ound on this numer of iterations can e deduced. More exlicitly, it is lower ounded y the numer of iterations which is required to achieve a it erasure roaility of )) P the LDPC ensemle whose degree distriutions are λ, ρ) and where the erasure roaility of the BEC is equal to. It is therefore temting to aly the lower ound on the numer of iterations in Theorem 3., which refers to LDPC ensemles, as a lower ound on the numer of iterations for the ARA ensemle. Unfortunately, the LDPC ensemle with the tilted air of degree distriutions λ, ρ) is transmitted over a BEC whose erasure roaility is, so the channel caacity is equal to zero and the multilicative ga to caacity is meaningless. This revents a direct use of Theorem 3.; however, the continuation of the roof follows similar lines in the roof of Theorem 3.. Let x denote the unique solution in [,] of the equation 3) P = L ρ x ) ). 35) From 28), 3) and 3), a necessary condition for achieving a it erasure roaility P of the systematic its is that λ ρ x) ) < x, x x,]. 36) In the limit where the fixed oint of the iterative decoding rocess is attained, the inequalities in 28), 3) and 3) are relaced y equalities; hence, 36) also forms a sufficient condition. Analogously to the case of LDPC ensemles, as in the roof of Theorem 3., we define the functions cx) = ρ x) and vx) = λ x). 37) Due to the monotonicity of λ in [,], the necessary and sufficient condition for attaining an erasure roaility P of the systematic its in 36) can e rewritten as cx) < ṽx), x x,]. for

14 ACCEPTED TO THE IEEE TRANSACTIONS ON INFORMATION THEORY, REVISED FEBRUARY 29. Since we assume that the sequence of ensemles asymtotically achieves vanishing it erasure roaility under message-assing decoding, it follows that cx) < ṽx), x,]. The next ste in the roof is calculating the area of the domain ounded y the curves cx) and ṽx). This is done in the following lemma which is analogous to Lemma.. Lemma.: ṽx) ) C R cx) dx = 38) R)a R where ṽ and c are introduced in 37), C = is the caacity of the BEC, R is the design rate of the systematic ARA ensemle, and a R is defined in ) and it designates the average degree of the arity-check 2 nodes when the edges that are connected to the code it nodes are ignored. Proof: From 37) ṽx) cx) ) dx = = = λ x)dx + ρx)dx ) λx)dx + ρ x)dx ρx)dx λx)dx 39) where the second equality is otained via integration y arts note that λ) = and λ) = ). From 32), we get λx)dx = L ) = L ) = ) a L see also [2, Eq. 23)]) where a L is defined in ), and it designates the average degree of the unctured it nodes in Fig. when the edges that are connected to the arity-check nodes are ignored. Similarly, 33) gives see also [2, Eq. 2)]). Sustituting ) and ) into 39) gives ρx)dx = R ) = R ) = a R ) ṽx) cx) ) dx = a R a L a) = [ )] al + a R a R a }{{ L } R = a R R R = C R R)a R 2) where a) follows since the design rate of the systematic ARA ensemle is given y R = ar a L+a R see Fig. ). To continue the roof, we consider a lot similar to the one in Figure 2 with the excetion that cx) and vx) are relaced y cx) and ṽx), resectively. Note that in this case the horizontal line h is reduced to the oint, ). Consider the two sets of gray-shaded right-angled triangles. The triangles shaded in dark gray are defined so that the height of triangle numer i counting from right to left and starting at zero) is the vertical line v i and

15 SASON AND WIECHMAN: BOUNDS ON THE NUMBER OF ITERATIONS FOR TURBO-LIKE ENSEMBLES OVER THE BINARY ERASURE CHANNEL5 the sloe of their hyotenuse is equal to c ) = ρ ). Since cx) is concave, these triangles form a suset of the domain ounded y the curves cx) and ṽx). The area of the i th triangle in this set is given y A i = ) 2 v vi i ρ = v i 2 ) 2 ρ ) where v i is the length of v i. The second set of right-angled triangles, which are shaded in light gray, are also defined so that the height of the i th triangle counting from right to left and starting at zero) is the vertical line v i, ut the triangle lies to the left of v i and the sloe of its hyotenuse is equal to ṽ ) = λ ) ) = λ ) = 2 λ ) = 2 λ 2 where the second equality follows since λ) = and the third equality follows from the definition of λ in 29). Since λ is monotonically increasing and convex over the interval [,] and it satisfies λ) = and λ) =, then it follows that vx) = λ x) is concave over this interval. Hence, the triangles shaded in light gray also form a suset of the domain ounded y the curves cx) and vx). The area of the i th light-gray triangle is given y B i = ) 2 v i v i 2 λ 2 = 2 λ 2 v i 2 2 Alying Lemma. and the fact that the triangles in oth sets do not overla, we get C R ) R)a R 2 ρ ) + l 2 λ 2 v i 2 3) where l is an aritrary natural numer. Since the sequence of ensemles asymtotically achieves vanishing it erasure roaility under iterative message-assing decoding, the staility condition for systematic ARA codes see 6) or equivalently [2, Eq. )]) imlies that 2 λ 2 ρ ) + 2R ) i= = ρ ) where the last equality follows from 3). Sustituting ) in 3) gives C R R)a R 2 λ 2 l i= ) v i 2. 5) Let x l) denote the x value of the left ti of the horizontal line h l. The value of x l) satisfies the recursive equation x l+) = λ ρ x l))), l N 6) with x ) =. As was exlained aove immediately following Lemma.2), from 28), 3), and the monotonicity of the function fx) = λ ρ x) ) over the interval [,], we get that x l) x l) for l N. The definition of h l and v l in Figure 2 imlies that Starting from 3) and alying the monotonicity of L and ρ gives P l ) L ρ x l)) = c x l)) l = v i. 7) i= = L ρ x l ) ) ) L ρ x l ))) ) l v i i=

16 6 ACCEPTED TO THE IEEE TRANSACTIONS ON INFORMATION THEORY, REVISED FEBRUARY 29. where the last equality follows from 7). Since L is strictly monotonically increasing in [,], then l v i L P l ) ). 8) i= Alying the Cauchy-Schwartz inequality as in 2)) to the RHS of 5), we get C R 2 l λ 2 R)a R 2 λ 2 l 2 λ 2 l v i 2 i= l ) 2 v i i= L P l ) ) 2 where the last inequality follows from 8). Since the design rate R is assumed to e a fraction ε of the caacity of the BEC, the aove inequality gives )) 2 2 λ 2 R)a R L P l ) Cε l where l is an aritrary natural numer. Let l designate the numer of iterations required to achieve an average it erasure roaility P of the systematic its, i.e., l is the smallest integer which satisfies P l ) P since we start counting the iterations at l = ). Note that l is deterministic since it refers to the smallest numer of iterations required to achieve a desired average it erasure roaility over the ensemle. From the inequality aove and the monotonicity of L, we otain that Cε 2 λ 2 R)a R L )) 2 P l which rovides a lower ound on the numer of iterations of the form l 2 λ 2 R)a R L )) 2 P Cε = 2 λ 2 ε)a L L )) 2 P 9) ε where the last equality follows since ar a L a lower ound on L x). Following the same stes which lead to 26) gives the inequality where 32) imlies that = R R L x) L 2 = L ) 2 see Fig. ) and R = ε)c. To continue the roof, we derive x L 2, x 5) = L ) 2 Under the assumtion that P < L 2, sustituting 5) and 5) in 9) gives λ 2 ε)a L L2 l L 2 ε = L 2. 5) P ) 2. 52) Finally, the lower ound on the numer of iterations in 8) follows from 52) y sustituting L 2 = λ2 al 2.