A Memory Efficient Array Architecture for Real-Time Motion Estimation

Transcription

1 A Memoy Efficient Aay Achitectue fo Real-Time Motion Estimation Vasily G. Moshnyaga and Keikichi Tamau Depatment of Electonics & Communication, Kyoto Univesity Sakyo-ku, Yoshida-Honmachi, Kyoto 66-1, JAPAN Abstact A new 2-D aay achitectue fo eal-time video pictue motion estimation is pesented. Due to incopoated concepts of video memoy distibution and shaing, the achitectue ensues feasible solutions fo the HDTV pictue fomat with twice lowe memoy equiements. It featues minimal I/O pin count, 1% pocesso utilization and is quite suitable fo VLSI implementation. 1 Intoduction 1.1 Motivation Motion estimation is a basic bandwidth compession method used in video-coding systems. Among seveal computation methods[3], the Full Seach Block Matching Algoithm (FBMA) is most popula. Having successive video fames divided into blocks of (N 2 N) pixels, the FBMA detemines a motion vecto, v, fo evey efeence block () of the cuent image by compaing it with all candidate blocks (Y ) within the seach aea suounding the position of the efeence block in the pevious fame. Let x(i; j) be the pixels of the efeence block, y(i; j) the pixels of the candidate block and p the maximum displacement allowed in both vetical and hoizontal diections. The indexes m and n indicate the position of the candidate block within the seach aea. Then, the position (m; n) of a candidate block that esults in the minimum distotion D denotes the motion vecto v: D(m; n) = N1 N1 i= j= jy(i + m; j + n) x(i; j)j; p m; n p 1 (1) v = ag min D(m; n) pm;np1 (2) The FBMA povides optimal pecision, egula data flow as well as highe paallelism, a chaacteistic that is advantageous fo VLSI implementation. Howeve, it is extemely time consuming pocess, since (2p) 2 compaisons have to be computed fo each of the displacements in the fame. If a fame has pixels (standad TV), and a fame ate of 25Hz, ove 9 million opeations on 6-16-bit data ae equied fo p = 8, N = 16. A numbe of hadwae achitectues have been poposed in ode to cope with eal time and high volume equiements. Refeences [1-3] give good suveys of them. These achitectues make use of massive pipelining and paallel pocessing povided by systolic[4,7-9] o linea aays[6] o tee-like stuctues[5]. Despite diffeences, all of them have one featue in common: they all assume that image data is stoed in a memoy extenal to the pocessing aay. Theefoe the achitectual solutions ae focused on the data flow within pocesso element () aay to keep the s as busy as possible and at the same time minimize the equied I/O bandwidth. In ode to educe the I/O s count, the achitectues include eithe an "on-chip" RAM memoy to stoe the seach aea / efeence block o a lage numbe of line buffes o pipelined egiste chains to boadcast the data between the s duing the computational pocess. As esult, the total memoy used fo stoing the seach aea / efeence block is duplicated in the motion estimation system inceasing its size. Since all the existing achitectues equie two sepaate memoies fo stoing the cuent fame and the pevious fame of 9 KByte each (fo the HDTV fomat), moe than 7% of the total system aea is occupied by memoy units only! Hence, an efficient memoy utilization becomes one of the most impotant design poblem, especially fo potable application. 1.2 Contibution In this pape, we popose a new distibuted memoy achitectue fo full-seach block matching motion estimation. In ou design, explicit povisions have been made to keep the total memoy size and the numbe of tansfes as small as possible, while maintaining high thoughput that is equied fo HDTV applications. Compaed to existing achitectues, ou achitectue does not equie lage extenal banks; it ensues minimal I/O bandwidth, 1% pocesso utilization and is linealy scalable. The achitectue featues egula and simple inteconnect scheme and is suitable fo VLSI implementation. 2 The achitectue The poposed achitectue consists of a two-dimensional mesh aay of (2p) 2 s with wap-aound connections and the minimum distotion block (MDB) at the output of the aay, as it illustated in Figue 1. The mesh aay computes equation (1) fo evey efeence block,, in the cuent image fame, while the MDB implements (2) and outputs the motion vecto, v, fo the fame. In the sequel, we will

2 Min distotion computation block Figue 1: The poposed achitectue assume that p = N=2 which is almost always acceptable in pactice. Two main featues distinguish ou achitectue fom othe 2-D mesh-aays: (1) non-ovelapping memoy distibution between s; (2) dynamic memoy shaing between cuent image and pevious image. 2.1 Memoy distibution The system memoy is distibuted between the s in a way that each is given a dedicated potion of the seach aea. The data to be pocessed by the (n; m) is stoed in its local memoy, LM (m; n). In ode to obtain a non-ovelapping data distibution ove the LMs, we goup the computations that ae executed on the same i j y -1,-1 y -1,1 y,-1 y, (a) y y1, -1,1 y y -1,-1-1, y,-1 y, P, P,1 P 1, P 1,1 z 1, z,1 z, (c) z, y -1,-1 y,-1 y, y -1,1 y, v z 1, z,1 (b) (,) (1,1) (d) y,-1 y, y, (,1) (1,) Figue 2: An illustation of the hadwae mapping fo N = 2 and p = 1: (a) seach aea of the efeence block (); (b) systolic mapping of computations to s; (c) the poposed mapping; (b) allocation of the s in aay z 1, z, z,1 seach aea pixels o on pixels whose coodinates diffe by q 2 (2p); (q = 1; 2;:::) and then map each goup to a pope of the aay. Let us illustate the mapping concept on a simple example. Assume N=2 and p=1. Then the seach aea of the efeence block,, includes 9 pixels, as it shown in Figues 2(a) with -pixels depicted in bold. The dependence gaph (DG) of the FBMA algotihm fo this case is dawn in Figue 2(b). In contast to a systolic mapping which assigns the DG nodes along the pattens of Fig.2(b) to a common, we map the nodes onto s such a way that the numbe of diffeent vaiables to be stoed in each is minimized. Figue 2(c) demonstates ou mapping. In this figue, vetical lines denote the seach aea pixels, hoizontal lines denote the efeence block pixels, cicles depict the AD-opeatos, pattens show the goups of nodes assigned to a common, ectangles epesent contents of the LMs in the coesponding s. Since, thee is no data ovelap between the LMs, matching of all the seach aea pixels against one efeence block pixel can be done in paallel. The mapping eleases us fom the buden of iteative seach aea boadcast (anothe big limitation of othe achitectues) and allows us to educe both the numbe of latches in the design and, which is vey impotant, powe consumption. In geneal, the numbe of tansfes is educed by a facto of (2p) 2. The only data which needs to be moved between the s is the accumulated AD-tem. So, the numbe of wiing inteconnections which have to be outed at the layout stage is also deceased. As esult, the VLSI outing becomes egula and simple. In addition, explicit povisions ae made duing hadwae mapping to localize the inteconnections in the mesh. Due to associativity of summation (1), a goup of computations, P (m; n), can be assigned to any in the ow m. Howeve, the closest neighboing connections emege only if the assignment is done in a left-otate fashion, that is by mapping the goup P (m; n) to the element (m; n m), as it shown in Fig.2(d). In this way, a egula and simple VLSI implementation is ensued. 2.2 Memoy shaing In ode to efficiently utilize the memoy space, we apply a gadual memoy enewal concept, when pixels of the cuent fame eplace pixels of the pevious fame that ae no longe needed. At evey moment in time, pixels of a cetain egion of the pevious fame ae being ead by the s, while anothe egion is being ovewitten with new pixels. Figue 3(a) illustates this concept fo fou N 2 N blocks (labeled with thick solid lines) which belong to two adjucent block slices in the fame. Due to displacement, p, in the seach aeas fo these blocks, the egion 1 of the pevious fame becomes vacant afte pocessing the efeence block B1, egion 2 afte pocessing B2, egion 3 afte B3 and egion 4 afte B4. Hence, memoy allocated fo stoing this pat of the pevious fame can be efficiently used fo stoing the cuent fame data (stating fom the block B4) without degading the quality of the algoithm. As esult, no othe memoy units ae needed except buffes (M) to stoe the N s 3 N=2 + N 2 p pixels of the cuent fame. Hee, N s denotes the size of the block slice. In ou achitectue the buffe memoy is also distibuted among the s. Figue 3(b) shows the elation of the indexes and the size of the buffe memoy (M) in the. Fo the seach ange of p = N=2 pixels, the seach aea

3 N p B1 1 2 B2 3 4 p N B3 B4 Seach aea of block 2,2 (a) ow indexes,,(p-1) p,,(2p-1) column indexes,,(p-1) (b) p,,(2p-1) N s 2+ N s (m,n-1) RAM (m-1,n) R M mx ADC Z Q Adde A (m,n+1) Figue 3: (a) An illustation of the memoy shaing concept, (b) the buffe memoy distibution among the s. (m+1,n) Figue 5: The stuctue LM(,) y(3p, p) y(3p,-p) y( p, p) y( p,-p) y(-p, p) y(-p,-p) LM(1,) y(3p+1,p) y(3p+1,-p) y(p+1,p) y(p+1,-p) y(-p+1, p) y(-p+1,-p) LM(p-1,) y(3p-1, p) y(3p-1,-p) y(p-1, p) y(p-1,-p) LM(,1) y(3p, p+1) y(3p,-p+1) y( p,p+1) y( p,-p+1) y(-p, p+1) y(-p,-p+1) LM(1,1) y(3p+1, p+1) y(3p+1,-p+1) y( p+1, p+1) y( p+1,-p+1) y(-p+1, p+1) y(-p+1,-p+1) LM(p-1,1) y(3p-1, p+1) y(3p-1,-p+1) y( p-1, p+1) y( p-1,-p+1) LM(, p-1) y(3p, p-1) y( p, p-1) y(-p, p-1) LM(1,p-1) y(3p+1, p-1) y( p+1, p-1) y(-p+1,p-1) LM(p-1,p-1) y(3p-1,3p-1) y( p-1, p-1) Figue 4: Geneal data distibution between the LMs data equied fo the adjacent blocks can shae the same memoy and need not to be loaded. Theefoe, we adopted a static image data distibution ove the local memoies of diffeent s as it shown in Figue 4. The bold lines in this figue indicate the pixels belonging to the seach aea ofthe fist block, dotted lines define the pixels coveed by the seach aea of the second block. As can be seen, the local memoies ae of diffeent sizes, with the lagest of the (,). Fo the standad TV fomat of pixels, N=16, p=8, the size of the lagest memoy unit is 3 KByte, while majoity of the LMs ae of 1.5 KByte each. 2.3 oganization Figue 5 shows the intenal stuctue of a. Each is divided in a two stage pipeline. The fist stage is constucted by Absolute Diffeence Calculato (ADC) and the second by the adde and RAM. The ADC computes the absolute diffeence (AD) between the seach pixel datum, y(m + i; n + j), stoed in the egiste (R) and the efeence pixel, x(i; j), fed fom the input,. Afte being delayed one clock cycle by the latch, Q, the AD value is added to the patial sum stoed in the egiste Z, and the esult is witten to the egiste Z of its ight o down-neighbo, depending on the contol of the multiplexo (mx). At the same stage, the RAM eads its addess (A) to the egiste R, o eceives in (A) data fom the buffe M. Due to egula data flow in the diagonal diection (fom bottom-left to the top-ight), no complicated addess geneatos ae needed. 2.4 The system opeation We assume that befoe pocessing a new efeence block, all the s have zeos in egistes Z and initial seach aea pixels y( + m; + n); p m; n p 1, in the egistes R. The opeation begins with boadcasting a cuent block pixel, x(i; j), to all the s. In the clock cycle (t) evey executes in paallel thee disjoint opeations: (1) computes the AD value, Q t = jy t1 x(i; j)j, to be stoed in its egiste Q; (2) adds the content of its egiste Z to the AD value, Q t1, calculated at the pevious clock cycle (t 1) and wites the esult to Z egistes of its ight o downneighboing s; (3) eads a new seach aea pixel to the egiste R, o wites the pixel fom the buffe M to the RAM. If we assume that the cuent block pixels ente the system in a ow-based fashion, the ight neighbos ae selected each of [;N 1] cycles, while the down-neighbo is chose at the end of the column, that is in each N cycle. Thus duing the N steps, the (2p) 2 absolute diffeences calculated in the fist clock step ae iteatively moved along the ows N times; each time accumulating the esult computed at the they visit. Figue 6 illustates the aay pefomance by outlining the esults of computations in the s (Fig.2,c) duing the fist 4 clock steps. The column () in this figue shows the cuent block data. The columns show contents of the egistes Q and Z fo each. As can be seen, the accumulation pocess is dynamic; unlike those of othe achitectues. The sum, Q + Z, computed in the (,) at the clock cycle 1 tavels the s in the ight diection, each time accumulating a new AD-tem. (The pattens show the opeands added in the coesponding clock step). At the end of cycle 2, the s wite the sums to the egistes Z of thei down-adjacent s. Thus the esults accumulated in s of the fist ow will be stoed in the Z egistes

4 t (,) (,1) (1,) 1 x(,) y(-1,-1)-x(,) y(-1,)-x(,) y(,-1)-x(,) 2 x(,1) y(-1, 1)-x(,1) y(-1,)-x(,) y(-1,)-x(,1) y(-1,-1)-x(,) y(,1)-x(,1) 3 4 x(1,) x(1,1) y(1,-1)-x(1,) y( 1, 1)-x(,1) y(,)-x(,1) P(1,1) Q Z Q Z Q Z Q Z y(,-1)-x(,) + y(,)-x(,1) y(,)-x(,) + y(,1)-x(,1) + y(1,)-x(1,) y(,)-x(,) + y(,1)-x(,1) + y(1,)-x(1,) + y( 1, 1)-x(,1) y(1,)-x(1,) y(1,)-x(1,1) y(,)-x(,) + y(,1)-x(,1) y(,-1)-x(,) + y(,)-x(,1) + y(1,-1)-x(1,) y(,-1)-x(,) + y(,)-x(,1) + y(1,-1)-x(1,) + y(1,)-x(1,1) y(,-1)-x(1,) y(,1)-x(1,1) y(-1,-1)-x(,) + y(-1,)-x(,1) y(-1,)-x(,) + y(-1, 1)-x(,1) + y(,)-x(1,) y(-1,)-x(,) + y(-1, 1)-x(,1) + y(,)-x(1,) + y(,1)-x(1,1) Figue 6: Data flow in the 2x2 aay achitectue y(,)-x(,) y(,)-x(1,) y(,)-x(1,1) y(,-1)-x(,) y(-1,)-x(,) + y(-1, 1)-x(,1) y(-1,-1)-x(,) + y(-1,)-x(,1) + y(,-1)-x(1,) y(-1,-1)-x(,) + y(-1,)-x(,1) + y(,-1)-x(1,) + y(,)-x(1,1) of the elements (1;);(1;1), of the second ow, while the Z egistes of (;);(;1) will eceive data accumulated in (1;1);(1;), espectively. Geneally, afte N 2 clock cycles, all pixels of the cuent block ae pocessed and all the (2p) 2 distotion measues (2) ae available simultaneously. The last ow in Figue 6 exemplifies the distotion measues accumulated in each. The minimum of them is detemined in the mininmum distotion block (MDB) in paallel to pocessing of a new cuent block in the aay. Figue 7 shows the distibution of memoy accesses in the mesh-aay duing pocessing of block,, with displacement p = 2. The fist two columns in this figue ae the same, as in the Fig.6, the -columns define ead () o wite (gey pattens) accesses to the local memoy of the coesponding in the coesponding clock cycle. Due to RAM s inability to pefom simultaneous ead/wite, the efeence block data is witten to the memoy with a delay to its aival time on the system input. Fo instance, the x(1; ) pixel aives in clock cycle 5 but the (1;) wites it to memoy only in the clock cycle 6. Note that evey in the aay makes one wite and seveal ead accesses to its RAM duing one block pocessing. Fo applications with N = 2p, the maximal numbe of accesses each makes to its LM is 2N, while the total numbe of memoy accesses in the aay pe block is N 2 [2p((2p 1)2) +1]+N 2. By dublicating the egiste R in each fo taking the epeatedly used data of y(n; m) and y(n; m + 2p) fom the egistes but not memoy, we educe the total numbe of memoy accesses to 4p(2p 1) +N +N 2. Although this numbe is quite low, all the s ae 1% utilized. 3 Compaison Since the poposed achitectue pefoms motion estimation in the block-scan mode taking N 2 clock cycles pe block, the latency fo estimating one slice of blocks is L s = N s 2 N 2. Fo an image of size N s 2 N v blocks, the latency fo estimating one fame is L f = N v 2 L s. Hence the achitectue can estimate up to f max = 1=clock peiod 2 L f fames pe second. s t,,1,2,3 1, 1,1 1,2 1,3 2, 2,1 2,2 2,,3 3, 3,1 3,2 3,3 1 x(,) v(3,2) v(3,3) 2 x(,1) x(,) 3 x(,2) x(,1) 4 x(,3) 5 x(1,) x(,2) x(,3) 6 x(1,1) x(1,) 7 x(1,2) x(1,1) 8 x(1,3) 9 x(2,) x(1,2) x(1,3) 1 x(2,1) x(2,) 11 x(2,2) x(2,1) 12 x(2,3) 13 x(3,) x(2,2) x(2,3) 14 x(3,1) x(3,) 15 x(3,2) x(3,1) 16 x(3,3) Figue 7: A ead-wite sequence in the aay The compaison of the poposed achitectue to the othe existing achitectues is pesented in Table 1-3. The standad TV fomat (CCIR Rec.61) with N = 16, p = 8 and the HDTV pictue fomat with N = 16, p = 16, and clock peiod of 25 ns ae used. In Table 1,2, the numbe of s, the total memoy size, the I/O count, the numbe of clock cycles equied to estimate a efeence block and a fame, and the maximum numbe of fames to be estimated pe second (f max =s) have been compaed. Table 3 shows the total numbe of data accesses pe block including efeence block data and the coesponding seach aea data. Fewe data access implies lowe powe dissipation, and hence is moe desiable. Compaing with othe achitectues, ou achitectue ensues feasible solutions fo the HDTV pictue fomat with the minimum numbe of I/O pins and as twice as less mem-

5 Table 1: Compaison esults: Video Fomat CCIR Rec.61, N = 16, p = 8 Refeence # Memoy # I/O # clock cycles f max size (KB) pe block pe fame /sec [1] 256 2(45) 136 2p(N + 2p 1) [4] 256 2(45) 16 (N + 2p 1) , [5] 512 2(45) 2,56 (2p) 2 + log 2 N , [6] 16 2(45) 32 N 2 (2p) 4,96 6,635,52 6 [7] 256 2(45) 136 N 2 + 2p(N + 2p 1) +N 768 1,244,16 32 [8] 256 2(45) 32 N , Ou N ,72 96 Table 2: Compaison esults: HDTV Fomat, N = 16, p = 16 Type # Memoy # I/O # clock cycles f max size (KB) pe block pe fame /sec [1] 256 2(9) 136 1,54 5,414,4 7 [4] 256 2(9) 16 2,29 7,952,4 5 [5] 512 2(9) 2,56 1,34 3,686,41 1 [6] 16 2(9) 56 4,96 14,745,6 3 [7] 256 2(9) 136 1,776 6,393,6 6 [8] 124 2(9) ,12 42 Ou ,6 42 Table 3: Video Fomat CCIR Rec.61, N = 16, p = 8 Refeence # Data accesses pe block [1] N 2 + N (2p)(N + 2p 1) 8,192 [4] N 2 +(N +2p1) 2 1,217 [5] N 2 ((2p) 2 + 1) 65,792 [6] (2p)(N 2 + N (N + 2p)) 12,288 [7] N 2 + N (2p)(N + 2p 1) 8,192 [8] 3N Ou 4p(2p 1) +N +N oy size. In contast to [8], whee each fist block in a ow is evaluated two times slowe than the othes, ou achitectue pocesses all the image blocks with equal speed (256 clock cycles pe block). As esult, it can estimate the maximum numbe of fames pe second. 4 Conclusions We pesented a new aay achitectue fo video motion estimation. Due to the video memoy distibution ove the aay elements, specific mapping of computations to aay and the dynamic memoy shaing among consequtive fames we deived a vey efficient solution fo eal-time HDTV applications. Peliminay layouts indicate that the physical aea of a motion estimation chip with N = 16, p = 8 is appoximately 2/3 smalle than the total aea of a conventional motion estimation system. Detailed chip design in.3m CMOS technology is cuently in pogess. Refeences [1] T.Komaek and P.Pitsch, Aay achitectues fo blockmatching algoithms, IEEE Tans. CAS, Vol.36, No.1, Oct. 1989, pp [2] P.Pisch, N.Demassieux, W.Gehke, VLSI achitectues fo video compession, Poceedings of the IEEE, Vol.83, No.2, Feb. 1995, pp [3] M.Sung, Algoithms and VLSI achitectues fo motion estimation, VLSI Implementations fo Image Communications, P.Pisch (Ed.), 1993, pp [4] C.Hsieh and T.Lin, VLSI Achitectue fo block-matching motion estimation algoithm", IEEE Tans.CAS fo Video Technology, Vol.2, No.2, June 1992, pp [5] Y.Jehng, et al., An efficient and simple VLSI tee achitectue fo the motion compensation block-matching algoithms", IEEE Tans. Signal Pocess., Vol.41, Feb. 1993, pp [6] K.Yang, et al., A family of VLSI designs fo the motion compensation block-matching algoithm", IEEE Tans. CAS, Vol.36, Oct. 1989, pp [7] E.Chan, et al., Motion estimation achitectue fo video compession", IEEE Tans. Consume Electon., Vol.39, No.3, Aug. 1993, pp [8] H.Yeo and Y.Hu, A novel modula systolic aay achitectues fo full-seach block matching motion estimation", IEEE Tans. CAS fo Video Technology, Vol.5, No.5, Oct. 1995, pp [9] J.Baek, et al, A fast aay achitectue fo block matching algoithm", Poc. IEEE ISCAS, 1994, pp