Reconfigurable Shape-Adaptive Template Matching Architectures

Transcription

1 Reonfigurable Shae-Adative Temlate Mathing Arhitetures Jörn Gause 1, Peter Y.K. Cheung 1, Wayne Luk 2 1 Deartment of Eletrial and Eletroni Engineering, Imerial College, London SW7 2BT, England. 2 Deartment of Comuting, Imerial College, London SW7 2BZ, England. Abstrat This aer resents reonfigurable omuting strategies for a Shae-Adative Temlate Mathing (SA-TM) method to retrieve arbitrarily shaed objets within images or video frames. A generi systoli array arhiteture is roosed as the basis for omaring three designs: a stati design where the onfiguration does not hange after omilation, a artially-dynami design where a stati iruit an be reonfigured to use different on-hi data, and a dynami design whih omletely adats to a artiular omutation. While the logi resoures required to imlement the stati and artially-dynami designs are onstant and deend only on the size of the searh frame, the dynami design is adated to the size and shae of the temlate objet, and hene requires muh less area. The exeution time of the mathing roess greatly deends on the number of frames the same objet is mathed at. For a small number of frames, the dynami and artially-dynami designs suffer from high reonfiguration overheads. This overhead is signifiantly redued if the mathing roess is reeated on a large number of onseutive frames. We find that the dynami SA-TM design in a 5 MHz Virtex 1E devie, inluding reonfiguration time, an erform more than 28, times faster than a 1.4 GHz Pentium 4 PC when roessing a 1 1 temlate on 3 onseutive video frames in HDTV format. 1 Introdution The develoment of multimedia tehnology and assoiated standards like MG-4 for oding of audio-visual objets in multimedia aliations [1] and MG-7 for desrition and searh of audio and visual multimedia ontent [2] leads to new tyes of video roessing algorithms and therefore new hallenges for their hardware imlementation. Comrehensive aetane of new multimedia servies and aliations deends on the availability of inexensive, omat hardware delivering the high erformane required. In addition to very high roessing demands, many multimedia roessing algorithms tend to be haraterised by a dereasing regularity and reditability of oerations. Tyial examles are algorithms to roess arbitrarily shaed multimedia objets: the omutations to be erformed need to be adated to the size and the shae of the objet. This alls for arhitetures with inreased flexibility at run time [3]. In this aer, a shae-adative temlate mathing (SA- TM) method to retrieve arbitrarily shaed objets within images or video frames is roosed. The algorithm is truly objet-oriented as it uses only the objet of interest as temlate, not the bakground ixels, and it does not divide the temlate into a number of square bloks in aordane with future generation multimedia tehniques [4]. Software solutions whih ould rovide the flexibility to adat to different temlates are too slow to allow real-time roessing at video frame rate, whereas an ASIC imlementation is imossible due to the infinite number of sizes of temlate and searh frame. Hene, the use of a reonfigurable omuting arhiteture, suh as SONIC [5] is roosed to imlement a fast and flexible SA- TM design, as shown in Figure 1. The urose of this aer is to investigate reonfigurable omuting strategies regarding their suitability for imlementing the SA-TM method as an examle of a tyial multimedia algorithm. A stati design, whih an math temlates stored in off-hi memory of all ossible shaes and sizes within a video frame using the same FPGA onfiguration is resented in this aer. A artially-dynami design whih uses on-hi memory, available in most reent FPGAs, to store the temlate is also introdued. While the iruit to omute the algorithm is stati, the temlate an be udated by reonfiguring the memory ortions of the devie. In addition, this aer resents a dynami design, where the onfiguration data are omletely adated to the shae and size of the temlate objet used, resulting in signifiant area savings. A generi systoli array arhiteture rovides the basis for the imlementation of the three reonfigurable SA-TM designs. We omare their area usage and omutation time, inluding reonfiguration and reomilation overheads. It is shown that the suitability of a artiular design regarding total exeution time of the algorithm strongly deends on the number of onseutive frames the oerations are arried out on. Setion 2 resents bakground information and revious work on multimedia searh and retrieval strategies and reonfigurable omuting, inluding run-time reonfiguration (RTR). Setion 3 desribes the shae-adative temlate mathing method alied in this aer. A systoli array for SA-TM is roosed in Setion 4, whih is used for a dynami, a stati and a artially-dynami realisation aroah. These designs are resented in Setion 5. In Setion 6 the FPGA imlementations results for all three designs are disussed. Finally, a onlusion follows in Setion 7.

2 Inut Frame Outut Frame Temlate Image best math (Arbitrary Shae) reonfigurable arhiteture (SONIC) [5] Figure 1 Reonfigurable Comuting for SA-TM 2 Bakground Multimedia searh and retrieval have beome an ative researh field due to the inreasing demand that aomany many new ratial aliations. More and more video data are generated every day. Amongst this large amount of multimedia information, searhing for seifi video setions or objets and retrieving them is a diffiult task. The traditional aroah of fast-forwarding of video and looking at the sreen to find interesting information is time-onsuming and labour-intensive. Ideally, video will be automatially annotated as a result of mahine interretation of the semanti ontent of the video data. However, given the state of the art in omuter vision, suh sohistiated data extration may not be feasible in ratie [6]. Video objet retrieval is onerned with returning similar video lis to a user given a visual objet as query. The reognition and retrieval an be feature-based or temlate-based. Colour is most frequently used for feature-based retrieval, as the retrieval algorithms using olour are haraterised by regular oerations and data aesses. Colour indexing and retrieval often involves the use of olour histograms whih reord the number of ixels in an image for eah olour [6]. However, simle histograms do not take the loation of a ixel with a artiular olour into aount. Temlate mathing is a lassial tehnique for loating the osition of a given small subimage inside a large image. It has been frequently used in the aliations of objet detetion, image registration and attern reognition. The mathing roess involves shifting a temlate image over a searh area, measuring the similarity between the temlate and the urrent searh area, and loating the best math osition. Due to their regularity, temlate mathing algorithms are suitable for ielined roessing in a systoli array. Various basi systoli array arhitetures with different degrees of arallelism are resented in [7]. However, temlate mathing entails great omutational omlexity for large searh areas and temlates. Hene, many ratial video and image retrieval systems whih are generally software based, use olour histograms to searh for a query image. Often, the query image has to be the same size as the searh image or video frame, although the user may only be interested in a artiular objet within an image. The use of bakground ixels, not belonging to the objet of interest, an therefore lead to unwanted results. Reonfigurable omuting, based on Field Programmable Gate Arrays (FPGAs) as roessing devies, has been identified to be well suited to deal with the requirements of roviding flexible, high-seed roessing, sine it ombines the advantages of software and aliation-seifi hardware. It allows user-level rogrammability at a low level and failitates general urose omuting due to its reonfigurability. Thus, many aliations an use the same hardware [13], [8]. Dynami or run-time reonfiguration (RTR) of FPGAs is reognised as an advaned aliation area within reonfigurable omuting. However, a lot of researh still has to be aomlished to fully understand and evaluate RTR and quantify the trade-offs of run-time reonfigurable devies and systems [8]. Currently only a small subset of available FP- GAs are aable of being reonfigured in this way, but there is a growing trend in the industry to rovide dynamially reonfigurable devies with varying degrees of onfiguration flexibility [9]. Dynamially reonfigurable devies are haraterised by their ability to ontinue to oerate without interrution while sub-setions of the array logi are being reonfigured. The authors of [1] distinguish between two modes of onfigurability: stati -- where the onfiguration

3 data of the FPGA is loaded one, after whih it does not hange during exeution of the task, and dynami -- where the FPGA s onfiguration may hange at any moment. The most ited motivations for using dynami reonfiguration are the aeleration of algorithms that might otherwise be imlemented on general-urose omuters, and the oortunity to inrease effetive logi aaity of rogrammable devies by maing only ative logi to FPGA resoures at any given time. It is redited in [11] that the imortane of dynamially reonfigurable logi will inrease as FPGAs beome larger. This statement is based on the observation that with more and more iruits resent in a single hi, there is a redued robability of them all being required to oerate at the same time. Three FPGA reonfiguration strategies (omile-time, run-time, and artial run-time reonfiguration) are evaluated in [12], based on a systoli array imlementation of a salar quantiser. It is shown that omile-time reonfiguration gives the best area-time rodut for the aliation used, whereas the suitability of run-time reonfiguration strongly deends on the number of reonfigurations. However, it is onluded that the results are aliation and tehnology deendent. In [13], the suitability of using reonfigurable omuting for imlementing omuter vision algorithms of different levels of regularity has been investigated. This inludes a systoli orrelation that an be used for temlate mathing. However, the design resented there is not shae-adative, and results are given only for relatively small and square masks (3 3) and images ( ). Video is not onsidered. Configurable omuting solutions for automati target reognition are resented in [14]. Here the temlates are binary and have a size of ixels, whereas the searh image is The temlates are maed onto the FPGA as simle adder trees, and the image ixels are shifted through and added at ositions where the temlate bit is 1. Hene, the FPGA onfiguration an be otimised to adat to the temlate harateristis. It is shown in [15] that dynami reonfigurability is well suited to adat to shae-adative image and video roessing algorithms if the reonfiguration overhead an be ket small. However, the examle used onsisted only of a limited number of ossible onfigurations sine only different objet shaes within an 8 8 blok were onsidered. 3 Shae-Adative Temlate Mathing The aim is to find a temlate objet of arbitrary shae and size within a searh image or video frame of any size using a reonfigurable omuting arhiteture, as shown in Figure 1. The searh image or frame onsists of W H ixels. The temlate objet onsisting of oaque ixels an have any shae. It is given by its bounding box of size w h, that is the smallest retangle surrounding the objet as shown in Figure 2. Within this bounding box, eah ixel ontains one mask bit whih is 1, if the ixel belongs to the objet, or otherwise, as defined in MG-4 [16]. h 1 w Figure 2 Temlate Mask T(i,j) Temlate objet and searh image The temlate is shifted through every ossible loation of the image that an ontain the entire temlate and omared to the resetive subimage of the same size. There are ( W w+ 1) ( H h+ 1) of those loations. Only ixels of the subimage that orresond to ixels belonging to the temlate objet are taken into aount. Simulations of various orrelation and histogram based mathing and image retrieval algorithms have been arried out in software to find a suitable and easily imlementable similarity measure. The sum of absolute distanes (SAD), arried out on luminane ixel values, was then hosen due to good mathing results and beause of its simle struture. For all ( W w+ 1) ( H h + 1) ossible ositions (y,x) of the temlate within the image, alulate h 1 SAD( yx, ) = ( Ii ( + y, j + x) Tij (, ) Mij (, )), (1) where I is the ixel value of the searh frame, and T and M are ixel value and mask bit of the temlate objet, resetively. For one frame, ( W w+ 1) ( H h+ 1) w h absolute distane alulations need to be omuted. A math is found at a osition (y,x) where SAD(y,x) is minimum and also smaller than a ertain threshold determined by the user. In this aer, only the alulation of the SAD values is onsidered. 4 Systoli Array for SA-TM y I(y,x) w 1 i = j = Sine in ratie video data are often streamed in a horizontal raster-san fashion (line after line), we assume that one ixel of the searh frame beomes available with eah lok yle [5]. As every ixel value of the searh image or video frame is read only one, it would be useful to erform in arallel all omutations where this ixel value is required, so that no inut data need to be stored. A simle examle of an SA-TM roess is shown in Figure 3. For various mathing ositions of the temlate on the searh image, the absolute differene (AD) omutations to be erformed for that osition and in whih lok yle, and the SAD these ADs ontribute to, are shown. In this examle, the ixel T(,1) does not belong to the objet, that is, it is transarent. Therefore, this ixel value does not ontribute to the omutations. x Image / Frame W H

4 Examle 1: Temlate: Mathing osition: w = 3, h = 3, =8 T(,) T(,1) transarent Searh image: W=7, H = 6 I(,) lok yle Comutations: 1. I(,) - T(,) 2. I(,1) - T(,) 3. I(,2) - T(,2) 4. I(,3) - T(,2) 8. I(1,) - T(1,) 9. I(1,1) - T(1,) I(2,2) - T(2,2) 18. I(2,3) - T(2,2) 5. I(,4) - T(,) 7. I(,6) - T(,2) 12. I(1,4) - T(1,) I(2,6) - T(2,2) 8. I(1,) - T(,) 1. I(1,2) - T(,2) 15. I(2,) - T(1,) I(3,2) - T(3,2) 26. I(3,4) - T(,) 28. I(3,6) - T(,2) 33. I(4,4) - T(1,) I(5,6) - T(2,2) -> SAD(,) -> SAD(,1) -> SAD(,4) -> SAD(1,) -> SAD(3,4) Figure 3 Examle 1 of SA Temlate Mathing Considering that ixel I(,) is available in the first lok yle, ixel I(,1) in the seond yle, and so on, if a line-byline raster-san fashion is used, some omutations an be arried out in arallel. In the first ste, only I(,) is available, and using T(,) the first AD ontributing to SAD(,) an be alulated. In the seond yle, I(,1) beomes available and the first AD for SAD(,1) is omuted using T(,). As T(,1) is transarent, no further omutation an be arried out in this yle. In yle 3, I(,2) is used to erform two omutations in arallel: I(,2) - T(,) whih ontributes to SAD(,2), and I(,2) - T(,2) whih yields the seond AD for SAD(,). In yle 4, when I(,3) beomes available, no ontribution to SAD(,) is alulated, as this image ixel is not neessary for the omutation of SAD(,). Not until yle W ( h 1) + w = 17 an all ossible eight ixels of the temlate objet be used in arallel... I(,2) I(,1) I(,), D,2 1, 1,1 1,2 2, 2,1 2,2 D D D D D D D D SAD(,) SAD(,1) Figure 4 Signal Flow Grah (SFG) for SA-TM Examle 1 Examle 1 an be reresented by the Signal Flow Grah (SFG) shown in Figure 4 whih is roosed as a general SFG for SA-TM. The nodes marked <i,j> reresent the absolute distane (AD) omutations I(y,x) - T(i,j). The ixel values I(y,x) are broadasted sequentially to all of those roessing elements (s), that is, all ossible AD omutations for a artiular image ixel are arried out in arallel. Note that some of those omutations (for examle, I(,) - T(,2) ) do not ontribute to any valid result. The AD results are added to the intermediate sums oming from the left and the new sum is registered and shifted out at the right of the. The delay nodes <D> are used as shift registers and are required at transarent ixels within the bounding box of the temlate objets and at all ( W w) ( h 1) laes where no AD ontribution for a artiular SAD result is rodued. For examle, the intermediate sum I(, ) - T(,) + I(,2) - T(,2) whih is omuted in the third yle in <,2> needs to be at the left inut of <1,> when I(1,) is broadasted to that node, that is in yle W + 1 = 8 when the first image or frame line has been omleted. Note that if the ixel values of the searh frame an be streamed in a vertial fashion (olumn after olumn), registers an often be saved as in reality video frames are more wide than high. In examle 1, the first valid result, SAD(,), is at the outut of node <2,2> after W ( h 1) + w = 17 yles, followed by SAD(,1) after the next yle. There will still be invalid results at the outut after eah temlate row, as ertain SAD ositions (for examle, SAD(,5), SAD(,6)) are not defined beause at that osition the temlate annot over the subimage. For instane, in examle 1, there are W Η = 42 ixels in the searh frame (and therefore 42 yles are required to rodue all results), but only 2 SAD values are rodued as there are only 2 ositions where the temlate fits omletely into the searh frame. In the 42nd (and last) yle, when the last image ixel of that frame I(5,6) is broadasted, the 2th result SAD(3,4) is omleted by adding I(5,6) - T(2,2) to the intermediate sum in <2,2>. Based on the SFG shown for examle 1 we roose the following generi systoli array adated to the shae of the temlate objet. Eah ixel belonging to the temlate objet is reresented by a where the AD omutation using the value of that ixel is erformed. The struture of a is shown in Figure 5.

5 . Sum_in m T(i,j) I(y,x) AD + D Sum_out n resulting in ( log ( 2 k ) ) = ( 2 k 1 k) + ( q + 1) q k = 2 q + 1 (8) = 2 q ( q 1) ( 2 q )( q + 1) (9) = ( q+ 1) 2 q (1) Figure 5 Struture of <i,j> for SA Temlate Mathing Hene, (5) leads to The word length of I(y,x) is, and usually =8. The ixel values of the temlate objet T(i,j) have the same word length as I(y,x). The values are onstant for a artiular and an be stored in ROM within the as shown, or ome from outside the. The word width of Sum_in and Sum_out deend on the osition of the in the SFG. For the first, Sum_in is, therefore m an be. Variable n needs to be at least the maximum of m and, in ase of the first that is. Further along the SFG, m and n will inrease, as more bits are required to reresent the intermediate sums. The maximum absolute distane (AD) for one is 2-1, the maximum intermediate sum of k of those ADs is ( 2 1) k for one, whih requires log 2 (( 2 1) k) bits to be fully reresented. The intermediate sum is then stored in a register. The area of a onsists of a onstant art for the AD module and a variable art whih grows with the word length n of the outut and an therefore be alulated as: A (n) = a n+ b = a log 2 (( 2 1) k) + b, (2) where a and b are onstants. The logi resoures required to imlement s an then be alulated as A = A ( k ) = ( a log 2 (( 2 1) k) + b). (3) To alulate A exliitly as a funtion of the following estimation is derived from (3), given that 2» 1 : A a ( + log 2 ( k) ) + b, (4) whih an be simlified further to A a ( log 2 ( k) ) + ( a+ b). (5) The sum term an be substituted as follows: ( log 2 ( k) ) = 2 q ( log 2 ( k) ) + ( log 2 ( k) ), (6) k = 2 q + 1 with q = log 2 ( ), (7) A a ( ( q+ 1) 2 q ) + ( a + b). (11) All ixels within the temlate bounding box but not belonging to the objet are registers used to delay the intermediate sums. s and registers are arranged aording to the shae of the temlate objet. If a ixel belonging to the objet is followed by a ixel not belonging to the objet in the same line, the reresenting the objet ixel is followed by a register, and vie versa. After eah line of s, additional W-w registers are required for all but the last line of the temlate to store the intermediate sums until a valid inut for that sum beomes available. The register area is omosed of the registers used for the wh - transarent ixels within the bounding box of the temlate (A Reg1 D ) and of the ( W w) ( h 1) registers used to delay the intermediate sums outside the temlate area (A Reg2 D ). Reg1 As the exat value for deends on the osition of the transarent ixels within the temlate bounding box, the average word length of the s is used to determine the average register size whih is then multilied by the number of transarent ixels: whih an be estimated as, (12) Reg1 wh ( ( q+ 1) 2 q ), (13) using the same value for q as in (7). Reg2 is estimated as resulting in Reg2 Reg1 wh log 2 (( 2 1) k) Reg2 h 1 ( W w) log 2 (( 2 1) k w), (14) ( W w) ( h 1) ( q h + log 2 ( w) ) 2 q h ( + 1), (15) with q h = log 2 ( h 1) + 1. (16)

6 To summarise, in an effiient systoli array solution for the resented SFG for SA-TM, the following oints need to be satisfied in order to searh for a w h temlate with ixels belonging to the objet of interest, in a W H searh frame: s are required to imlement the AD omutations and summation of intermediate results the s are arranged in the same way as the ixels belonging to the temlate objet; the wh - gas reresenting transarent ixels are filled with registers for horizontal raster-san, after eah, but the last, row of the array, W-w registers are added in the omutation flow to store the intermediate sums until the next searh frame ixel ontributing to a artiular SAD beomes available the kth in the omutation flow 1 k requires a log 2 (( 2 1) k) bit adder the size of eah register is the same as the outut size of the revious in the omutation flow 5 Reonfigurable design strategies for SA-TM The following design aroahes for a systoli array for SA Temlate Mathing an be distinguished. As all s have the same struture, aart from different word length, their funtion an be hanged to use a different number of ixel values and/or a different similarity measurement in the future. 5.1 Dynami design In this aroah, the devie is reonfigured for every ossible temlate size and shae, and for every ossible searh frame size. This generally inludes the re-omilation of the design ode, as there are an infinite number of solutions. As the temlate an be art of the onfiguration data and word lengths an be otimised, an effiient systoli array solution as desribed above an be ahieved. A as shown in Figure 5 an be used with the temlate ixel value stored in on-hi memory. As one inut of the AD omutation is onstant, the AD module an be substituted by a look-u table (LUT) storing the AD value for eah value of the inoming frame ixel I(y,x). In addition to the s, w h + ( W w) ( h 1) registers are required. The area for the dynami design onsists of the area used by the s (A D ) and the area required for the registers (A Reg1 D + A Reg2 D ): = Reg1 Reg2 + +, (17) and an be estimated using (11), (13) and (15), resetively. The total exeution time T D for the dynami design to find a temlate objet in N video frames onsists of the omutation time T D omuten to alulate the SA-TM oerations, the reonfiguration time T D reonf to udate the FPGA onfiguration for a new temlate, and the omilation time T D omile whih is required in most ases as the number of temlate objets and therefore the number of different devie onfigurations is unlimited: T D = T D omuten + T D reonf + T D omile. (18) The exeution time for N frames an be alulated as, (19) where f D is the lok frequeny the iruit an run at. As a systoli array is ielined by eah, f D is determined by the ritial ath through the slowest, if additional FPGA routing delays are negleted. The reonfiguration time and omilation time deend on the size of the iruit to be imlemented. The reonfiguration time is generally roortional to the number of resoures to be reonfigured, but also deends on the devie used and the reonfiguration strategy. Comilation time deends on the hardware and software used to translate and ma the ode and is hard to estimate. 5.2 Stati design Due to long reonfiguration and reomilation overheads, the dynami design aroah is exeted to be useful only if the a temlate objet is searhed for within a large number of video frames of the same size. As an alternative, a stati design is roosed, where the onfiguration of the FPGA is not hanged when a new temlate is used. As the number of different searh frame sizes and temlate shaes and sizes is unlimited, only a subset of all solutions an be imlemented. Sum_in m Figure 6 T(i,j) T D omuten I(y,x) AD N WH f D struture for stati design = If the size of the searh frame is fixed, the following solution is ossible. The s have to be modified, as shown in Figure 6, so that the temlate ixel values T(i,j) ome from external memory, and a multilexer ontrolled by a further inut signal sel needs to be added to selet between erforming an addition, if the resetive ixel belongs to the temlate objet, or otherwise just delaying the signal. All delay elements D have to be substituted with those modified s. By hanging the memory ontent, different temlates an be + sel 1 D Sum_out n

7 searhed for. The word length n of the kth outut is n = log 2 (( 2 1) k), with 1 k W H, in order to over for all ossible temlate sizes. The area A S for the stati design whih onsists of WH s is alulated using (2) as A S WH = A ( S k ) = ( a S log 2 (( 2 1) k) + b S ) WH and hene estimated aording to (11) as, (2) Sum_in m Figure 7 T(i,j) I(y,x) AD + M(i,j) struture for artially-dynami design 1 1 D Sum_out n A S a S q WH 2 q WH ( + 1) + ( a S + b S ) (21) with q WH = log 2 ( WH) + 1. The exeution time T S for the stati design to erform the SA-TM algorithms on N frames an be alulated as T S omuten = T S = ( N + 1) WH (22) Advantages of this design are that no reomilation of the design ode or reonfiguration of the devie are neessary for a onstant searh frame size as the temlate is stored off-hi. However, the large, external RAM to store all ossible WH temlate ixels and mask bits are exeted to make the design slower and more omlex than an otimal design. In addition, for large frame sizes the number of I/O ins required for all WH temlate ixels is extremely large. Another disadvantage is that in all ases when < WH (temlate smaller than searh frame), area is wasted beause some of the logi is unused. 5.3 Partially-dynami design To ombine the advantages of the dynami and the stati design, a third design is roosed. The differene to the stati design is that this design stores the temlate ixels and mask bits in on-hi memory available on most FPGAs. To hange the temlate, only a reonfiguration of the memory arts is neessary, the rest of the iruit remains the same. As the temlate is dynami while the iruit remains stati, this design is alled artially-dynami. The struture of a for this design is shown in Figure 7. Both temlate ixel value T(i,j) and mask bit M(i,j) for that ixel are stored within the. If M(i,j) is 1, that is, if the ixel belongs to the temlate objet, the absolute distane of I(y,x) and T(i,j) is added to the inoming intermediate sum sum_in. Otherwise, sum_in is shifted through and registered. As the artially-dynami design, like the stati one, onsists of WH s, the area an be estimated in the same way: f S The total exeution time T PD for the artially-dynami design to find a temlate objet in N video frames onsists of the time T PD omuten to alulate the SA-TM oerations and the reonfiguration time T PD reonf to udate the FPGA memory for a new temlate. The exeution time for N frames is alulated as omuten T PD with f PD as the maximum lok rate the iruit an run at. Considering a artially reonfigurable design is used, the reonfiguration time to load the data for a new temlate into memory an be alulated as reonf T PD = ( N + 1) WH , (24) f PD = ( + W H) t bit, (25) where t bit is the time to reonfigure 1 bit. Note that only the ixel values belonging to the objet need to be loaded into memory in addition to mask bits for all WH s, while any ossible ixel values belonging to an old objet but not belonging to the new temlate objet do not need to be hanged as their use will be disabled by the new mask bits. 6 FPGA Imlementation and Results The s for the three reonfigurable designs desribed above have been imlemented for =8 (eight bits er ixel) targeting Xilinx Virtex XCV1E devies. n [LCs] f D [MHz] A S [LCs] f S [MHz] A PD [LCs] f PD [MHz] A PD a PD q WH 2 q WH ( + 1) + ( a PD + b PD ) using q WH as in the stati design., (23) Table 1 Number of LCs (A) and lok frequeny (f) on Xilinx Virtex 1E, for s with different word length n, for dynami (D), stati (S) and artially-dynami (PD) design

8 Table 1 shows the results for area in logi ells (LCs) and lok frequeny in MHz for s of different word length n for the dynami (D), stati (S), and artially-dynami (PD) designs. An LC ontains a look-u table with four inuts (4- LUT) and one fli-flo (FF). Using the results shown in Table 1, the values a and b for the area alulation of an, as in (2), and hene the area estimation for all s, an be determined for all three designs: dynami design: a D = 1.5, b D = 1, hene ( n) = 1.5 n + 1, and 1.5 q 2 q ( + 1) + 13, (26) with = log 2 ( ) + 1. q stati design: a S = 1.5, b S = 18, hene A S ( n) = 1.5 n + 18, and A S 1.5 q WH 2 q WH ( + 1) + 3, (27) with = log 2 ( WH) + 1. q WH artially-dynami design: a PD = 1.5, b PD = 5, hene ( ) = 1.5 n + 5, and A PD 1.5 q WH 2 q WH ( + 1) (28) 6.1 Results for small examle A PD n For Examle 1, onsidered in Setion 4 (w = h = 3, = 8, W = 7, H = 6), the results are shown in Table 2. Design D S PD A measured [LCs] A alulated [LCs] f [Mhz) T omute [ns] for 1 lok yle # yles for N frames 42N 42N+41 42N+41 T reonf [ms] n/a 55.1 Table 2 Results for Examle 1, for dynami (D), stati (S) and artially-dynami design (PD) It an be seen from rows 1 and 2 in Table 2 that the alulated areas using the area estimations desribed in Setion 5 are fairly aurate and are used in further examles, where omilation takes too muh time or is imossible due to the size of temlate and/or searh frame. The reonfiguration times to load the iruit for the dynami design and to udate the memory of the dynami design were alulated for a Virtex 1E devie using a reonfiguration lok frequeny of 5 MHz and 8 bits er lok yle, as desribed in [17]. 6.2 Results for HDTV format For a more ratial examle, the following arameters are used, as in the HDTV (SMPTE 26M) video roessing format: W = 192, H = 18, frame rate 3 Hz. Area and exeution time estimations were alulated aording to the equations given in Setion 5. However, in most ases the designs are too omlex to fit into the largest urrently available FPGAs, suh as Xilinx Virtex XC2V1. Table 3 shows the number of logi ells required to imlement the dynami design for different temlate sizes, searated into area and resoures to register intermediate results, aording to (11), (13), and (15), resetively. The bounding boxes are onsidered quadrati with 8% of the ixels belonging to the objet to be maed. It an be seen that the number of LCs required grows with the size of the temlate objet. For relatively small temlates, the largest share of resoures are required for the registers outside the temlate bounding box neessary to delay intermediate results until valid inut signals beome available. While for a 1 1 temlate only.7% of the area is used for s, the rest for registers, this share grows to 28.7% for a 5 5 temlate and to 86.% for a HDTV format size temlate with 8% oaque ixels. w h ,69 7,714 55,93 247,714 1,86,85 7,66,789 Reg ,232 8,988 39, ,88 1,234,464 Reg2 246,39 59,9 1,714,79 3,732,82 7,776,12 17,697,46 248, ,846 1,779,78 4,2,486 7,951,928 26,538,713 Table 3 Area results [in LCs] for dynami design using different temlate sizes and =8% of wh However, for all ossible temlate objet sizes and shaes, the area for the dynami design is at least 34% smaller than the area required for the stati design and at least 16% smaller than the area of the artially-dynami design, whih are both onstant for all temlates used and equivalent to the logi resoures of about 1, and 8 Xilinx Virtex XC2V1 devies, resetively. Figure 8 shows the total area required to imlement the dynami design for different temlates with the same number of ixels belonging to the objet (=32, = 8% of wh), but of different shaes, that is different roortions of w to h. It an be seen that the smaller the quotient w/h, the larger the number of logi ells required to imlement the design. This is due to the fat that temlates with a small width, but a large height require far more registers to store intermediate results. In fat, and REG1 are onstant for all ases. Note that this effet would be reversed if instead of a horizontal raster-san a vertial raster-san fashion ould be used to stream in the video frame ixels.

9 Number of Logi Cells Figure 8 Area [in LCs] for dynami design using temlate objets with different shae for =32, = 8% of wh Figure 9 shows the total exeution times T required to erform an SA-TM oeration on HDTV format video for a 1 1 size temlate and for a 1 1 temlate, deending on the number of onseutive frames N, for all three reonfigurable designs. Inluded are the omutation time and the reonfiguration time, where aliable. There is only one grah shown for the stati design (dotted) as the exeution time does not deend on the temlate objet used. For the dynami designs (full lines) and the artially-dynami designs (dashed lines), the thin grahs show the results for the smaller temlate whereas the results for the larger temlate are shown as thiker lines. Exeution Time T [ms] Area Requirement for HDTV (w*h=4, =8%) Temlate shae (w/h) Time Requirement for HDTV T_D_1x1 T_D_1x1 T_PD_1x1 T_PD_1x1 T_S Number of Frames N Figure 9 Exeution time T in ms using a 1 1 ixel temlate and a 1 1 ixel temlate (=8%), for dynami (D), stati (S) and artially-dynami design (PD) While the exeution time for the stati design is shortest if the algorithm is erformed on only one frame as there is no reonfiguration overhead, it inreases more steely due to a lower maximum lok rate, and after about 35 frames the stati design will be slower than both dynami and artially-dynami designs with either temlate. Another notieable fat is that the exeution times for the artially-dynami design for both temlates are not very different due to similar reonfiguration overheads (quotient of reonfiguration time over omutation time), while the reonfiguration times for the dynami design deend muh more on the size of the temlate objet. For the 1 1 temlate, the reonfiguration overhead is very small, and hene the dynami design is the fastest for any number of N>1 and even allows real-time roessing, while the time required to reonfigure the devie for the 1 1 temlate is signifiantly larger, resulting in the dynami design being the slowest if less than 33 frames are oerated on. The reonfiguration overheads (in %), as a funtion of number of frames N are diagrammed for the dynami (full lines) and artially-dynami (dashed lines) designs for both 1 1 temlate (thin line) and 1 1 temlate (thik line). Not inluded in the results are additional overheads due to ossible reomiling the dynami design for a new temlate. Reonfiguration Overhead [%] Results for HDTV (=8%) T_D_1x1 T_D_1x1 T_PD_1x1 T_PD_1x Number of Frames N Figure 1 Reonfiguration overhead in %, using a 1 1 temlate and a 1 1 temlate (=8%), for dynami (D) and artially-dynami design (PD) For omarison, the SA-TM algorithm was also erformed on a 1.4 GHz Pentium 4 PC with 384 MB RAM, for both temlates on a HDTV format image (= 1 frame). We measure the following exeution times T PC, and estimate the resulting seed-us S of the dynami (D), artially-dynami (PD), and stati design (S), running at their resetive maximum lok seeds (omare Table 1) and inluding reonfiguration time, where aliable, on a Virtex 1E FPGA, for 3 onseutive frames (= 1 seonds), as shown in Table 4. w h T PC S D S PD S S se ,98 se 28,6 17,6 11,8 Table 4 Exeution time T PC for omuting SA-TM on 1 frame on 1.4 GHz Pentium 4 PC, and Seed-us S of dynami (D), artially-dynami (PD) and stati (S) designs, for different temlates mathed on 3 onseutive frames in HDTV format It an be shown that signifiant seed-us an be ahieved by using reonfigurable omuting to erform SA-TM, eseially for large temlates, sine many oerations an be omuted in arallel.

10 7 Conlusion In this aer, a shae-adative temlate mathing (SA- TM) method to retrieve arbitrarily shaed objets within images or video frames was roosed. Different strategies of reonfigurable omuting regarding their suitability for imlementing the SA-TM method as an examle of a tyial multimedia algorithm were investigated. A dynami design, where the onfiguration data is omletely adated to the shae and size of the temlate objet used, was resented in this aer. A stati design whih an math temlates, stored in off-hi memory, of all ossible shaes and sizes within a video frame using the same FPGA onfiguration was also resented. In addition, a artially-dynami design whih uses on-hi memory, available in most urrent FPGAs, to store the temlate was introdued. A generi systoli array arhiteture rovided the basis for the imlementation of the three reonfigurable designs. Formulae have been develoed to exliitly estimate the logi resoures required to imlement the designs deending on the size of the searh frame and the size and shae of the temlate objet. While the number of logi ells to imlement the stati design and the artially-dynami design is onstant for a ertain frame size, the dynami design is adated to the temlate objet leading to signifiant savings in area, eseially if relatively small temlates are used. The suitability of a artiular design regarding total exeution time of the algorithm strongly deends on the number of onseutive frames the oerations are arried out on with the same temlate. Sine the stati design does not suffer from reonfiguration overheads, it is most suitable for an oeration on one or only a few frames. However, as the artially-dynami design and eseially the dynami design an oerate at higher lok frequenies, they erform better if the mathing algorithm is exeuted on a large number of frames. For the examle of mathing a 1 1 temlate on an HDTV format frame, the stati design is fastest for an oeration on u to 33 frames, at whih oint the artially-dynami design beomes faster. After 38 frames the dynami design, whih has the highest reonfiguration time for that temlate size, will be the fastest design. Real-time SA-TM of HDTV video is ossible using the dynami design for temlates with u to 128 ixels. Comared to a software imlementation, all three reonfigurable SA-TM designs an ahieve signifiant seed-us, eseially if large temlates are used. Referenes [1] MG Grou, Overview of the MG-4 Standard, ISO/IEC JTC1/SC29/WG11 N43, Marh 21. [2] MG Grou, Overview of the MG-7 Standard, ISO/IEC JTC1/SC29/WG11 N431, Marh 21. [3] P. Pirsh, C. Reuter, J.P. Wittenburg, M.B. Kulazewski, H.-J. Stolberg, Arhiteture Conets for Multimedia Signal Proessing, Journal of VLSI Signal Proessing, vol. 29, no. 3, , November 21. [4] L. Torres, M. Kunt, F. Pereira, Seond Generation Video Coding Shemes and Their Role in MG-4, Euroean Conf. on Multimedia Aliations, Servies, and Tehniques, May 1996, [5] S. D. Haynes, J. Stone, P. Y. K. Cheung, W. Luk, Video Image Proessing with the SONIC Arhiteture, IEEE Comuter, vol. 33, no. 4,. 5-57, Aril 2. [6] Y. A. Aslandogan, and C. T. Yu, Tehniques and Systems for Image and Video Retrieval, IEEE Trans. Knowledge and Data Engineering, vol. 11, no. 1, , January/February [7] T. Komarek, P. Pirsh, Array Arhitetures for Blok Mathing Algorithms, IEEE Trans. on Ciruits and Systems, vol. 36, no. 1, Otober [8] J. Villasenor, B. Huthings, The Flexibility of Configurable Comuting, IEEE Signal Proessing Magazine, , Set [9] G. MGregor, and P. Lysaght, Self Controlling Dynami Reonfiguration, in Pro. 9th International Worksho on Field-Programmable Logi and Aliations, FPL 99, 1999, [1] E. Sanhez, M. Sier, J.-O. Haenni, J.-L. Beuhat, A. Stauffer, and A. Perez-Uribe, Stati and Dynami Configurable Systems, IEEE Trans. on Comuters, vol. 48, no. 6, , June [11] P. Lysaght, Asets of Dynamially Reonfigurable Logi, in IEE Colloquium on Reonfigurable Systems, Glasgow, Sotland,. 1-5, Marh [12] J. O. Cadenas, G. M. Megson and T. P. Plaks Quantitative Evaluation of Three Reonfiguration Strategies on FPGAs: A Case Study, in HPC-Asia 2. Pro. of the Fourth Int. Conf. on High-Performane Comuting in the Asia-Paifi Region, May 2, Beijing, China, [13] N. K. Ratha, A. K. Jain, Comuter Vision Algorithms on Reonfigurable Logi Arrays, IEEE Trans. on Parallel and Distributed Systems, vol. 1, no. 1, , January [14] J. Villasenor, B. Shoner, K.-N. Chia, C. Zaata, H. J. Kim, C. Jones, S. Lansing, B. Mangione-Smith, Configurable Comuting Solutions for Automati Target Reognition, in Pro. FCCM,. 7-79, [15] J. Gause, P. Y. K. Cheung, W. Luk, Stati and Dynami Reonfigurable Designs of a 2D Shae-Adative DCT, in Pro. FPL, 2, [16] MG Grou, MG-4 Video Verifiation Model Version 13., ISO/IEC JTC1/SC29/WG11 N2687, Marh [17] Xilinx In., Virtex Series Configuration Arhiteture User Guide, Aliation Note XAPP151, Setember 2.