MOTION PANORAMA CONSTRUCTION FROM STREAMING VIDEO FOR POWER- CONSTRAINED MOBILE MULTIMEDIA ENVIRONMENTS XUNYU PAN

Transcription

1 MOTION PANORAMA CONSTRUCTION FROM STREAMING VIDEO FOR POWER- CONSTRAINED MOBILE MULTIMEDIA ENVIRONMENTS by XUNYU PAN (Under the Drecton of Suchendra M. Bhandarkar) ABSTRACT In modern tmes, more and more multmeda applcatons are mplemented on wreless computer networks and used to entertan users through moble devces. In power-constraned envronments such as pocket PCs, PDAs, and cellular telephones, the large amount of vdeo nformaton transmtted from the server-end to the user-end s often compressed to reduce the power and band wdth consumpton. Ths thess ntroduces an effcent method for the constructon of moton panoramas and panoramc vdeos from streamng vdeo. The technque nvolves the extracton of moton components from the background mosac whch s generated by a hybrd algorthm that combnes both feature-based methods and drect methods. Expermental results show ths heurstc approach reduces the sze of the vdeo nformaton transmtted and summarzes the entre contents of the moton vdeo for the moble end users. INDEX WORDS: Moton Panorama, Panoramc Vdeo, Image Mosacs, Moton Components Extracton, Multmeda Applcatons, Computer Networks, Power-constraned Envronments

2 MOTION PANORAMA CONSTRUCTION FROM STREAMING VIDEO FOR POWER- CONSTRAINED MOBILE MULTIMEDIA ENVIRONMENTS by XUNYU PAN B.S., Nanjng Unversty, Chna, 2000 A Thess Submtted to the Graduate Faculty of The Unversty of Georga n Partal Fulfllment of the Requrements for the Degree MASTER OF SCIENCE ATHENS, GEORGIA 2004

3 2004 Xunyu Pan All Rghts Reserved

4 MOTION PANORAMA CONSTRUCTION FROM STREAMING VIDEO FOR POWER- CONSTRAINED MOBILE MULTIMEDIA ENVIRONMENTS by XUNYU PAN Major Professor: Commttee: Suchendra M. Bhandarkar Walter D. Potter Khaled Rasheed Electronc Verson Approved: Maureen Grasso Dean of the Graduate School The Unversty of Georga August 2004

5 ACKNOWLEDGEMENTS I would lke to express my apprecaton to my advsor Dr. Suchendra M. Bhandarkar, for hs patent gudance and nstructons at every step from the preparaton of the research to the draft of ths thess. I would also lke to thank Dr. Walter D. Potter and Dr. Khaled Rasheed for ther tme and support as members of my commttee. My parents support me all the tme. I thnk I would not complete ths wthout ther love and encouragement. v

6 TABLE OF CONTENTS Page ACKNOWLEDGEMENTS... v CHAPTER 1 INTRODUCTION OVERVIEW OF THE PROJECT STATIC BACKGROUND GENERATION The Detecton of Interest Ponts Pont-to-pont Correspondences Computaton of Homography Image Blendng Background Mosac Generaton FOREGROUND AND BACKGROUND SEGMENTATION Dynamc Foreground and Statc Background Mahalanobs Dstance Probablty Image False Moton Detecton Segmentaton of Dynamc Foreground Connected Components Detecton Sze Flterng Moton Components Extracton...35 v

7 v 5 MOTION PANORAMA CONSTRUCTION Network Transmsson Moton Panorama Constructon at the User-end EXPERIMENTAL RESULTS CONCLUSIONS AND FUTURE DIRECTIONS Concludng Remarks The Orgnal Contrbutons of Ths Thess Future Work and Drectons...51 BIBLIOGRAPHY...52

8 CHAPTER 1 INTRODUCTION Transmsson of vdeo streams of large sze s always a bottleneck n multmeda applcatons on computer networks. The requrement of effcent vdeo transmsson s a key factor n mprovng overall system performance, especally n the power-constraned multmeda envronments conssts of moble devces such as PDAs, pocket PCs, and cellular telephones. Automatc constructon of large and hgh qualty mage mosacs s an actve research area n the felds of computer vson, mage processng, and artfcal ntellgence. Effcent methods for mosac generaton can be wdely used n networked or moble applcatons wth the expanded requrement of transmsson, storage and manpulaton of multmeda nformaton. The problem of acqurng panoramc mages can be solved manly n two ways, namely: Usng wde feld of vew lenses and magng devces. Mosac constructon technques. Wde feld of vew lenses or magng devces can be used to capture the whole scene of the vdeo sequences, such as Columba s OmnCam [1, 2]. One shortcomng of ths technque s that panoramc mages acqured are of low mage qualty because of the mappng of the entre scene nto a fxed resoluton vdeo camera. The dstorton n the shape of the objects n the scene s another problem ntroduced by ths method. Mosac constructon technque, whch s also called panorama constructon, s another approach to dsplay the entre scene. Ths s an effcent and convenent representaton of the 1

9 2 moton vdeo by sttchng the ndvdual frames nto a unque wde-angle panoramc mage. It does not requre any specal magng devces or hardware. The fnal panoramc mage, whch covers the entre scene, does not lose any mage qualty ether. Former works on panoramc mosacs can be dvded nto two major categores: Mosac constructon from statc scenes. Mosac constructon from dynamc scenes under two stuatons: - Dynamc scenes captured wth a statc camera. - Dynamc Scenes captured wth a movng camera. Statc-scene based mosac constructon deals wth the stuaton where the vdeo sequences have statc foreground and background. In another words, no obvous moton object s ncluded n the vdeo sequences. A number of papers, e.g. [3], concentrate on ths case. Fgure 1.1 and fgure 1.2 gve an example of a panoramc mosac constructed from a statc scene. Fgure 1.1 shows several orgnal frames extracted from a vdeo captured n the Vsual and Parallel Computng Laboratory (VPCL). The fnal panoramc mosac s llustrated n fgure 1.2. Ths panoramc mosac s generated usng a subset of the algorthm mplemented n ths thess. Other researchers have explored effcent methods to represent the dynamc scenes stuaton where the vdeo sequences of movng objects captured by a statc or a movng camera are analyzed. In ths case, the scenes are dynamc contanng moton or deformaton wthn. Many real-lfe vdeo sequences are nstances of ths stuaton. The mosacs of dynamc scenes captured by a statc camera have been studed for many years and several approaches have been developed [4, 5, 6]. The man dea s to segment the frame nto two parts or two layers: foreground and background or dynamc layer and statc layer. The movng objects can be extracted by pxel-to-pxel comparson between the pre-stored

10 3 background and the current frame. These methods work satsfactorly only when the background nformaton s already avalable whch means that the scene should be sampled frst. Fgure 1.1: The extracted orgnal frames from vdeo sequence Fgure 1.2: The panoramc mosac constructed from the orgnal frames

11 4 To create panoramc mosacs for dynamc scenes captured by a movng camera, [7] [8], and [9] have presented several effectve methods. In [9] the authors use the blockng moton detecton technque to compute a moton vector feld whch s then clustered to fnd the domnant moton regons. In [8], the authors propose a drect method to acqure the moton parameters, algn the frames usng these parameters, and locate the frame regons whch do not observe the moton parameters. The last case, namely, mosac constructon for dynamc scenes captured by a movng camera, s the most popular stuaton n real lfe. In general, two major methods can be appled to ths category of problems: feature-based methods [10] and drect methods [11]. The former treats pars of nterest ponts as features and uses the correspondences of these features n vdeo sequences to estmate the homography between frames. The latter algns the frame ntensty values to acqure the best mappng between frames. Feature-based methods were ntroduced by P. H. S. Torr and A. Zsserman n 2000 [10]. These methods nvolve a strategy for the ntal estmaton of frame matchng whch s also called nterframe homography based on the detecton of pont features. In other words, the recovery of the entre scene should be acheved by frst extractng the features, and then usng these features to compute the relatons or homographes between the frames. The feature-based methods can also be combned wth outler-rejecton technques such as Random Sample Consensus (RANSAC) [12]. Snce the combned methods can estmate the frame matchng correspondng to the moton of the camera whle rejectng the movng parts of the frame whch correspond to a dfferent moton, the technque s qute robust to many real-lfe stuatons. However, many algnment problems are caused by the cases where the detected features are not homogeneously dstrbuted across the frames.

12 5 Drect methods [11] deal wth problems of camera moton and correspondence of every pxel smultaneously. The moton estmaton s obtaned by ths class of methods usng measurable nformaton such as brghtness varants or mage cross-correlaton measures. It fnds the mappng relatons between frames by mnmzng the dscrepancy between every pxel value n the frame. Ths category of methods s n contrast to the feature-based methods that rely on the correspondence of a sparse set of hghly relable mage features. Snce the nformaton of every pxel n the frame s used to estmate frame matchng whch corresponds to the moton of the camera, the drect methods have better performance n terms of the fnal mosac qualty. Both feature-based methods and drect methods contrbute to the estmaton of moton parameters of the camera between frames. These parameters are essental for the algnment procedure. In partcular, feature based methods are more robust n many real-lfe stuatons where several motons dfferent from the camera moton, namely outlers, are present n frames. On the other hand, drect methods provde more accurate frame algnment by takng nto account every pxel n the frames. Consderng the complementary characters of these two categores of methods, the combned technque that ncludes both of them s obvously appealng. In ths thess, a combned approach based on both feature-based methods and drect methods s proposed. In practce, a feature-based method s mplemented for the statc background generaton whle the drect method s used to segment the dynamc foreground. Some orgnal contrbutons for amendng the drawbacks of these two categores of methods are also provded. The statc background generaton and dynamc foreground extracton are performed at the server end. Durng the last phase, the statc background and the dynamc foregrounds along wth the assocated nformaton of ther relatve locatons n the fnal moton panorama are transmtted to the user-end. At the user-end, the

13 6 dynamc foregrounds correspondng to each frame n the vdeo sequence are pasted back onto the statc background. Fnally, a moton panorama or a panoramc vdeo s constructed under userspecfed requrements. Expermental results show that the sze of the nformaton transmtted s, on average, from 1/8 to 1/10 of the orgnal moton vdeo. The savngs n computaton tme and memory storage at the user-end are very useful and effcent n power-constraned multmeda envronments.

14 CHAPTER 2 OVERVIEW OF THE PROJECT The method of moton panorama constructon proposed n ths thess s descrbed n three major phases: statc background generaton, background/foreground (movng objects) segmentaton, and moton panorama constructon. The frst two phases are performed at the server-end. The last phase s performed at the user-end. The frst phase s statc background generaton. Based on the vdeo sequence extracted from the orgnal moton vdeo, the homographes correspondng to the moton of the cameras are computed for each frame. The statc background of the entre scene expressed n the vdeo sequence s generated by sttchng the ndvdual frames nto a large wde-angle panoramc mage usng the homographes. The dynamc foreground whch ncludes regons of both movng objects and false detectons exstng n the scene s segmented by warpng together three consecutve frames n the vdeo sequence and consequently detectng the ntensty dscrepancy at each pxel. The dynamc foreground s smoothed of nose usng a Gaussan flter and then fltered of false moton usng a sze flter to generate the components of real movng objects. These components are small n sze compared to the orgnal frames and hence convenent for network transmsson. After the statc background, foreground objects and ther locaton nformaton are receved at the user-end, the foreground objects are pasted back to the statc background usng ther locaton nformaton such as homographes and poston coordnates whch were computed at the server- 7

15 8 end. The fnal output can be constructed n the form of a moton panorama or a panoramc vdeo determned by the user requrements. From the perspectve of practcal applcaton, the whole procedure of moton panorama generaton at the server-end, transmsson through the moble network and constructon at the user-end s llustrated n fgure 2.1.

16 9 Background and Components of Foreground Request Mosacng, Segmentaton and Extracton Vdeo Frame Sequence Panorama Informaton Network Transmsson Vdeo Lbrary Request Orgnal Vdeo Server-end Fgure 2.1: The procedure for moton panorama generaton, transmsson and constructon. Panorama Vdeo Constructon User-end

17 CHAPTER 3 STATIC BACKGROUND GENERATION 3.1 The Detecton of Interest Ponts Knowng the correspondng ponts between frames enables one to estmate the mathematcal expressons for the geometrc transformatons of frames caused by the moton of the camera such as pan and tlt. The same moton usually holds for most of the pxels n the frames except for these assocated wth the movng objects. If all possble correspondng ponts are scanned, the computatonal complexty usually s very expensve. The process can be smplfed by examnng only the smaller number of ponts called nterest ponts. Interest ponts have some local property. For example, the corners of the objects are good examples of nterest ponts. Interest ponts can be detected by a corner detector. Instead of usng the Harrs corner detector [13] used n feature based regstraton, the Moravec corner detector [14] s mplemented n ths thess. The reasons for usng ths detector are: The detector s effectve. Based on the auto-correlaton functon, t captures the ntensty change around a pont. A pont s detected as an nterest pont f the change s bg enough. Ths property s helpful to a subsequent cross-correlaton matchng algorthm whch can fnd the correspondences for the current nterest ponts. The detector s smple and computatonally nexpensve. The Harrs corner detector, another wdely used corner detector, calculates the egen values whch usually nvolves 10

18 11 complex matrx computaton. In comparson, the Moravec corner detector s computatonally more effcent. The Moravec corner detector works n the followng manner: (1) The nterest value of each pxel n the frame can be calculated by the followng equaton + 1 j+ 1 1 MO (, j) = f ( k, l) f (, j) (1) 8 k = 1 l= j 1 where (, j) and ( k, l) are the coordnates of the pxels n the frame. In the current mplementaton, 7 7 wndows are used to calculate the nterest values of every pxel n the frame. (2) A threshold should be set to flter out the ponts wth relatvely small nterest value. Only ponts wth large enough nterest values can be treated as nterest ponts. In practce, a value of E + 3σ s used as the threshold, where E andσ are the mean and standard devaton of the nterest values of all the pxels n a frame. (3) To solve the problem of detected nterest ponts that are not homogeneously dstrbuted across the frames, an amended method s adopted. Each frame to be processed s dvded nto a number of neghbored and non-overlappng wndows. For each wndow, the pxel wth maxmum nterest value s extracted as the nterest pont of ths regon. Another ssue that needs to be mentoned here s that the nterest ponts extracted by the above method are also called nterest features or pont features [10]. Snce ths category of ponts consttute the basc regstraton nformaton for frame algnment and the subsequent frame mosac generaton, the so approaches to generate panoramc mosacs based on these

19 12 pont features are called feature-based methods. Fgure 3.1 shows the nterest ponts detected by the Moravec corner detector. Fgure 3.1: Interest ponts (features) detected by the Moravec corner detector 3.2 Pont-to-pont Correspondences The nterest ponts extracted from the frames by the corner detector are then tracked over the vdeo sequence n order to establsh the pont-to-pont correspondences. Template matchng [15] s one of wdely used method to detect nstances of a template n an mage frame. Gven a template t [, j], n order to detect ts nstances n a frame f [, j], an obvous method s to place the template n the frame and compare the ntensty values n the template wth the

20 13 correspondng values n the frame. In many cases, the ntensty value wll not match exactly. Hence, the sum of squared errors s the most popular matchng measure. Cross-correlaton s an operaton that can be used to acheve template-matchng. Gven the nterest ponts extracted from the orgnal frames, the pont-to-pont correspondences of these nterest ponts are matched usng proxmty and smlarty of the ntensty value n ther neghborhood. The ntensty values of all neghbors of each nterest pont are used to rank possble matches by computng a normalzed cross-correlaton. For an m n template t [, j], the match measure M can be computed usng C ft [, j] t[ k, l] f [ + k, j + l] (2) = m n k = 1 l= 1 C ft[, j] M[, j] = m (3) n 2 1/ 2 { f [ + k, j + k]} k = 1 l= 1 In the experment, ths method s mplemented by usng a template n the current frame wth the nterest pont at the top left corner. Then a regon n the next frame s consdered as the search area. By movng the template wndow column by column n the search area, the local maxma where the matchng pont s the putatve correspondence of the nterest pont under can be found. The pont-to-pont correspondences between two consecutve frames n the vdeo sequence are llustrated n fgure 3.2.

21 14 Fgure 3.2: The putatve pont-to-pont correspondences between two consecutve frames The word putatve ndcates that the correspondences detected by the cross-correlaton operaton are not necessary the real correspondences. It has been reported that more than 40% of the putatve correspondences obtaned by the best cross-correlaton score and proxmty are ncorrect [10]. Hence robust estmaton methods, such as RANSAC whch wll be descrbed and appled later, are an essental part of the whole procedure of statc background generaton. 3.3 Computaton of Homography Intal Homography estmaton In real lfe, people usually use the pn-hole camera to capture the world. Ths camera model projects the 3-dmensonal world onto a 2-dmensonal mage plane. Let each mage to be consdered to le n a projectve plane 2 Ρ. Gven a set of nterest ponts x n 2 Ρ and a

22 15 correspondng set of ponts x ' 2 lkewse nρ, the 2-dmensonal homography s the projectve x transformaton that maps x = (,, ) onto x ' = (, ', ' x w ) T. In practce, x and x' are y w T ' y ponts n two dstnct frames. For a set of pont correspondences x x ', the problem can be descrbed as beng requred to compute a 3 3 homography matrx H for each such that H x = x ' (4) The above equaton nvolves homogeneous vectors and hence the 3-vectors x ' and H x are not equal. They have the same drecton but may dffer n magntude by a non-zero scale factor. The equaton can be expressed by vector cross product as x ' H x = 0. If the j-th row of matrx H s denoted by h jt, then a smple lnear soluton for H can be derved as follows: 1T h x 2T H x =. h x 3T h x ' x ' y ' w T Suppose x ' = (,, ). The cross product may then be gven as ' 3T ' 2T yh x wh x ' 1T ' 3T x ' H x = wh x x h x. ' 2T ' 1T xh x yh x

23 16 Snce h x = x T h for j = 1, 2, 3, ths gves a set of three equatons n the entres of H, whch may be wrtten n the form T j j 0 0 x x 0 x x T T ' T ' T ' T T ' T ' T ' T = h h h x x x y x w y w. (5) These equatons all have the form A h = 0, where A s a 3 9 matrx, and h s a 9 vector made up of the entres of matrx H. Although there are three equatons n (5), only two of them are lnearly ndependent. In other words, each pont-to-pont correspondence gves two equatons n the entres of H. The thrd equaton s usually omtted n computng H [16]. Then the set of equatons becomes 0 x 0 x x T ' T T ' T ' T ' T = h h h x x w y w. (6) The set of equatons (6) holds for all ponts expressed n homogeneous coordnates x = (,, ) T, where = 1, and (, ) are the coordnates of the pont n the mage. ' ' x ' y ' w ' w ' x ' y Gven n correspondng ponts, 2 n such equatons can be obtaned. A set of four pont correspondences yelds a set of eght equatons whch can be wrtten as: Ah = 0,

24 17 where A s the matrx of coeffcents bult from the matrx rows A contrbuted by each pont-topont correspondence, and h s the vector of unknown entres of H. So four pont correspondences are the mnmum number needed to solve the problem. In general, gven n 4 pont correspondences {x x ' }, the homography matrx H such that H x = x ' can be computed by the Drect Lnear Transformaton (DLT) Algorthm [17] descrbed by the followng steps: (1) For each correspondence {x x ' } compute the matrx A usng equaton (6). (2) Generate a sngle 2n 9 matrx A from the n 2 9 matrces A. (3) Compute the Sngular Value Decomposton (SVD) of A [18]. The unt sngular vector correspondng to the smallest sngular value s the soluton of h. In detal, f A = T UDV wth D a dagonal matrx wth postve dagonal entres, arranged n descendng order down the dagonal, then h s the last column. (4) The matrx H s determned from h as followng: 1 h h h h h =, H =. h h h h h h h h Robust Estmaton For a set of correspondences {x ' x } obtaned by the cross-correlaton algorthm, the assumpton up to now s that the only error s n the measurement of the pont s poston, whch follows a Gaussan dstrbuton. However, n practcal stuatons, two other categores of

25 18 msmatched correspondences, also called outlers, exst. One category represents the spurous correspondences caused by mscalculaton n some specal cases. The other conssts of the pont matches correspondng to movng objects n the scene and not to the moton of the camera. The outlers can severely dsturb the estmated homography, and hence should be dentfed. In reallfe applcatons, robust estmaton can deal wth the stuaton where less than 50% of the ponts n the frame are outlners. Robust estmaton s an essental part of homography computaton process. One popular robust estmaton technque called the Random Sample Consensus (RANSAC) [12] s used n ths thess. Unlke the classcal technques for parameter estmaton such as leastsquares that only average the measurement errors, RANSAC has a heurstc mechansm for detectng and rejectng gross errors caused by outlers. For the correspondences detecton problem, the faulty measurement of a pont s poston s a measurement error and follows a Gaussan dstrbuton. Ths category of errors can be averaged out by classcal least-squares technques. The other two categores of msmatched correspondence, namely spurous correspondence and pont matches correspondng to movng objects, are gross errors and can only be fltered out by the RANSAC technque. The mplementaton of the RANSAC technque n ths project s descrbed by the followng steps: (1) Randomly select 4 correspondences whch may nclude both the correct ones and the msmatched ones to compute a homography H. Ths step consttutes an ntal homography computaton whch has already been descrbed prevously n detal n the chapter

26 19 (2) Compute the Eucldean dstance for every correspondence {x x }usng the followng functon: ' d 2 ' ( x, Hx ). (7) (3) Compute the number of nlers whose Eucldean dstance s less than a threshold D. These nlers consttute a consensus set S. (4) If the sze of S s larger than a threshold T, re-compute H from the nlers n S and termnate. (5) If the sze of S s smaller than T, repeat the above steps from (1)-(4) for N samples. (6) After N samples, recompute H from the consensus set wth the largest number of nlers. Several parameters need to be determned here: The samplng number N. If one chooses to try all possble samples, then N = C, where n s number of correspondences. Even for a modest value of n, the total number of possbltes could be huge, whch mples very expensve computaton. Snce the try-allpossbltes method s nfeasble, N can be chosen accordng to probablty p whch makes that at least one of the random samples of s ponts s not an outler. Suppose w s the probablty that any selected data pont s an nler, and hence ε = 1 w s the n 4 probablty that t s an outler. At least N samples can make (1 w ) s N = 1 p, so that N = log (1 p) / log (1 w s ). (8)

27 20 Because the w and ε are usually unknown, they can be determned adaptvely [17] by the followng procedure: (1) N =, sample_count = 0. (2) Whle N > sample_count Repeat - Choose a sample and count the number of nlers. - Calculate ε = 1 w. - Compute N usng equaton (8) wth p = Increase sample_count by 1. (3) Termnate The Eucldean dstance threshold D. Hartley and Zsserman [17] assume that the measurement error s a Gaussan random varable wth zero mean and standard 2 2 devatonσ, and, n ths stuaton, that d s a χ dstrbuton. The probablty that a χ random varable s less than any gven number k s gven by the cumulatve chsquared dstrbuton, F (k ), whch can be found n any standard mathematcal table. If 2 k s set to 0.95, D = 5.99σ 2. In practcal experments, ths threshold value s too large and hence not practcal. Also, the dstrbuton of measurement error s certanly not Gaussan, snce many outlers exst. So a relatvely small value of D = 1.25 s chosen, whch works well for the experments. The threshold T for the sze of an acceptable consensus set. To ensure that the correct model can be found and to satsfy the fnal smoothng procedure, for n sample ponts, T = (1 - ε ) n s a good choce. For the stuaton whereε s unknown, a T wth value a lttle larger than that necessary for a smoothng computaton can be used.

28 Optmal Estmaton The homography obtaned by robust estmaton can be used as a gudelne for further optmal estmaton. All correspondences {x x ' } between any two frames are calculated by the functon gven n equaton (7). The outlers of these correspondences are fltered out usng the same threshold value used n RANSAC procedure. The correspondences classfed as nlers are then used to determne a maxmum lkelhood estmate of H by mnmzng the followng object functon: 1 ' 2 ' 2 d( x, H x ) + d( x, Hx ). In the experment, a lnear least squares method s used to obtan the optmal H whch best satsfes all the nlers. 3.4 Image Blendng Based on the optmal homography H, the frames can be well algned. However, there are stll dfferences n the ntensty values of pxels, whch are caused by the changng of the camera s nternal parameters durng dfferent perods of the capturng process, especally n the regons where the frames overlap. To solve ths problem, a functon to weght each pxel n all frames s ntroduced: w(, j) = h / 2 h/ 2 w/ 2 w/ 2 j,

29 22 where h and w are the heght and the wdth of the frame. Heurstcally, the pxels at the edge of the frames are gven less weght. 3.5 Background Mosac Generaton The homography H of any two non-consecutve frames s obtaned exactly by the composton of homographes of all the frames between them. For example, the homography of the frst and the thrd frame H 13 can be computed by the composton of the homography between the frst frame and the second frame and the second frame and the thrd frame as H 13 = H H 12. By usng a specal frame as the reference frame such as the frst frame or the last frame, the homographes between all frames n the vdeo sequence and ths reference frame can be computed. Consequently, all frames can be mapped onto the reference frame to generate the background mosac. It should be noted that the orgn of the generated background mosac mage s dfferent from the frame orgn. The orgn of the background mosac shfts durng the frame warpng procedure. A boundng box for the current mosac orgn should be recorded durng the processng of homographes computaton. When the computaton of homographes between all frames and the reference frame s completed, the lower left corner of the boundng box of the entre mosac namely (x, y ) s obtaned. Ths orgn of the entre background mosac s used to calculate the nverse of the homography matrx used to generate a background mosac from the frame located at the orgn of the entre mosac. The shfted nverse of any homography s computed by the composton of the nverse of the homography and a translaton matrx. For example, H = H 1 12 mn mn T, where T s gven by: 23 21

30 x mn T = 0 1 y. mn A sample of the statc background panorama generated by the algorthm mplemented n ths thess s llustrated n fgure 3.3. The man steps of background mosac generaton can be summarzed as follows: (1) Detect the nterest ponts usng the Moravec corner detector n each frame. (2) Fnd the correspondences between frames usng these nterest ponts by a crosscorrelaton operaton. (3) Compute the ntal homographes between frames usng the Drect Lnear Transformaton (DLT) Algorthm. (4) Use the RANSAC technque to flter out outlers of correspondences for each par of frames. (5) Compute a maxmum lkelhood estmate to obtan the optmal homographes over all frame pars. Specfcally, the lnear least squares algorthm s mplemented to compute the maxmum lkelhood estmate based on the nlers of the correspondences. (6) Use the estmated optmal homographes to sttch all the frames onto a reference frame to generate the statc background mosac.

31 Fgure 3.3: The statc background generated from the orgnal frames n the vdeo sequence wthout the dynamc foreground. 24

32 CHAPTER 4 FOREGROUND AND BACKGROUND SEGMENTATION 4.1 Dynamc Foreground and Statc Background The frames extracted from the orgnal moton vdeo can be segmented nto two layers: statc background and dynamc foreground. The statc background generated by the procedure descrbed n the prevous chapter ncludes all relatvely statc objects n the scene such as buldngs or mountans. The dynamc foreground that needs to be segmented, on the other hand, s assocated wth the movng objects such as walkng people or movng cars. Dynamc foreground segmentaton s relatvely easer for the cases where the dynamc scenes are captured by statc cameras. Snce the camera s always located n the same poston and there s no moton of the camera such as pan and tlt, the movng objects n the scene can be extracted by pxel-to-pxel comparson between the pre-stored background and the current frame beng processed. Ths strategy works well only when the background nformaton s avalable beforehand. The foreground segmentaton for dynamc scenes captured by movng cameras s computatonally much more complex. The camera motons such as pan and tlt usually compensate for the moton of the movng objects n the scene such that these objects reman n the center of the frame. For example, actors or athletes always stay n the center of the mages or frames of the move sequence because the camera s panned or tlted n order to follow them. So 25

33 26 a more sophstcated background/foreground separaton technque s requred to deal wth ths complex stuaton. 4.2 Mahalanobs Dstance In the vdeo sequence, the prevous and the next frames can be mapped onto the current frame usng the estmated homographes. The color values of every pxel n the frame are then compared at each pxel locaton. The pxels belongng to the statc background follow the estmated camera moton and hence the changes of ntensty value between the correspondng pxels are relatvely small. On the other hand, large dscrepancy n ntensty values occurs at pxels whch do not conform to the estmated homography. The comparson of color values at each pxel locaton s acheved by the followng dstance functon: ( Γ ( q 1), Γ ( q )) + d( Γ + 1( q 1), Γ ( q )) (9) d ( ) where Γ q s the ntensty value of the pxel q n the frame Γ, and s the number of frames n the vdeo sequence. Note that d s the Mahalanobs dstance [19] whch represents the dscrepancy of n color values between the two pxels when they appear n two consecutve frames. The Mahalanobs dstance s gven by T 1 d( Γ ( q ), Γ ( q )) = ( Γ ( q ), Γ ( q )) C ( Γ ( q ), Γ ( q )) where C s the covarance matrx for the RGB color space, and s estmated usng red, green, and blue color values for all the pxels and for each frame n the vdeo sequence.

34 Probablty Image Based on the values obtaned for each pxel locaton n the frame computed usng the functon n equaton (9), a probablty mage s generated. It s made up of the lkelhood of every pxel n the frame belongng to the dynamc foreground. For example, a large dscrepancy n color value at the same pxel poston q n three consecutve frames has a large probablty of beng the dynamc foreground. Here the three consecutve frames are any set of prevous frame, current frame and next frame n the vdeo sequence. The pxels belongng to movng objects whch do not conform to the homography between consecutve frames have a greater probablty of possessng larger dscrepancy n color values. 4.4 False Moton Detecton Although the above approach, whch borrows deas from the drect method to mosac reconstructon, can detect the dynamc layer of the frame, there are stll problems wth t. Two categores of false detecton of movng objects exst n the vdeo frames. One s caused by a certan level of pxel-level nose whch s ntroduced by the camera capture or the vdeo producton process. The other category s caused by the presence of large homogeneous regons and complex motons such as artculated body motons whch are wdely present n many reallfe vdeos. Large homogeneous regons are the nteror of the movng objects. The artculated body motons are characterzed by the fact that some body parts move whle some other body parts reman stll. The problem of the presence of false moton can be solved by performng a Gaussan flterng on the probablty mage. The Gaussan smoothng flter s very well suted for removng nose

35 28 that s drawn from a normal dstrbuton. In the context of mage processng, the two-dmensonal zero-mean dscrete Gaussan flter s gven by g [, j] = e 2 2 ( + j ) 2 2σ and s used as a smoothng flter. A typcal two-dmensonal Gaussan flter s llustrated n fgure 4.1. Fgure 4.1: A typcal two-dmensonal Gaussan flter

36 29 The smoothng procedure s performed on the probablty mage nstead of the orgnal frame. Although Gaussan smoothng s essentally a procedure to blur the mage, applyng the flter on the probablty mage can retan the mage qualty as well as flter out the nose. The detected regons correspondng to the movng objects pror to smoothng and the regons correspondng to the movng objects after smoothng by a Gaussan flter are llustrated n fgure Segmentaton of Dynamc Foreground To detect the moton foreground from the orgnal frames, a probablty threshold s set to optmstcally segment the dynamc layer from the statc layer. The threshold s appled onto the probablty mage whch has been smoothed by the Gaussan flter. In general, a probablty value of less than half of the maxmum n the probablty mage s suggested as the threshold. In the experment, the recommended threshold s so large that many parts of moton objects were deleted. So n practce, a value of 1/8 of the maxmum value s used. When applyng ths threshold to the probablty mages, the pxels wth the probablty value larger than the threshold were classfed as the dynamc foreground and kept. At the same tme, the pxels whch are smaller than the threshold were classfed as the statc background and removed. Though the dynamc foregrounds have been segmented, they are stll stored wth the background n the mage fle. The only dfference s that all the pxels belongng to the background change from the orgnal color to black. In order to reduce the multmeda nformaton transmtted through the moble network, the detected dynamc foreground wthn each frame could be dvded further nto several connected components and then extracted from the background.

37 30 (a) (b) Fgure 4.2: (a) The frame showng the detected regons correspondng to the movng objects pror to smoothng; (b) The detected regons correspondng to the movng objects after smoothng by a two-dmensonal Gaussan flter.

38 Connected Components Detecton All the frames n the vdeo sequence are now dvded nto two layers: the statc background layer and the dynamc foreground layer. To fnd all the connected components n the dynamc foreground whch ncludes both real movng objects and the nosy or spurous regons, each frame s converted to a bnary mage where the value 1 s assgned to the pxels of the dynamc foreground and the value 0 s assgned to the pxels of the statc background. Here a connected component s a set of pxels n whch each pxel s connected to all other pxels n that set. Unlke the gray scale mage, bnary mage contans only two gray levels. The advantages of bnary mage nclude they are well understood and tend to be less expensve and faster durng the procedure of mage processng than the gray level or color mages. Bnary mages are used n bnary vson systems to reduce the memory and computng power requrement. Tradtonally, pxels of assumed objects whch could nclude both movng objects and statc objects are set to whte whle the other pxels belongng to background are set to black. In ths thess, the bnary mage such as fgure 4.3 was generated by set the color of pxels belongng to statc background to black and the color of pxels belongng to dynamc foreground to whte. The teratve connected-component labelng algorthm s appled to the bnary mage and usually requres two passes over the mage. Ths algorthm checks the two neghbors of a current pxel, namely, the one above and to the left of the current pxel and tres to assgn an already used label to the current pxel. When the two neghbors have dfferent labels, an equvalence table s used to keep track of all labels that are deemed equvalent. Ths table s used n the second pass to assgn a unque label to all the pxels of a connect component.

39 32 a b Fgure 4.3: (a) The orgnal frame; (b) The correspondng bnary mage whch has already been smoothed by a Gaussan flter.

40 33 The algorthm dvdes the neghborhood relaton of pxels nto three cases and assgns dfferent labels for them. The equvalence table ncludes the nformaton of unque labels for each connected component. Durng the frst scan, all labels assgned to one component are clamed as equvalent. In the second pass, the smallest correspondng label from the equvalence table s selected to be assgned to all pxels of a certan component. When all connected components have been detected, the equvalence table s renumbered to elmnate the gaps between labels. The connected components n the mage are then reassgned the new label under the drecton of the equvalence table. The man steps of the teratve connected-component labelng algorthm are summarzed as follows: (1) Scan the bnary mage from left to rght, top to bottom. (2) If the current pxel s 1, then (a) If only one of ts upper and left neghbors has a label then copy that label. (b) If both of them have the same label, then copy the label. (c) If both of them have dfferent labels, then copy the label of upper pxel and note n a equvalence table that label (upper) = label (left). (d) Otherwse assgn a new label to the pxel and note the label n the equvalent table. (3) Repeat steps (2) (a) - (2) (d) untl all 1 -pxel have been vsted. (4) For each equvalence class n the equvalence table, assgn a unque label, typcally the lowest. (5) Rescan the mage and replace the label of each 1 -pxel by the label of ts equvalence class.

41 34 The above algorthm detects all the connected components n an mage. Many propertes of the component such as sze, poston and boundng box can then be computed for each component for later processng. 4.7 Sze Flterng Even after the segmented dynamc foreground has been smoothed by the Gaussan flterng, a certan number of nosy or spurous regons stll persst. The moton components are found by the Mahalanobs dstance method whch detects the moton based on the color dscrepances of the correspondng pxels n the consequent frames. Sometmes the small changes n reflectance and llumnaton characterstcs of other objects n the scene can lead to ncorrect detectons of the moton. One mportant property of these spurous regons s that ther szes are small compared wth those of the real movng objects n the scene and hence can be removed by a sze flter. The connected components detected by the teratve connected-component labelng algorthm consst of components belongng to both the real movng objects n the scene and the unexpected noses. In order to remove these noses, a sze flter s used based on the sze property of ths category of nose outlned above. When all connected components have been found n the dynamc foreground, the sze flter s used to suppress the nosy artfacts wth relatvely small sze n terms of number of pxels. The threshold of the sze flter can not be set to large, whch wll remove the real movng components. At the same tme, the threshold can not be set too small, whch wll keep too many noses. Consderng the dfferent applcatons, the algorthm should be robust to dfferent cases.

42 35 In the experments presented n ths thess, a threshold of 1/3 or 1/4 of the maxmum sze of the component n the dynamc foreground s used. The result s llustrated n fgure Moton Components Extracton After the sze flterng operaton, only large components correspondng to movng objects are kept. A boundng box whch s composed of the mnmum and maxmum coordnates of a certan component n the frame s recorded nto an nformaton fle for later transmsson. The thresholded components are extracted from the orgnal frames and used to generate a set of small mage fles whch store only the pxels correspondng to regons n the boundng boxes. The procedure for the extracton of these small mages correspondng to movng components s shown n fgure 4.5. Based on the expermental observatons, the small mage fles representng the movng components n the frame are only, on average, 1/4 to 1/5 of the orgnal frame n terms of sze. The compresson rate s satsfactory. Ths mples a very good compresson rato for multmeda nformaton and s really convenent for moble networked transmsson. The man steps of the foreground and background segmentaton procedure whch s performed at the server-end can be summarzed as follows: (1) Compute the Mahalanobs dstance n color space for every pxel n all the frames usng any three consecutve frames. (2) Generate the probablty mage for each frame n the vdeo sequence based on the Mahalanobs dstance and equaton (9). (3) Use the Gaussan flter to smooth out regons of false moton caused by large homogeneous regons and complex motons.

43 36 a b Fgure 4.4: (a) The orgnal bnary mage; (b) The bnary mage fltered by the sze flter.

44 37 (4) Set a probablty threshold to classfy each pxel n the frame as belongng to the dynamc foreground or the statc background. (5) Use teratve connected-component labelng algorthm to detect connected components n the bnary mage of the segmented dynamc foreground. (6) Apply the sze flter to remove the nosy artfacts and dentfy components belongng to the real movng objects n the vdeo stream by the applcaton of Mahalanobs dstance. (7) Generate certan numbers of sets of small mage fles correspondng to moton components of real movng objects n each frame n the vdeo sequence. Fnd the boundng box of the detected moton components and extract the related locaton nformaton for the later moble networked transmsson.

45 38 a b c Fgure 4.5: (a) The orgnal frame; (b) The boundng box for a movng component; (c) The extracted small mage of the regon belongng to the movng object.

46 CHAPTER 5 MOTION PANORAMA CONSTRUCTION 5.1 Network Transmsson Three categores of fles are transmtted through the network, namely: A sngle large mage fle contanng the statc background. The fle ncludes all background nformaton n the scene captured by the movng camera. A certan number of small mage fles contanng the dynamc foreground. These fles nclude all the varous components correspondng to movng objects n the scene for each frame n the vdeo sequence. An nformaton fle for each frame. The fle ncludes all assocated parameters such as boundng boxes of dynamc components and the homography between each frame n the vdeo stream and the reference frame. 5.2 Moton Panorama Reconstructon at the User-end When all the nformaton has been transmtted from the server to the user-end, t s used to reconstruct a moton panorama. The statc background mage and the dynamc foreground for each frame n the vdeo sequence are avalable now. The dynamc foregrounds are then pasted onto the statc background based on the parameters n the nformaton fle to reconstruct the moton panorama. 39

47 40 The homographes between each frame n the vdeo sequence and a reference frame are computed durng the procedure of statc background generaton. The dynamc foreground of each ndvdual frame s mapped onto the background mosac usng these homographes. Ths s almost the same procedure as the prevous generaton of background mosac except that only the extracted regons of foreground are now pasted nstead of the entre orgnal frames. In [10], the authors propose a method to buld the background panorama by consderng each potental pxel n the background mage plane. For each of these pxel locatons, the contrbutons from a certan number of frames are accumulated and weghted to obtan the fnal ntensty value for that pxel. The ndvdual frames are then mapped onto the background and consequently used to extract the dynamc foreground. Ths method entals a sgnfcant amount of computaton because all pxels n the large background mage, whch ncludes the pxels from both the statc background and the dynamc foreground, are determned va the computaton of an average of the correspondng pxels from 20 related frames n the vdeo sequence. In ths thess, the regons comprsng the dynamc foreground n each ndvdual frame are segmented from the background. Subsequently, only these foreground regons are pasted onto the statc background to reconstruct the moton panorama. More specfcally, the segmented components of the dynamc foreground n each frame are mapped onto the background usng both, the boundng boxes whch nclude the locaton nformaton of the dynamc components n each frame, and the homography between that frame and the reference frame. In the statc background mage, when the ntenstes of pxels n the mapped regons of dynamc components are the same as those of correspondng pxels n the background, the ntensty of the correspondng pxel n the background mage does not change. Otherwse, the ntensty of a pxel n the background mage s replaced by the ntensty of the correspondng pxel n the mapped

48 41 regons representng the dynamc components. In other words, the regons correspondng to the dynamc foreground or movng objects are pasted onto the statc background mage. Followng the vdeo sequence, f the dynamc foreground s pasted onto the statc background once n every few frames, a moton panorama s generated. A statc representaton of ths form contanng a large background mage wth a seres of moton objects n t expresses the content of orgnal moton vdeo wth much less space. For the applcaton where the panoramc vdeo s requred, an alternatve strategy s mplemented. The dynamc foreground of each ndvdual frame n the vdeo sequence s pasted onto the background separately. Each frame n the vdeo sequence generates one moton panorama. When the generaton of panorama mages from all frames s completed, one can combne all these mages of panorama together to create an MPEG or AVI format fle for vewng.

49 CHAPTER 6 EXPERIMENTAL RESULTS The technque for moton panorama constructon descrbed n ths thess s appled to several moton vdeos captured by a dgtal camcorder. The scenes of these moton vdeos were acqured on the campus of the Unversty of Georga. A typcal sample used n the experment s a 10 second vdeo wth multple persons walkng n front of Dawson Hall. The vdeo, whch ncludes around 210 frames, s M bytes n sze. The results shown n fgure 6.1 and fgure 6.2 consst of the procedure of the moton panorama constructon usng the proposed approach. The panorama s constructed wth both the large statc background and dynamc foregrounds extracted for every 40 frames. The panoramc vdeo whch conssts of a sngle statc background and a certan number of foregrounds correspondng to each frame of the orgnal moton vdeo can also be constructed at the user-end usng the smlar technque. In ths form of representaton, the dynamc foregrounds move n a sngle large background wthout losng any nformaton from the orgnal moton vdeo. In table 6.1, three forms of moton representaton, namely, orgnal moton vdeo, moton panorama and panoramc vdeo are compared n terms of the type and the sze of fles based on the multmeda nformaton transmtted through the moble computer network. The results are satsfactory wth an average compresson rate of around 0.1. The technque of moton panorama or panoramc vdeo constructon can greatly reduce the amount of nformaton transmtted and 42

50 43 hence conserve the power consumed at the user-end n power-constraned moble networked envronments.

51 44 Table 6.1: The comparson of three forms of moton representaton based on the type and the sze of fles transmtted through computer networks Orgnal Moton Vdeo Moton Panorama Panoramc Vdeo 1 fle of the statc 1 fle of the statc background mosac- background mosac- JPG (165 K Bytes) JPG (165 K Bytes) 5 set of fles of the 210 set of fles of dynamc foreground- dynamc foreground- Transmtted 1 fle of the moton JPG (49.9 K Bytes, JPG (3.92 M Bytes, Fle(s) wth Type vdeo-avi (41.25 M average 9.98 K Bytes average 18.7 K Bytes and Sze Bytes) or 210 fles of per fle) per fle) orgnal frame-jpg 5 fles of assocated 210 fles of assocated (17.85 M Bytes, average locaton nformaton- locaton nformaton- 85 K Bytes per fle) TXT (0.42 K Bytes, TXT (20.6 K Bytes, average 84 Bytes per average 98 Bytes per fle) fle) Total Sze M Bytes / M Bytes K Bytes 4.11 M Bytes

52 45 a b c d e Fgure 6.1: (a) An orgnal frame (138); (b) The detected movng components and ther locaton nformaton; (c) and (d) Extracted small mages of movng component; (e) The sngle mage of statc background. (c), (d) and (e) are the actual fles transmtted from the server-end to the user-end.

53 Fgure 6.2: The moton panorama wth multple movng objects, whch s generated based on the method proposed n ths thess. Ths panorama s constructed usng one large statc background and a certan number of dynamc foreground objects extracted once n every 40 frames n the moton vdeo sequence whch was captured n front of the Dawson Hall on the Unversty of Georga campus. 46