Fast Approximate Matching of Videos from Hand-Held Cameras for Robust Background Subtraction

Transcription

1 ast Approximate Matching of Videos from Hand-Held Cameras for Robst ackgrond Sbtraction Raffay Hamid DigitalGlobe Labs. Atish Das Sarma, Dennis DeCoste, Neel Sndaresan eay Research Labs. {adassarma, ddecoste, Abstract We identify a novel instance of the backgrond sbtraction problem that focses on extracting near-field foregrond objects captred sing handheld cameras. Given two ser-generated videos of a scene, one with and the other withot the foregrond object(s), or goal is to efficiently generate an otpt video with only the foregrond object(s) present in it. We cast this challenge as a spatio-temporal frame matching problem, and propose an efficient soltion for it that exploits the temporal smoothness of the video seqences. We present theoretical analyses for the error bonds of or approach, and validate or findings sing a detailed set of simlation experiments. inally, we present the reslts of or approach tested on mltiple real videos captred sing handheld cameras, and compare them to several alternate foregrond extraction approaches.. Introdction ver the years, there has been a tremendos increase in the nmber of videos recorded from handheld mobile devices. A majority of these videos captre a set of near-field objects in visally clttered environments. Atomatic extraction of these foregrond objects in sch videos is an important problem, with potential impacts on several application areas, e.g., commerce, entertainment, and jornalism. or videos captred sing handheld cameras, it is relatively easy to record a few frames of the scene withot the foregrond objects present, to coarsely characterize what the backgrond looks like. We arge that this extra, albeit approximate, information abot the backgrond scene can be very sefl to extract foregrond objects in sch videos. The motivation of or novel problem setting is gronded in several real-world applications. In online commerce for instance, thosands of sellers se mobile phones to captre videos of their prodcts. Most of these sellers are amater videographers, and an important factor adversely affecting their video qality is the backgrond scene cltter. These sers are therefore natrally incentivized to provide a short additional video of the backgrond scene in retrn for having prodct videos with improved qality. Similarly on filming rigs, it is a standard practice to captre backgrond and foregrond videos along careflly planned camera paths sing expensive compterized dollies to generate varios special effects. y enabling robst and efficient video matching from hand-held cameras, or work attempts to make this process more efficient and affordable. The goal of or work is, given two handheld videos, one with and the other withot foregrond object(s), to efficiently generate a video with only the foregrond object(s) shown. Note that if camera paths for both the videos in or problem were identical, we cold simply perform a pixelwise backgrond sbtraction. n the other hand, if the two camera paths were completely different with no scene overlap, the backgrond video wold be redndant and or problem wold transform to object tracking. In practice however, one is somewhere in the middle of this spectrm with some partial path overlap between the two videos. The challenge therefore is, for each foregrond frame, to find a set of backgrond frames with sfficient scene overlap, sch that with the appropriate frame-alignment, they cold be sed to sbtract ot the backgrond pixels. A naive approach to this spatio-temporal frame matching reqires operations qadratic in the nmber of foregrond and backgrond frames. or only one minte long backgrond and foregrond videos captred at 30 frames per second, a naive approach reqires more than 3 million pairwise frame comparisons. Even with an optimized implementation, each comparison can take p to 00 ms, which wold reslt in the entire process taking several days to complete. It is therefore crcial to find relevant backgrond frames for each foregrond frame more efficiently. The main contribtions of or work are: We propose an approximation algorithm for efficient spatio-temporal matching of videos from hand-held cameras. r algorithm permits a significantly improved comptational complexity compared to exhastive approaches.

2 We qantify bonds on the error incrred by or approximate matching algorithm, and corroborate or theoretical findings by a set of controlled simlation experiments. We present a comptational framework sing or frame matching approach to efficiently sbtract the scene backgrond given a backgrond and a foregrond video. b 8 f9 b 9 f8 b 7 f7 f b p 0 b b 2 f0 f2 b 3 f4 f3 f6 b 5 b b 6 4 f5 In the following, we start by going over the relevant previos work, followed by or problem frmlation. We se the insights from or formlation to make appropriate choices for or comptational framework explained in 5. ackgrond at b2 2. Related Work oregrond at f2 oregrond extraction in videos has previosly been explored in great detail [22] [25], however most of this work has either focsed on static camera(s), or for cameras with limited motion [3] [4]. To overcome this challenge, there have been several efforts to look at the problem prely from a tracking based perspective [4] [7]. While these approaches can sccessflly localize object position, they are not geared for accrately extracting object silhoettes. A soltion for this challenge is to se tracking and segmentation in closed-loop form [23] [8]. However, sch approaches either reqire hman intervention to provide object silhoette, or assme specific lighting conditions. Another method to estimate object silhoette to initialize tracking is to segment the scene into mltiple motion layers [2] [24]. However, sch methods are limited to nonstatic scenes with distinct object-level motion bondaries. There have been some recent attempts to combine appearance and stereo ces for foregrond extraction. Most of these methods however perform fll seqence calibration [6], or assme that the backgrond scene can be approximated sing piecewise planar geometry [0]. There have also been attempts to extract foregrond object(s) based on frame saliency [5] [] and co-segmentation [7]. Recently there has been work on tilizing the general backgrond appearance from nregistered images [2] []. We show in 6.3. however that this model does not generalize for handheld cameras as their nconstrained motion reqires qite a dense sampling of backgrond scene appearance, reslting in prohibitively high captre and comptational cost. Similar scene alignment approaches have been explored for videos [9] [5] [6], however nlike or work, they assme same or near identical camera paths. 3. Problem Setp Consider igre, where a 3-D scene is shown to be captred by a handheld camera tracing two different paths. or one of these captres, the scene contains a (set of) foregrond object(s) reslting in a foregrond video. or the second captre, a backgrond video is recorded withot the foregrond object(s). Let n and m denote the nmber of : : igre : An example 3-D scene is captred by a handheld camera tracing two different paths, f and b, one with and the other withot a foregrond object (red tray). This reslts in a foregrond and a backgrond video, f and b respectively. Example frames from these videos for this scene are also shown at f 2 and b 2. foregrond and backgrond video frames respectively. Let f = {f, f 2,..., f n } and b = {b, b 2,..., b m } denote the foregrond and backgrond frame seqences respectively. We se d(x, y) to denote the distance between frames x and y, the exact notion of which is formally defined in Problem Assmptions We make the following assmptions to setp or problem. 3.. Path Smoothness We assme that consective frames in a video are similar to each other, i.e., the camera-paths are generally smooth. ormally, i ( i < n), we have d(f i, f i+ ) δ, and j ( j < m), we have d(b j, b j+ ) δ, for δ > Captre Completeness Given or problem setp, it is expected that several foregrond frames wold have strong backgrond matches. We begin by making this captre completeness assmption f f, and address the scenario when this is not always the case in 5.6. ormally, for each i between i n, j between j m sch that d(f i, b j ) ɛ Problem ormlation Given f and b, i n, we want to find a corresponding j m to minimize d(f i, b j ). More formally, we want to identify fnction π : [, n] [, m] sch that avg i=n i= d(f i, b π(i) ) is minimized.

3 Algorithm NEAR-LINEAR(k) Inpt: f, b, and parameter k. tpt: π, matched backgronds f f. : Let Si k = {i : i n, i = 0 mod k} {} {n}. 2: Let Sj k = {j : j m, j = 0 mod k} {} {m}. 3: i Si k, set π(i) = arg min j S d(f i, b j k j ). 4: i / Si k, let near(i) = arg min t S t i. i k 5: i / Si k, set π(i) = π(near(i)). Algorithm 2 LCAL-SEARCH( f, b) Inpt: f, b, k, and γ. tpt: Matched backgrond f f. : Let Sj k = {j : j m, j = 0 mod k} {} {m}. 2: or j Sj k, compte if b j is a strong match for f. 3: b j that are strong matches, compare f with each b b x where x : [j γ, j + γ] to find other strong matches. 4: Retrn all strong matches fond. 4. Spatio-Temporal Matching We first present a naive video frame matching algorithm, followed by or proposed approximate algorithm that garantees a significant speedp while only incrring minimal accracy cost. or all the proofs of theorems and lemmas, we refer the reader to the spplementary material. 4.. Naive Algorithm Recall that matching is determined by a fnction π from [, n] to [, m], and that the cost of any matching, π, is given by C(π) =avg i=n i= d(f i, b π(i) ). The optimal matching is one that minimizes this cost. Let s denote the optimal matching by π and the associated cost by C(π ). That is, π = arg min π avg i=n i= d(f i, b π(i) ). The path smoothness assmption reslts in the following lemma. Lemma 4.. C(π ) ɛ. A naive brte-force matching finds the best backgrond match for every foregrond frame that minimizes d(f i, b j ). The time complexity of this algorithm is Θ(mn) Speed-Up with Provable Error ond We now present a more efficient matching scheme detailed in Algorithm (cf. NEAR-LINEAR). The main intition here is that rather than iterating throgh every foregrond frame, we only consider foregrond frames in chosen intervals of k. The remaining foregrond frames can then be matched based on the two considered foregrond frames sandwiching it. To obtain an additional improvement, a similar idea is leveraged in the backgrond video, by making comparisons only in jmps of k backgrond frames. De to path smoothness, we know that every foregrond frame that got omitted in consideration is close enogh to a chosen foregrond frame, in particlar, at most k 2 steps away. rthermore, de to captre completeness, there exists a backgrond frame that is sfficiently close to each chosen frame. Even thogh we may have skipped the ideal match, the crcial observation here is that we can apply smoothness on the backgrond as well. We now concretely formalize this intition. Theorem 4.2. Algorithm NEAR-LINEAR(k) rns in ( nm k ) time and retrns π sch that every foregrond 2 matches to a backgrond with additive error of at most kδ, implying C(π) C(π ) + kδ ɛ + kδ. Note that we may choose k as any vale between and min{n, m}. This allows s to trade-off between efficiency and accracy parametrically. or example, if we choose k m, we get a near-linear rnning time. We state a corollary for an alternative parameter choice that admits the k 2 speed-p while garanteeing near-optimality. Corollary 4.3. or k = ɛ δ, NEAR-LINEAR(k) garantees an (ɛ) cost, and yet rns k 2 times faster. The effect of varying k on the algorithm s accracy is empirically analyzed in a systematic manner in Clstered Matches We now show that given any foregrond frame, if there are strong matches (defined below) in the backgrond, they tend to be in close temporal proximity of each other. This nderstanding allows for a more efficient matching algorithm that can exploit a local constraint. In this section we show the theoretical reslt based on a realistic randomized model that we introdce, and 6 corroborates or findings throgh controlled simlation experiments. Recall that a backgrond frame b corresponds to a strong match for f if the distance d(f, b) is small. Let s formalize this notion more crisply here. Strong Match: We define b to be a strong match for f if d(f, b) Ψ. Moreover, given a strong match {f, b}, we model the measre of distance d(f, b) as a sample from the niform distribtion [0, Ψ]. efore presenting the main reslt of this sbsection, we first state a few prereqisite reslts. Lemma 4.4. Sppose that b i is a strong match for a foregrond frame f. Then, the probability that b i±γ is also a strong match for f is at least Ψ γδ Ψ. Lemma 4.5. Given a strong match between f and b i, the expected nmber of strong matches for f in the backgrond seqence range [b i γ, b i+γ ] is at least 2(γ +)( δγ 2Ψ ).

4 Recall that captre completeness only gives an pper bond garantee even for the best match, specifically a distance of at most ɛ. We therefore consider strong matches as having a weaker pper bond garantee, i.e., Ψ cɛ (i.e. Ψ δ c). In sch a sitation, we wish to interpret Lemma 4.5 with reasonable parameter vales. We present one sch choice of parameters in the following theorem and then proceed to describe the algorithm that wold garantee efficiency in terms of a speed-p similar to k 2 as before, withot the added cost of missing ot on strong matches. Theorem 4.6. Given a strong match, with a window size γ = Ψ δ, the expected nmber of strong matches in the entire window of size (2γ + ) is at least γ. a P - Key Point Detection & Matching oregrond rame (t) b ackgrond rames N ackgrond rame Matches ond? YES 2 - Perspective Transformation c Track frame t sing otpt from t- P This leads to an algorithm that sggests making γ skips in the backgrond frames and whenever encontering a strong match, adopting a local search approach. Theorem 4.6 therefore sggests finding several strong matches withot performing exhastive search. This procedre is listed for one foregrond frame in Algorithm 2, and can be extended f f with strong backgrond matches. or foregrond frames with no strong backgrond matches, we propose to rely on foregrond tracking (see 5.6 for details). 5. Comptational ramework We now present a foregrond extraction framework that exploits or frame matching algorithm as explained in 4. The main steps of or framework are illstrated in igre 2. There are two important insights to be noted here: While or scene might have depth, we show that sing projective transform followed by non-rigid frame warping generally works sfficiently well for aligning the backgrond and foregrond frames. rthermore, the warping error obtained for a foregrond frame varies over different matched backgrond frames. This allows s to integrate over mltiple noisy warping hypotheses to obtain a robst inference abot the foregrond. Recall that approximating scene geometry withot sing the complete 3-D scene strctre has previosly been explored [20]. The novelty of or framework however lies in applying this idea to obtain mltiple noisy foregrond estimates, and to then fse these hypotheses to obtain the final robst inference. We now explain or framework steps. 5.. Key-Point Detection & rame Matching or each foregrond and backgrond frame, we compte key-points descriptors sing the SUR pipeline [3], and find matches between them sing the RANSAC algorithm [8]. We define the frame distance d(f, b) as the reciprocal of the nmber of their key-point matches and search for corresponding frames sing or method explained in 4. a 2 c ptical low Comptation a 3 c Non-Rigid Transformation c 3 a 4 c rame Sbtraction a 5 c Voting tpt (t) igre 2: Illstration of or comptational framework Perspective Transformation or each f, given a strong match b b, we compte a perspective transform ζ f,b between them [2] sch that b = ζ f,b f. Using ζ f,b, we warp b to align it with the given f, and generate b, sch that b = ζ f,b b [9] Non-Rigid Transformation Using perspective transformation, most parts of f and b get well-aligned with each other. However, there still may be parts where this alignment is not accrate. This disparity is possible becase of the planar assmption that perspective transformation makes regarding the scene. However, for or problem this assmption is not necessarily tre. To this end, we first compte the optical flow between f and b. We then apply the compted flow-fields to warp b onto f, to find the non-rigidly transformed backgrond frame b [9].

5 Algorithm 3 VTING-SCHEME( f, b) Inpt: f f, and a set of strong match backgrond frames b s (f) {b, b 2,..., b m }. Threshold parameter t. tpt: ackgrond sbtracted image corresponding to f. : Let b s (f) = r. 2: or every pixel that is labeled as foregrond on at least τ v strong matches, mark it as foregrond in the reslt. 3: All other pixels are labeled as backgrond rame Sbtraction or each pair of f and b, we perform a per-pixel colorbased frame sbtraction. The sbtracted pixels are thresholded (sing τ s ) to obtain the foregrond mask, i.e. { if f(x, y) b (x, y) > τ s (x, y) = () 0 otherwise Per-Pixel Voting rame sbtraction gives mltiple candidate foregrond hypotheses i (x, y). or each foregrond frame pixel we cont the nmber of times it is classified as foregrond. If this fraction is more than a certain threshold τ v, it is finally classified as a foregrond scene pixel R(x, y), i.e., { if avg i( i (x, y)) > τ v R(x, y) = (2) 0 otherwise. This procedre is listed in Algorithm 3 (cf. VTING- SCHEME). Note that the likelihood of classifying a pixel correctly increases exponentially with the increase in the nmber of strong matches. ormally, assme that for any pixel p, we have the following two properties: If the pixel is from foregrond, then it is reflected as so with probability p, and If the pixel is from backgrond, then it is reflected as so with probability p 2 where p + p 2 >. Then the following lower-bond holds on the probability of correctly classifying p. Theorem 5.. Given r strong matches, and a voting threshold t in VTING-SCHEME in the interval ( p 2, p ), the probability of correctly classifying a pixel is e (r). Intitively, Theorem 5. sggests that for each foregrond frame, we only need a few backgrond frames with strong matches to perform foregrond extraction robstly Selective oregrond Tracking There might be foregrond frames for which there are no strong backgrond matches to be fond. or sch frames, While color as a measre of pixel difference worked well for s, one cold also se more robst pixel featres instead e.g., dense SUR [3] etc. we propose to track the latest detected foregrond from their previos frame that did have strong backgrond matches. or this work, we fond Mean-Shift tracking [7] to be an efficeint choice for tracking the foregrond when needed. Since for any foregrond frame, tracking the foregrond pixels is only a constant time operation, it does not effect the overall complexity of the algorithms described in Simlation Analysis Since the paths traced by handheld cameras can be nconstrained, it is important to analyze how the accracy of or framework varies as a fnction of the difference between the cameras paths traced by the backgrond and foregrond videos. We now present a set of controlled simlation experiments to analyze this qestion in a systematic manner. 6.. Video Data Generation We sed a 3-D animation software (lender) to design a desktop scene (igre ) where we modeled the backgrond camera path as a circle parameterized by a set of key points. To generate the foregrond camera path, we randomly pertrbed these key-points by different amonts, and fitted a spline crve to them. We qantified the added pertrbation added as a fraction of its radis. or each pertrbation amont, we generated 0 sets of foregrond paths Simlation Experiments We ran three main experiments on or generated data. Experiment : In this experiment (igre 3, top row), we analyzed or precision and recall as a fnction of foregrond path pertrbation 2. Here, we searched for strong matches in a naive manner ( 4.). Experiment 2: or or second experiment, we analyzed how strong backgrond frame matches are temporally distribted. We performed this experiment for different amonts of foregrond path pertrbations (igre 4). Experiment 3: inally, for or third experiment (igre 3, bottom row), we analyzed for a given path pertrbation, how does or performance vary as we search for strong backgrond frame matches in a more greedy way ( 4.2) Main indings We now report or main findings from these experiments Matching Alone is Not Enogh Relying prely on strong matches between the foregrond and the backgrond frames reslts in poor accracy (ble plots in igre 3, top row). This is becase for larger disparity between camera paths, there may be no strong matches 2 As we know the grond trth, we can compte the precision and recall based on their standard definitions.

6 Percent Precision 00 Correspondence nly Correspondence nly Correspondence & Tracking (rd.) Correspondence & Tracking (rd.) Correspondence & Tracking (rd. & Rvrs.) Correspondence & Tracking (rd. & Rvrs.) Percent Pertrbation Percent Pertrbation Percent Precision 70 Correspondence (Corr.) 65 Correspondence (Corr.) Corr. + Tracking orward (rd.) Corr. + Tracking orward (rd.) 50 Corr. + Tracking rd. / Reverse(Rvrs.) Corr. + Tracking rd / Reverse (Rvrs.) Corr.+ Tracking rd/rvrs & Local Scan Corr.+ Tracking rd/rvrs & Local Scan Corr.+ Tracking rd/rvrs linear Corr.+ Tracking rd/rvrs linear ackgrond rame Step Size ackgrond rame Step Size 0 igre 3: Precision (top left) and Recall (top right) plots with increasing amonts of camera path pertrbation. le graph shows reslts obtained by only sing frame matching (corr.), while green and red graphs represent also sing forward (frd.) and reverse (rvrs.) tracking of foregrond pixels in frames that do not have strong matches in the backgrond video. Precision (bottom left) and Recall (bottom right) are shown as a fnction of backgrond frame step-size. The black and prple graphs se local search in the backgrond, and both backgrond and foregrond videos. available, i.e., captre completeness ( 3..2) may not always hold. However, as can be seen from the red plots in the top row of igre 3, adaptively switching between backgrond sbtraction of registered frames, and tracking the foregrond pixels based on their latest available location information sbstantially improves the performance Temporal Smoothing Improves Accracy Perform a forward tracking pass along with a backgrond tracking pass for frames with no strong backgrond match can improve the framework accracy (igre 3, top row). This is becase for the foregrond frames for which we do have strong backgrond matches, the noisy pixels appear and disappear erratically, while the foregrond pixels are mch more consistent over frames. Therefore, fsing information of foregrond pixels obtained over the forward and reverse tracking passes reslts in accracy improvement at constant additional complexity cost Matches are Temporally Clstered We analyzed how strong backgrond frame matches are temporally distribted in the backgrond video. This analysis corroborates or theoretical reslt of Theorem 4.6 which sggested that several strong matches are likely to be cls Percent Recall Percent Recall oregrond rame Index Pertrbation = 5% Pertrbation = 0% Matched ackgrond rame Index oregrond rame Index Matched ackgrond rame Index igre 4: or each foregrond frame, we show how likely a backgrond frame is to be a strong match. Here white implies high likelihood. Reslts for 5% & 0% pertrbation are shown. Greater pertrbation amonts follow similar trends. tered together. This can be observed from igre 4, where light regions show pairs of foregrond and backgrond frames that form a strong match (averaged over 0 independent trials). As can be seen, light regions clster together, lending empirical evidence to or theoretical reslt Selective Coarse-to-ine rame Search Works Strong matches for foregrond frames can be accrately fond by first sparsely searching a well-distribted set of backgrond frames, followed by a dense search arond backgrond frames that match well with the given foregrond frame. This is in accordance with Algorithm 2, and is corroborated by the reslts in igre 3 (bottom row). As shown, with the forward and reverse tracking, and local search (dashed prple graph), the algorithm s precision and recall are nearly naffected. Specifically, the precision and recall are both above 90% for step sizes all the way p to k = 0. Note that as the step size increases, it significantly otperforms all approaches that do not leverage local search. or these experiments, the nmber of video frames was 00 each. This implies that or framework can achieve greater than 90% accracy while taking time only linear in the lengths of the backgrond and foregrond videos. 7. Experiments & Reslts We now present reslts of or framework tested on real videos taken by mltiple sers. 7.. Explored Test Scenarios We performed a set of experiments designed to test different challenges for handheld cameras. Sample video frames and otpt from for these are presented in igre 6. The fll videos are provided in the spplementary material. oregrond bject Geometry: The first five reslts in figre 6 are presented in the order of increasing foregrond complexity. As shown, or framework performs well for a range of object geometries inclding mostly convex (reslt ) to qite non-convex (reslt 5).

7 Precision Video Video 2 Video 3 Video 4 Video 5 Video 6 Video 7 Video 8 Recall V V 2 V 3 V 4 V 5 V 6 V 7 V 8 A A A Table : Algorithms clock times in mintes. Here A, A2, and A3 represent GMMs, saliency, and or proposed method. mlti-frame co-segmentation based approach [7] Video Video 2 Video 3 Video 4 Video 5 Video 6 Video 7 Video 8 igre 5: Precision and Recall plots of or proposed method (red) and five comparative approaches (explained in 7.2). Indoors Verss tdoors: In igre 6, reslts to 4 were captred indoors, while reslt 5 was captred otdoors in daylight. Even within the indoor environment the ato white-balancing featre of mobile cameras can reslt in sbstantial variation in the global illmination of the video. r framework is able to handle these variations qite well. Mltiple oregrond bjects: r framework is agnostic to the nmber of foregrond objects present in the video. This is shown in reslt 6 of igre 6, where we were able to sccessflly extract all three foregrond objects. Moving oregrond bjects: We captred videos for both simple and articlated moving objects, shown in the last two reslts of igre 6. Not only do we preform well for the limited movement of the table fan, we were also able to extract the articlated hman motion Comparative Reslts We compared or algorithm with five alternate foregrond extraction methods. These methods are explained below. In the first comparison (ble plot), we sed a Gassian Mixtre model (GMM) to learn the global backgrond appearance which was sed for backgrond sbtraction [22]. In the second comparison (green plot), we manally provided a foregrond mask for the first frame of the foregrond video to learn a foregrond GMM. This model was sed to sbtract backgrond from all the foregrond frames. r third comparison (green plot) was with the previosly proposed approach of video matching [9], and highlights its limitations de to its strong assmptions abot the foregrond and backgrond camera paths being the same. r forth comparison (orange plot) was with a perceptal saliency based approach for the foregrond frames [5]. r fifth and final comparison (yellow plot) was with a 7.2. Precision & Recall We manally extracted the foregrond in every 0 th frame of the test videos presented in igre 6. Using this, we compted the nmber of tre/false positives and negatives. As shown in igre 5, or approach (red plot) gives better recall as well as comprehensively otperforms previos approaches on precision Time Comparisons Clock times taken by three of the five considered algorithms on the test videos are shown in Table. Note that the remaining two algorithms from igre 5 are derivatives of the three considered methods in Table, with times similar to the first two rows of the table. As can be observed, or proposed method takes the least amont of processing time while generating sperior reslts compared to all the considered alternatives. 8. Conclsions & tre Work In this paper, we identified a novel instance of the backgrond sbtraction problem focsing on extracting nearfield foregrond objects captred sing handheld devices. We posed or problem as efficient matching of foregrond and backgrond videos tracing significantly different camera trajectories. We proposed an approximate algorithm for this problem with provable theoretical garantees that admit a significant comptational speed-p. Using insights from or theoretical model, we proposed an end-to-end comptational framework, and presented controlled simlation experiments to thoroghly analyze its object-extraction accracy. Moreover, we presented reslts of or framework on mltiple hand-held videos, and compared them with five alternate approaches. Going forward, we plan to also learn the shape and appearance of the extracted foregrond object(s). We expect this to allow s to improve or accracy frther, while reqiring even fewer matches. References []. Aghazadeh, J. Sllivan, and S. Carlsson. Novelty detection from an ego-centric perspective. IEEE CVPR, pages , 20. 2

8 Reslt Reslt 2 Reslt 3 Reslt 4 Reslt 5 Reslt 6 Reslt 7 Reslt 8 [2]. Aghazadeh, J. Sllivan, and S. Carlsson. Mlti view registration for novelty/backgrond separation. CVPR, [3] H. ay, T. Tytelaars, and L. Van Gool. Srf: Speeded p robst featres. Compter Vision ECCV 2006, pages 4 47, , 5 [4] A. evilacqa and P. Azzari. High-qality real time motion detection sing ptz cameras. AVSS 06, [5] R. Carceroni,. Pa da, G. Santos, and K. Ktlakos. Linear seqence-to-seqence alignment. CVPR, [6] Y. Caspi and M. Irani. Spatio-temporal alignment of seqences. PAMI, [7] D. Comanici, V. Ramesh, and P. Meer. Real-time tracking of non-rigid objects sing mean shift. CVPR, , 5 [8] M. ischler and R. olles. Random sample consenss: a paradigm for model fitting with applications to image analysis and atomated cartography. Commnications of ACM, 24(6):38 395, [9] C. Glasbey and K. Mardia. A review of image-warping methods. JAS, 25(2):55 7, [0] M. Granados, K. I. Kim, J. Tompkin, J. Katz, and C. Theobalt. ackgrond inpainting for videos with dynamic objects and a free-moving camera. In ECCV, [] J. Harel, C. Koch, and P. Perona. Graph-based visal saliency. In NIPS, pages , [2] R. Hartley and A. Zisserman. Mltiple view geometry in compter vision. 2, [3] E. Hayman and J. Eklndh. Statistical backgrond sbtraction for a mobile observer. CVPR, [4] M. Isard and J. MacCormick. ramble: A bayesian mltipleblob tracker. ICCV, 2:34 4, [5] L. Itti, C. Koch, and E. Niebr. A model of saliency-based visal attention for rapid scene analysis. PAMI, 20(), , 7 [6] A. Kowdle, S. N. Sinha, and R. Szeliski. Mltiple view object cosegmentation sing appearance and stereo ces. In ECCV, [7]. Meng, H. Li, G. Li, and K. Ngan. bject co-segmentation based on shortest path algorithm and saliency model. ECCV, , 7 [8] K. Papotsakis and A. Argyros. bject tracking and segmentation in a closed loop. AVC, [9] P. Sand, S. Teller, et al. Video matching. ACM Transactions on Graphics, 23(3): , , 7 [20] S. M. Seitz and C. R. Dyer. Complete scene strctre from for point correspondences. In ICCV, [2] Y. Sheikh,. Javed, and T. Kanade. ackgrond sbtraction for freely moving cameras. CVPR, [22] C. Staffer and W. Grimson. Adaptive backgrond mixtre models for real-time tracking. CVPR, 2, , 7 [23] J. Sn, W. Zhang, X. Tang, and H. Shm. ackgrond ct. IEEE ECCV, pages , [24] P. Torr, R. Szeliski, and P. Anandan. An integrated bayesian approach to layer extraction from image seqences. PAMI, 23, [25] K. Toyama, J. Krmm,. rmitt, and. Meyers. Wallflower: Principles and practice of backgrond maintenance. CVPR, igre 6: Reslts captring varios parameters inclding object geometries, illmination, mltiple, and moving foregrond objects are presented. Here,, and denote backgrond, foregrond, and otpt videos respectively.