Variational Recursive Joint Estimation of Dense Scene Structure and Camera Motion from Monocular High Speed Traffic Sequences

Transcription

1 Variational Recursive Joint Estimation of Dense Scene Structure an Camera Motion from Monocular High Spee Traffic Sequences Florian Becker, Frank Lenzen, Jo rg H. Kappes an Christoph Schno rr HCI an IPA, Heielberg University, Speyerer Str. 6, 695 Heielberg, Germany Abstract We present an approach to jointly estimating camera motion an ense scene structure in terms of epth maps from monocular image sequences in river-assistance scenarios. For two consecutive frames of a sequence taken with a single fast moving camera, the approach combines numerical estimation of egomotion on the Eucliean manifol of motion parameters with variational regularization of ense epth map estimation. Embeing this online joint estimator into a recursive framework achieves a pronounce spatio-temporal filtering effect an robustness. We report the evaluation of thousans of images taken from a car moving at spee up to 00 km/h. The results compare favorably with two alternative settings that require more input ata: stereo base scene reconstruction an camera motion estimation in batch moe using multiple frames. The employe benchmark ataset is publicly available.. Introuction Overview an Motivation. Computer vision research has a strong impact on river assistance technology. Besies esigning eicate etectors for specific object classes [4, 7], current major trens inclue low-level estimation of ense scene structure from stereo sequences [22], the transition to monocular imaging sensors [23, 5], an context-base 3D scene representation an labeling supporte by high-level assumptions an constraints [24]. This paper focuses on the low-level task to jointly estimate ense scene structure an egomotion uner minimal assumptions, averse conitions an requirements, that are typical for river assistance scenarios see Fig. : Online joint estimation from only two consecutive frames in view of on-boar implementations later on; No assumptions about scene structure in orer to cope with arbitrary scenes; No aitional input (e.g. oometer reaings) besies internal camera parameters estimate offline (calib.); c 20 IEEE Figure. Best viewe in color. Upper row: Two consecutive frames (size pixels) of the Ben sequence with large isplacements up to 35 pixels inuce by a fast moving camera. Lower row: Our approach jointly estimates, from sparse noisy isplacement estimates, ense epth maps (left) an camera motion in an online recursive framework. Right: Reconstruction of ense scene structure base on the epth maps from the camera s viewpoint, an the corresponing camera track. Ability to cope with large isplacements inuce by a fast moving camera; Comprehensive evaluation using image sequences recore in real scenarios. In this connection, the major issue to be aresse concerns the esign of an integrate approach that ensures sufficient regularization to achieve robust an accurate estimation, without compromising real-time capability through unrealistically complex computations. Our approach therefore combines highly accurate numerics on the low-imensional Eucliean manifol in orer to isambiguate an track translational an rotational egomotion from ill-pose two-frame isplacement estimates, with less accurate variation moels for estimating highpublishe by the IEEE Computer Society

2 imensional scene structure, leaing to efficient overall inference. Applying the resulting online joint estimator within a recursive preiction-estimation loop to an image sequence achieves favorable spatio-temporal filtering an increase robustness. The estimates compute with our approach provie a basis for subsequent tasks like obstacle an collision warning, an further relate problems of avance scene analysis, to be consiere in future work. Relate Work. Most approaches to scene reconstruction rely on stereo imaging or multiple view reconstructions in batch moe. Stereo set-ups [22, 6] are only relevant for sensing closeup ranges at low spees, ue to the small baseline in river assistance scenarios, an are less attractive than just a single camera from the technological system oriente viewpoint. Factorization [9] an bunle ajustment [2] have become a mature technology for jointly etermining camera an scene structure from tracke features. While this requires to accumulate several frames an more expensive numerics, recent local an more efficient approaches, e.g. for visual oometry [2, 4], entail only sparse representations of scene structure. Further recent work on the reconstruction of accurate epth maps from arbitrary multiple views inclues [5, 8]. These works however require the camera motion to be etermine in a preceing step using feature tracking. Other relate approaches only allow for camera translation but no rotation [23], or estimate the epipole but require images to be aligne with respect to a common reference plane [0]. An attractive alternative woul employ irect featureto-epth mappings, learne offline from groun truth atabases [7]. Besies the tremenous effort necessary to compile a sufficiently large set of in particular, far fiel groun truth ata, we on t currently know how such an approach generalizes to arbitrary scenes, an if it can compete with reconstructions that rely on measurements efficiently estimate online, as in our case. Contribution an Organization. We present an approach that estimates from a monocular high-spee image sequence of arbitrary static scenes both camera motion an ense scene structure (epth maps), using noisy sparse isplacements compute from two consecutive frames at each instant of time. The approach combines, by joint optimization, geometric integration over the Eucliean manifol SE 3 for incremental motion parameter estimation, with large-scale variational epth map estimation, subject to spatial an short-time temporal regularization. The novelty of our approach is ue to the ability to recover ense scene structure an egomotion from monocular sparse isplacement estimates within a truly recursive online estimation framework. Sect. 2 provies an overview of the overall approach an specifies unerlying assumptions an approximations, followe by etailing each component of our metho in Sect. 3. We report in Sect. 4 results of an evaluation of our approach using thousans of real images provie by a novel atabase [3], that aims at proviing a benchmark for computer vision algorithms in the context of automotive applications. All image ata is available online. Moreover, we show that our approach compares favorably to results compute with less restricte approaches (stereo, bunle ajustment) using public implementations (Voooo Camera Tracker 2 v..0b) an [20, 6], to ensure reproucibility of all results. 2. Problem Statement, Approach (Overview) Preliminaries. We aopt the common concepts of multiple view geometry [8]. We assume the internal camera parameters to be known (offline calibration) an enote incremental external parameters corresponing to frame k by C k = (R k, h k ), moving the camera from its position at time k, see Fig. 2. The manifol M := SE 3 of Eucliean transformations C = (R, h) M, parametrize by rotations R an translations h, is ientifie with the matrix Lie group { ( ) } R h G := Q = 0 : R SO 3, h R 3, () an SO 3 enotes the group of proper rotations. T C M an T Q G enote the tangent spaces of M, G at C M, Q G, respectively. The Lie algebra se 3 of SE 3 is given by ( 0 ω3 ω 2 ω 3 0 ω ω 2 ω 0 ), se 3 = { W = ( ) ˆω v 0 0 : ω, v R 3 }, ˆω := (2) where so 3 ˆω enotes the Lie algebra of SO 3 ientifie with the linear subspace of skew-symmetric matrices. We equip SE 3 with the Riemannian metric W, W 2 G := ˆω, ˆω 2 + c G v, v 2, c G > 0, (3) for all W i se 3, i =, 2, where, on the right-han sie enotes the canonical matrix an vector inner prouct, respectively, an c G is a constant parameter scaling the rotational vs. the translational part. Note that unlike for general Riemannian metrics, the metric (3) oes not epen on Q G, hence is the same for all tangent spaces T Q G, justifying the notation, G. The exponential mapping Exp: se 3 SE 3 that iffeomorphically maps tangent vectors close to 0 onto the manifol within a neighborhoo of the group ientity I, can be Publishe by the IEEE Computer Society

3 Figure 2. A scene point is efine by image coorinates x an epth k (x) in the coorinate system of camera k. Its projection moves by u k (x) to x when the camera is rotate an translate backwar by C k = (R k, h k ). compute in close form, Exp [( ˆω v 0 0 )] ( ) = R(ω) Q(ω)v, (4) 0 R(ω) = I + sin( ω ) ˆω + cos( ω ) ω ω 2 ˆω 2, (5) Q(ω) = I + cos( ω ) ω sin( ω ) ω 2 ˆω + ω 3 ˆω 2. (6) Problem Statement. Let Ω R 2 be the image omain an I 0:k := {I 0, I,..., I k } a given image sequence of frames I i : Ω R, measure at times i {0,..., k} with cameras C 0:k. We wish to jointly estimate in a recursive manner both C 0:k an a sequence 0:k of epth maps i : Ω R + that assign to each image point x Ω its epth i (x) along the viewing ray, up to a common global unknown scale factor see Fig. 2. The ifficulty of this problem is (i) ue to a monocular river assistance scenario (see Fig. ) inucing less favorable motion parallax, (ii) a fast moving camera leaing to isplacements of consecutive frames up to 35 pixels (frame size pix.), an (iii) a recursive online processing moe that upates the camera parameters an epth map base on two consecutive frames only. Approach: Overview. The natural approach to this problem is to consier sequences of state variables X 0:k = (C 0:k, 0:k ) an observations Y 0:k = u 0:k, together with probabilistic moels of state transitions p(x k X k ) an the observation process p(y k X k ) uner Markovian assumptions, in orer to recursively estimate X k base on the posterior marginal istribution p(x k Y 0:k ) (cf., e.g. [2]). Approximations to this general approach are inevitable, however, ue to the nonlinearity of the unerlying processes, ue to the high imensionality of epth maps k an isplacement fiels u k (cf. Fig. 2), an ue to a strict requirement for computational efficiency impose by the scenario shown by Fig.. We aopt therefore the variational moeling perspective as accepte alternative in situations where sampling base approaches are too time consuming (cf., e.g. []). Accoringly, as etaile in Sect. 3, we evise Gaussian approximations p(y k X k ) = N (u k ; µ k u, Σ k u) an N ( k ; ˆµ k, ˆΣ k ) for the high-imensional observations Y k = u k an states k, respectively, that sufficiently take into account uncertainties ue to the aperture problem an the viewing geometry (regions aroun the epipole). Evaluating the former Gaussian entails routine parallel coarse-tofine signal processing, whereas the latter aitionally takes into account spatial an temporal context (regularization) in term of preictions ˆµ k, ˆΣ k. N ( k ; ˆµ k, ˆΣ k ) is complemente by a local Gaussian moel N M (C k ; C k, σc 2 I) of motion parameters on the tangent space of the Eucliean manifol M at C k (cf. [6]), to form together an approximation of the state transition p(x k X k ), X k = (C k, k ) = (R k, h k, k ). Putting all components together, we efine an compute our upate as moe of the posterior marginal approximation p(x k Y 0:k ) p(y k X k )p(x k X k ). Concerning the motion parameters C k, we prefer working irectly on M using establishe concepts of numerics [], rather than to represent the two-view geometry by the essential matrix an to recover C by aitional factorization [9]. 3. Approach: Details Our approach jointly estimates egomotion an a ense epth map from a monocular image sequence. The recursive formulation requires constant amount of storage an aims at real-time applications. Large isplacements inevitable in the consiere scenario are hanle in the common coarse-to-fine manner [5]. Uncertainty of observations an epth estimates are hanle by probabilistic moels. 3.. Observation Process We etail the observation process p(y k X k ), with state variables X k = (C k, k ) (camera, epth map) an the camera C k given by C k in the previous frame an the egomotion parameters (R k, h k ). To simplify notation, we refer to frame k with primes (e.g. C ) an temporarily rop inices k an k. Using the known internal camera parameters, we uno the corresponing affine transformation of the image plane (cf. [8]) an enote the normalize image coorinates by x Ω R 2. Note that all relate quantities like isplacements, means an covariance matrices have to be transforme as well. To keep the notation simple, however, we only refer to normalize quantities in what follows. Publishe by the IEEE Computer Society

4 Any scene point (x) ( x ) at epth along the viewing ray ( x ) projects to the image point with inhomogeneous coorinates x. We enote this projection of scene points (X, X 2, X 3 ) in camera C by P C, P C (X, X 2, X 3 ) := ( ) X. (7) X 3 X 2 Consier any two subsequent points in time an the camera motion C C given by parameters (R, h). The motion inuces an apparent motion (R, R h) of scene points ( ) ( (x x x ) (x) = R ) ( ( ) (x x ) ) h. (8) Now we efine the isplacement u(x) in the image plane (see Fig. 2) by x = x u(x). (9) Using eqns. (7) an (8) we obtain u(x; R, h, ) = x P C ((x)r ( ) ) x + h. (0) Simple transformations show that x an x are corresponing points w.r.t. the essential matrix (cf. [8]) E = R ĥ, i.e. ( x ) E ( ) x = 0, () which implicitly efines the epipolar line in C corresponing to x for fixe R, h. Observations Y correspon to estimates û(x) of the isplacements (0) for all x Ω using the Lucas-Kanae metho [3], ( ( ( t û(x) := Σ u (x) G ρ (x) I(x) )( I(x) ))), (2a) ( ( ( I(x) )( ) )). Σ u (x) := G ρ (x) I(x) (2b) Here, G ρ (x) enotes element-wise Gaussian convolution of the subsequent matrix comprising partial erivatives t, ( ) x x2 := of the image sequence function I(x, t). They are estimate by 3 3 binomial filters an first-orer ifferences erive by linearizations at time k. Likewise, we choose a rather small smoothing kernel of size ρ = 2 pix., leaing to a fast processing stage. We point out that stronger regularization (smoothing) is not necessary as the embeing multiscale framework an the state preiction (see Sect. 3.2) ensure small incremental isplacements u(x). As for the unknown observation process p(y k X k ), our ansatz is p(y k X k ) = N (û k ; µ k u, Σ k u), (3) where µ k u is compose position-wise of µ k u(x) := u(x; R k, h k, k (x)) ue to eqn. (0), an Σ k u is a blockiagonal covariance matrix with component matrices (2b). Figure 3. Detaile view of an image frame an an ellipse representation of the estimate flow uncertainty Σ k u(x). Highly texture regions (upper right) can be correctly istinguishe from locations with low confience ue to low signal-to-noise ratio (left) an image eges (aperture problem; mile). Note that the efinition of µ k u makes explicit the conitioning on the state parameters X k = (C k, k ) = (R k, h k, k ). Moel (3) only approximates the true unknown observation process (2a). The uncertainty of observations u k is moelle by Σ k u an internally represente by the precision matrices (Σ k u) (cf. eqn. (2b)). Hence, homogeneous image regions an the aperture problem are represente as rank-0 an rank- matrices, respectively, (see Fig. 3) an thus can be correctly accounte for within the overall recursive estimation framework see Sect State Transition an Preiction We etail the state transition moel p(x k X k ) for the state variables X = (C, ). Camera. We take C k =: Ĉ k both as preiction Ĉk of C k an as mean of the probabilistic moel C k p(c k C k ) = N M (C k ; C k, σci) 2 (4) ( exp 2σC 2 ist M (C k, C k ) 2), (5) where ist M (, ) enotes the geoesic istance on the Eucliean manifol M = SE 3. The preiction Ĉk = C k is justifie by the fast frame rate. Distribution (4) represents an isotropic Gaussian istribution of ranom points C M aroun C k M (cf. [6]). σ C is a user parameter that we set an keep constant throughout all experiments. Moel (4) will be further etaile in connection with inference in Sect Depth Map. The preicte epth map ˆ k is compute by transporting k by the motion parameters ( ˆR k, ĥk ) = (R k, h k ). To obtain preicte epth values ˆ k (x) at gri positions x in frame k, we approximately infer corresponing positions x in frame k using eqns. (9) an (0), ( ( ) ) x P C k (x)r k x + h k. (6) We bilinearly interpolate k at x to obtain (x ) an the accoring space point (x ) ( ) x in camera C. Its Publishe by the IEEE Computer Society

5 x ĥ x ˆR ˆ(x) ( ) ( x = (ˆR) T (x ) ( ) x ĥ) (. Step: map x to x P C (x)r ( ) ) x +h 2. Step: estimate ˆ(x) ( ) ( x = (ˆR) T (x ) ( ) x ĥ ) Figure 4. Preiction ˆ of the state variable. Using ( ˆR, ĥ) := (R, h ) an epth estimation, we can map the coorinate system x of the current camera set up to the previous x. With this mapping, we approximate ˆ using (8) with (R, h) = ( ˆR, ĥ). transition to camera C is given by (8), an we efine the epth (x) as preiction ˆ k (x). Figure 4 illustrates this process. Note that eqn. (6) only is an approximation because we o not know the correct argument k (x) as require by eqn. (0), an that ˆ k = ˆ k (x; X k ) = ˆ k (x; R k, h k, k ) (7) is a function of X k = (C k, k ). We assume that a local variance map σ k of k in the previous frame is known. In Sect. 3.3 we will etail on how this information is obtaine. Preiction errors of the epth map are accounte for by assuming a constant increase σ of the local variance, which is transporte ientical to k, i.e. (ˆσ k(x))2 = (σ k (x )) 2 + σ 2. Experiments confirm this assumption, see Fig. 7. Base on this relationship, we make a Gaussian ansatz as approximate probabilistic moel of k, p( k X k ) exp ( f ( k ; ˆ k, ˆσ k ) ). (8) The energy functional f inclues a prior penalizing the eviation from the preiction ˆ k an a spatial smoothness prior, f ( k ; ˆ k, ˆσ ) k = ( k (x) ˆ k ) 2 (x) 2 Ω ˆσ k(x) + k (x) 2 x. (9) Here, we use continuous notation to facilitate interpretation of the terms. After iscretization, k, ˆ k, ˆσ k R Ω are vectors inexe by gri positions x Ω, an we re-use the symbol to enote the matrix : R Ω R 2 Ω approximating the graient mapping. Furthermore, we efine the preicte covariance matrix of ˆ k as Ŝk := Diag(ˆσk (x))2. Inserting the iscretize functional f (9) into (8) an ignoring normalizing constants, we obtain after multiplying out an rearranging terms using some basic matrix algebra, p( k X k ) N ( k ; ˆµ k, ˆΣ k ), with (20a) ˆµ k = ˆΣ k (Ŝk ) ˆk, ˆΣ k = ( (Ŝk ) + ). (20b) Notice that the prior ˆ k fixes a single, but arbitrary global scale of an h that cannot be inferre from monocular sequences State Upate Having observe Y k = û k in terms of the isplacement vector fiel (3) that epens on the unknown state variables X k = (C k, k ) = (R k, h k, k ), we upate the state by estimating X k as moe of the istribution p(x k Y 0:k ) p(y k X k )p(x k X k ) = N (û k ; µ k u, Σ k u) N M (C k ; C k, σ 2 CI) N ( k ; ˆµ k, ˆΣ k ) base on eqns. (3), (4) an (20). Accoringly, the objective function f(, C) := log p(x k Y 0:k ) ecomposes into f(, C) = f u (, C) + f C (C) + f () with f u (, C) = 2 (ûk µ k u) ( Σ k u) (û k µ k u), (22a) f C (C) = 2σC 2 ist M (C k, C k ) 2, f () = 2 (k ˆµ k ) (ˆΣk ) ( k ˆµ k ). (22b) (22c) Note that µ k u(x) = u(x; R k, h k, k (x)) epens nonlinearly on R k, h k an k. Our approach to solving min C, f(, C), C SE 3, R Ω 0 (23) consists in alternating upate steps for C an, etaile below, embee into a multiscale framework. Camera Motion Parameter. We consier the partial minimization of any functional f specifie by (22) with respect to the motion parameters C = (R, h). Aopting the ientification (), the problem reas min f(q), Q G. (24) Q Let Q (i), i = 0,, 2,... enote the minimizing sequence to be etermine. Given Q (i) for some i, we etermine Q (i+) = ϕ(t; Q (i) ) in terms of a step size t an an oneparameter flow ϕ(t; Q i ) G, with ϕ(0; Q i ) = Q (i) an ϕ(0; Q (i) ) T Q (i)g, given by ϕ(t; Q (i) ) := Q (i) Exp(tW (i) ), W (i) se 3. (25) Step size t is compute by line search along ϕ(t; Q i ) (cf. []) in orer to locally minimize f, t arg min t>0 f( ϕ(t; Q (i) ) ). (26) Matrix W (i) etermining the flow (25) is compute by consiering t f( ϕ(t; Q (i) ) ) t=0 = f ( ϕ(0; Q (i) ) ), ϕ(0; Q (i) ) = f(q (i) ), Q (i) W (i), (27) Publishe by the IEEE Computer Society

6 where f enotes the (orinary) graient of f with respect to the ambient matrix space R 4 4. The graient G f on the manifol G is obtaine from the equation (cf. []) G f(q (i) ), Q (i) W G = f(q (i) ), Q (i) W, W se 3, (28) which is equivalent to Q (i) G f(q (i) ), W G = Q (i) f(q (i) ), W for arbitrary W se 3. Hence, ( ) G f(q (i) ) = (Q (i) ) â c b G, (29) 0 0 ( â b ) ( 0 0 = Πse3 Q (i) f(q (i) ) ), (30) where Π se3 enotes the orthogonal projection ( onto ) the linear subspace (2). Choosing W (i) â c b = G in (27) 0 0 shows ue to the equalities in (28) an (29), t f( ϕ(t; Q (i) ) ) t=0 = ( â c b G 0 0 ) 2 G 0, (3) Figure 5. Representative performance of the minimization approach (32), (33) in a real scenario. The objective function is effectively minimize after few iterations at each resolution level (soli curve), as oppose to graient escent that converges slowly (ashe curve). i.e. that the flow ϕ(t; Q (i) ) given by (25) ecreases f. Depth Map. A escent irection of f() = f(c, ) in (23) is given by δ (i) := ( B (i)) f( (i) ) (32) with some positive efinite, symmetric perturbation matrix B (i). Then for any f 2 > 0, there is a t > 0, such that f( (i) + tδ (i) ) < f((i) ), see e.g. []. To ensure the element-wise non-negativity of (i), we efine a quality function q() := f() + χ 0 (), where χ 0 () is the characteristic function of the set R Ω 0, an utilize a line search metho to etermine a nearoptimal t such that t arg min t>0 q( (i) + tδ (i) ). The choice of B (i) is crucial. Graient escent, i.e. B (i) = I, turne out to be unsatisfactorily slow ue to the high imension of an the spatial variable interactions ue to (20). Positive efiniteness of the Hessian Hf of the objective is not guarantee ue the non-convexity of the projections (0) in the observation term (3). Thus, a stanar Newton metho (B (i) = Hf) might iverge. Therefore, we propose the choice (with ɛ > 0) B (i) := γ(hf u ( (i), C (i) )) + Hf ( (i) ) + ɛi, (33) where γ(s) ajusts the eigenvalues of the matrix S by replacing them by their absolute values to guarantee positive semi-efiniteness. Due to the form of (20) an the thir term in (33), the positive efiniteness of B (i) is guarantee. Figure 5 emonstrates the fast convergence of our metho within only few iterations uner real conitions. Along with the results in Sect. 4, this justifies the choice (33). Note that ue to (), searching along parameterizes a search along the epipolar lines efine by fixe R, h. Figure 6. Best viewe in color. Left: input frame, right: calculate epth map, bottom: triangulation of the scene, camera position an irection (green camera symbol), camera track (re, ots inicate subsequent positions). Due to the moving trees, homogeneous sky regions between trees are naturally assigne to the trees (in terms of epth) in the temporal context, rather than to the horizon an in contrast to the center region. Local Depth Variance. We approximate the local variance σ k of the epth map by the secon erivatives of f in (C k, k ), boune to 0, 2 (σ(x)) k 2 = max{0, f u(c k, k )} + f ( k ). (x) 2 (x) 2 Along with the compute k, this information is incorporate as prior (cf. Sect. 3.2) in the estimation of k+ to ientify regions with high uncertainty cause by the parallax an/or the lack of image features. 2 Publishe by the IEEE Computer Society

7 os to our ataset, see Sect Concerning egomotion, we applie the Voooo Camera Tracker (VCT) that, unlike our approach, works in batch moe an therefore can be expecte to return more precise results, see Sect Results (σk ) of epth variance σk for increasfigure 7. Histogram pf Ω ing k, isplaye as contour plot. The lines ten to the left an thus inicate that estimation uncertainty is effectively reuce, espite online processing with only two frames at each instant of time. Depth Maps an Egomotion. Figures an 6 show representative reconstructions. Comments are given in the captions. Further results are available online3 as vieos. Uncertainty Reuction. Figure 7 epicts several fixe (σ k ) of the variance σ k of levels of the histogram pf Ω k epth, taken over the whole image plane, as a function of the frame inex k (orinate). The resulting lines ten to the left an thus emonstrate that our approach significantly reuces uncertainty of the epth estimation within a perio of about 50 frames. Temporal Filtering. The preiction step exploite by (k ˆk )2 in (9) is essential for robust epth estimation. Figure 8 shows for two ifferent scenes epth maps estimate without an with prior in the left an right column, respectively. The ifferences are striking an show that just relying on the observations yiels corrupte estimates Comparison to Stereo Figure 8. Best viewe in color. Importance of regularization over time (preiction prior) emonstrate for two ifferent scenes, top an bottom row; left without prior, right with prior. Exploiting temporal context through the prior is essential, in particular aroun the epipole. Omitting the prior leas to corrupte estimates. As a baseline for epth estimation, we use an implementation of [20] inclue in the Milebury MRF Library4 (parameters: Birchfiel-Tomasi metho with α-βswap, 80 isparities, L -regularization parameter 0.5), an alternatively the LIBELAS library, see [6] (efault param.). Figure 9 (an further results3 ) reveals that our monocular approach provies a competitive reconstruction Comparison to Voooo Camera Tracker (VCT) 4. Experiments 4.. Data Sets an Performance Measures Data Sets. We evaluate the algorithm propose in Sect. 3 by means of a novel atabase [3] of image sequences recore from of a car riving at high spee in an everyay environment. This atabase was compile for the evaluation of stereo reconstruction algorithms an is available online. Thus, two camera recorings from a stereo rig are available, one of which only was use as input to our algorithm. Each image sequence consists of up to 400 gray-value frames of size pix., taken at 25 Hz. Below, we refer to the Ben (Fig. ) an to the Avenue sequence (Fig. 6). They are available online3 along with further sequences an results compile as vieos. Performance Measures. As a baseline for epth estimation, we applie two ifferent stereo reconstruction meth3 c 20 IEEE To evaluate egomotion estimation, we applie the VCT (parameters: free move, bunle ajustment with previous images, fixe internal parameters), that estimates camera parameters using bunle ajustment an tracke features. Figure 0 emonstrates for the Ben an Avenue sequences a remarkable agreement of our monocular online estimates. 5. Conclusion an Further Work We presente an approach to the estimation of ense scene structure an camera motion from monocular image sequences, taken from a camera positione insie a fast moving car. The approach optimizes the traeoff between moel expressiveness an computational efficiency. In particular, it works in an online two-frame moe an competes well with less esirable settings (stereo, bunle ajustment), as emonstrate by a comprehensive evaluation using real ata in ifferent scenarios. 4 Publishe by the IEEE Computer Society

8 Figure 9. Best viewe in color. Comparison of two stereo implementations ([20]: top row, [6]: center row) to our monocular approach (bottom row). Left column: iniviually rescale color maps for best epth visualization. Right column: same color maps for all three approaches. Stereo approaches suffer from low resolution (iscrete isplacements), resulting in erroneous partitions (e.g. trees an sky) an stair-casing effects (roa) that may cause problems at subsequent processing stages (scene analysis). Figure 0. Left&mile: camera trajectories for Avenue an Ben sequences, resp., returne by VCT (gray) an our approach (black). The tracks are uniformly scale to the overall camera movement (first to last frame). No rotational fitting was applie. Right: Eucliean istances between the trajectories, relative to the trajectory length. Differences are upper-boune by 3%. Our further work will focus on occlusion hanling an reliable segmentation of inepenently moving objects, an relate mi-level tasks of traffic scene analysis. Acknowlegements HCI is supporte by the DFG, Heielberg University an inustrial partners. Authors thank Dr. W. Niehsen, Robert Bosch GmbH. References [] P.-A. Absil, R. Mahony, an R. Sepulchre. Optimization Algorithms on Matrix Manifols. PUP, , 5, 6 [2] A. Bain an D. Crisan. Funamentals of Stochastic Filtering. Springer, [3] S. Baker an I. Matthews. Lucas-Kanae 20 Years On: A Unifying Framework. IJCV, 56(3), [4] M. Enzweiler an D. Gavrila. Monocular Peestrian Detection: Survey an Experiments. PAMI, 3(2), [5] D. Fleet an Y. Weiss. Optical Flow Estimation, pages Springer, [6] A. Geiger, M. Roser, an R. Urtasun. Efficient Large-Scale Stereo Matching. In Asian Conference on Computer Vision, Queenstown, New Zealan, November , 7, 8 [7] D. Gerónimo, A. López, A. Sappa, an T. Graf. Survey of Peestrian Detection for Avance Driver Assistance Systems. PAMI, 32(7), 200. [8] R. Hartley an A. Zisserman. Multiple View Geometry in Computer Vision. Cambrige Univ. Press, , 3, 4 [9] U. Helmke, K. Huper, P. Lee, an J. Moore. Essential Matrix Estimation Using Gauss-Newton Iterations on a Manifol. IJCV, 74(2), [0] M. Irani, P. Ananan, an M. Cohen. Direct Recovery of Planar-parallax from Mult. Frames. TPAMI, 24(), [] M. Joran, Z. Ghahramani, T. Jaakkola, an L. Saul. An Introuction to Variational Methos for Graphical Moels. Mach. Learning, 37, [2] K. Konolige an M. Agrawal. FrameSLAM: From Bunle Ajustment to Real-Time Visual Mapping. IEEE Trans. Robotics, 24(5): , [3] S. Meister, B. Jähne, an D. Konermann. An Outoor Stereo Camera System for the Generation of Real-Worl Benchmark Datasets with Groun Truth. Technical report, HCI, IWR, Heielberg University, 20. 2, 7 [4] E. Mouragnona, M. Lhuilliera, M. Dhomea, F. Dekeyserb, an P. Say. Generic an Real-Time Structure from Motion Using Local Bunle Ajustment. Image Vis. Comp., 27(8):78 93, [5] R. A. Newcombe an A. J. Davison. Live Dense Reconstruction with a Single Moving Camera. In CVPR, 200., 2 [6] X. Pennec. Intrinsic Statistics on Riemannian Manifols: Basic Tools for Geom. Measurements. JMIV, 25, , 4 [7] A. Saxena, S. H. Chung, an A. Y. Ng. 3-D Depth Reconstruction from a Single Still Image. IJCV, 76, [8] J. Stühmer, S. Gumhol, an D. Cremers. Parallel Generalize Thresholing Scheme for Live Dense Geometry from a Hanhel Camera. In CVGPU, [9] P. Sturm an B. Triggs. A Factorization Base Algorithm for Multi-Image Projective Structure an Motion. In ECCV, Cambrige, Englan, 996. Springer. 2 [20] R. Szeliski, R. Zabih, an D. Scharstein et al. A Comparative Stuy of Energy Minimization Methos for Markov Ranom Fiels with Smoothness-Base Priors. TPAMI, , 7, 8 [2] B. Triggs, P. F. Mclauchlan, R. I. Hartley, an A. W. Fitzgibbon. Bunle Ajustment A Moern Synthesis. LNCS, 883, [22] A. Weel, C. Rabe, an T. Vaurey et al. Efficient Dense Scene Flow from Sparse or Dense Stereo Data. In ECCV, volume 302 of LNCS, 2008., 2 [23] A. Weishaupt, L. Bagnato, an P. Vanergheynst. Fast Structure from Motion for Planar Image Sequences. In EUSIPCO, 200., 2 [24] C. Wojek, S. Roth, K. Schinler, an B. Schiele. Monocular 3D Scene Moeling an Inference: Unerstaning Multi- Object Traffic Scenes. In ECCV, 200. Publishe by the IEEE Computer Society