Wide-baseline Multiple-view Correspondences

Transcription

1 Wide-baseline Multiple-view Correspondenes Vittorio Ferrari, Tinne Tuytelaars, Lu Van Gool, Computer Vision Group (BIWI), ETH Zuerih, Switzerland ESAT-PSI, University of Leuven, Belgium Abstrat We present a novel approah for establishing multiple-view feature orrespondenes along an unordered set of images taken from substantially different viewpoints. While reently several wide-baseline stereo (WBS) algorithms have appeared, the N-view ase is largely unexplored. In this paper, an established WBS algorithm is used to extrat and math features in pairs of views. The pairwise mathes are first integrated into disjoint feature traks, eah representing a single physial surfae path in several views. By exploiting the interplay between the traks, they are extended over more views, while unrelated image features are removed. Similarity and spatial relationships between the features are simultaneously used. The output onsists of many reliable and aurate feature traks, strongly onneting the input views. Appliations inlude D reonstrution and objet reognition. The proposed approah is not restrited to the partiular hoie of features and mathing riteria. It an extend any method that provides feature orrespondenes between pairs of images.. Introdution In the last few years, several wide-baseline stereo (WBS) mathing algorithms, apable of finding orrespondenes between two images taken from substantially different viewpoints have appeared. In [,, 4, ] loal features are extrated independently from the two images, then haraterised by invariant desriptors and finally mathed. The power of these approahes is twofold. First, loal features bring tolerane to olusions. Seond, the onstrution proess and desriptors are invariant under affine deformations, allowing large viewpoint hanges, and under linear photometri transformations, allowing hanges in illumination. Tuytelaars and Van Gool [] onstrut small image regions around orners (nearby edges provide orientation and skew, while sale and streth are given by the extrema of a D affinely invariant funtion) and intensity extrema (based This researh was supported by EC projet CIMWOS. Tinne Tuytelaars is a Postdotoral Fellow of the Fund for Sientifi Researh Flanders (Belgium) on the intensity profile along rays emanating from it). Matas et al [4] propose regions as onneted omponents of pixels whih are all brighter or darker than all pixels on the region s ontour, or as regions bounded by yles of edge pixels. Baumberg [] desribes regions with their loation and sale determined by a multi-sale Harris orner detetor, while an adaptive proedure based on the seond moment gradient matrix reovers orientation and skew. Mikolajzyk and Shmid [] present a similar extration approah, but dealing with larger sale hanges. Lowe [6] proposes the SIFT detetor, where a feature s loation and sale are determined by extrema of a DoG funtion in sale spae, and its orientation by the dominant, loal image gradient orientation. It is less immune against viewpoint hanges, being only invariant under similarity transformations. Tell and Carlsson [7] offer an interesting alternative where a orner is haraterized by intensity profiles on line segments onneting it to other orners. This avoids the problem of onstruting affinely invariant regions, but it is less loal and does not produe loal shape desriptions. While these methods have foused on two views, this paper presents a method for obtaining wide-baseline mathes among a larger set of unordered images. The output is a set of region-traks, eah aggregating the image regions of a ertain physial surfae path along several views. The only alternative approah to date is reently due to Shaffalisky and Zisserman [5]. Wide-baseline N-view mathes are preious for a number of appliations. In D reonstrution (e.g. from unalibrated images), the D model of a sene ould be generated from still images rather than video.in objet reognition, appearane-based strategies ould be developed with a limited number of referene views [6], bridging in fat the gap with model-based approahes. In order to build region-traks, a natural idea is to apply an established WBS algorithm to view pairs and then integrate the outputs. A naive implementation is bound to fail, due to two main problems. The first is strutural: WBS algorithms return sets of mathes between pairs of views. Region-traks must be derived from this, even in the presene of mathing errors, whih an lead to ontradditory information about their omposition. The seond problem is statistial: suppose a physial region is visible in n views,

2 there is no guarantee that the image region will be extrated in all views. Suppose the probability p of extrating the region is the same, and independent, in every view where it is visible. The region will be extrated in its n views with probability p n. Thus, even assuming a perfet mather, the probability dereases exponentially with the number of views. Failures to math even lower the odds. We build upon a multi-sale extension of the WBS algorithm proposed in []. However, these problems are ommon to all WBS shemes based on independently extrated features. In pratie, given views v, v, v, the method typially finds many mathes between view pair (v, v ) and view pair (v, v ), but the two sets of mathes will often differ substantially. A smaller number of regions (typially half) are mathed over the three views. It goes without saying that taking more views, the situation deteriorates further (e.g.: only a few, if any, 5-view mathes an be found). Without an a priori ordering of the input image set, we start by applying [] to all pairs of views. This may seem omputationally expensive, but the appliation is meant for wide baseline onditions, in whih ase there will only be a limited number of views (typially 0 to 0). Otherwise, other and more appropriate approahes than WBS ould be applied. Besides, this allows to use all initial mathes, whih in turn reflets on the quality of the final results. In setion we propose an algorithm to onstrut lean, disjoint region-traks from the large set of all pairwise mathes, taking expliitly into aount the inevitable presene of mathing errors. The registration between the regions within eah region-trak is then improved by a novel way to refine the affine transformations between them (setion ). Unmathed regions are propagated to other views by exploiting the geometri and photometri transformation of nearby mathes. This extends the region-traks to previously unovered views. This step is vital for esaping the aforementioned statistial trap. Finally, erroneous parts of the region-traks, due to original mismathes and oasional mispropagations, are removed based on the analysis of the spatial arrangements of multiple regions in pairs of views (setion 5). Setion 6 presents experimental results and onludes the paper. The approah is not restrited to [], but an omplete any other WBS method based on loal affine features.. Extrating region-traks The two-views mathing algorithm [] is applied between all pairs of views (V i, V j ), i j produing separate sets of pairwise mathes, in the form (A l, B m ), indiating that region A in view V l has been mathed to region B in view V m. In order to fulfill the main goal, we need to integrate this information into region-traks: eah aggregating the image-regions of a ertain physial surfae path along the views. Suh a region-trak should be in the form R = {A l, B m,..., J z }, indiating that regions A l, B m,..., J z are all in mutual orrespondene, and represent a single physial region R as it s imaged on views l, m,..., z. The regiontraks should be mutually disjoint, i.e.: no two region-traks should share a ommon region. Define a similarity measure between two regions A, B drgb(a, B) sim(a,b) = NCC(A, B) + ( ) () 00 where NCC is the normalized ross-orrelation between the regions greylevel patterns, drgb is the average pixel-wise Eulidean distane in RGB olor-spae after independent normalization of the olorbands (neessary to ahieve photometri invariane). R, G, B ranges in [0, 55]. The two regions are aligned by the affine transformation mapping A to B, before omputation. This mixed measure proved onsiderably more disriminant than NCC alone in various experiments. How to get region-traks out of pairwise mathes? Let s onsider the graph G where verties represent regions and edges are weighted by funtion (). No edge is present between two unmathed regions. Suppose for a moment that the region extration and two-views mathing proesses were perfet, so that there are no mismathes and no missing mathes. In this ideal ase, G will be omposed of ompletely onneted, disjoint subsets of verties (liques). Sine eah lique orresponds to a region-trak, our task is easily solved (figure a). Unfortunately real data is plagued by two kinds of errors: mismathes, whih insert spurious edges in G, and missingmathes, whih make G lak some orret edges (figure b). With these errors, G is no longer in a disjoint-liques form and ambiguities about the omposition of the regiontraks arise. Three properties ome to help. First, mathing is transitive: if (A l, B m ) and (B m, C n ) are mathes then (A l, C n ) must be a math as well. Seond, a region annot be mathed to two different regions whih are on the same view (one-to-one-onstraint). Third, given the orret math (A l, B m ) and the mismath (A l, C m ), the similarity measure is, almost always, higher for the former. We propose a onflit-resolution algorithm (CR) that exploits these properties to transform G into a disjoint-liques form. CR reonstruts missing edges and disovers spurious edges by the same basi proess of triangulation: given any two edges (A l, B m ) and (B m, C n ), the edge (A l, C n ) is added to G if not yet present. The two edges generating the new one are its parents. An original edge has no parent (leaf). Two edges (A l, B m ) and (A l, C m ) are in onflit, as they violate the one-to-one-onstraint. At least one edge in a onflit is aused by a mismath. Adding an edge an generate a onflit, whih is resolved by removing the least weighted edge (i.e.: the math with the lower similarity measure). When an edge is removed also the weakest

3 view E F B A D C H view 4 I 4 G view view view E F view B A G 4 D 4 C H view 4 I 4 view view view 4 A D 4 E F view view E F view B B G A G C D 4 C H I 4 view H view 4 I 4 view Figure : a) ideal ase, b) real ase, dotted edges are mismathes, ) and d): two steps of CR. Thik edges are in onflit. of its parents is removed, unless the edge is a leaf. This is justified by the fat that, if an edge is onsidered a mismath, then at least one of its parents must be regarded as a mismath. CR starts by triangulating the math with highest similarity. This new math is used to triangulate further. CR reurses until no new edge an be onstruted with the latest generated one as a parent, or if this is removed as onsequene of a onflit. At this point, the next math with highest similarity is triangulated, and so on. Cyles are avoided by not allowing the triangulation of a previously removed edge, thus ensuring CR s onvergene. After termination, G is omposed of disjoint liques. Figure b shows an interesting ase. First, (B, D 4 ) is triangulated from (A, B ) and (A, D 4 ). This in turn generates (B, C ) (from (C, D 4 ) and (B, D 4 )). Now (B, C ) onflits with (F, C ), and the latter is removed as it has lower similarity measure (mismath, figure ). Sine (B, C ) survived, it ombines with (A, B ) to give (A, C ). No triangles an be further onstruted based on (A, C ), so original edges are proessed. (F, I 4 ) is triangulated from (E, F ) and (E, I 4 ), whih then indues (F, H ). A onflit between (F, H ) and (G, H ) is deteted and auses the removal of (F, H ) (figure d). This alls upon the removal of its weakest parent (F, I 4 ), whih in turn auses the removal of grandparent (E, I 4 ). Finally, the last possible edge (G, I 4 ) is added and the algorithm terminates on the ideal solution (figure a). This example shows how CR disovers and solves ambiguities in a onflitual set of mathes. The onflits are often hidden within the dataset, but are triggered by reursive triangulation, and then disambiguated by maximal similarity. CR is guaranteed to find all onflits, although it s not sure to always solve them orretly: in a few ases the similarity of the mismath might exeed the math s one. However, CR s goal is not to find the optimal solution, but instead to effiiently yield a reasonably good starting point for the further proessing stages. The proposed approah leans the initial set of mathes, yielding a oherent, onflit-free one, respeting the transitivity of mathing and the one-to-one onstraint. Hene region-traks are extrated out of a large set of onflitual pairwise mathes (000s for 0 views). CR is time-effiient, as it evaluates the similarity measure only when needed, and autious in that it has no fixed similarity threshold for rejeting a math, but instead only ompares similarities, relying on the weaker assumption that a orret math sores higher than a wrong math. This is important in this initial phase where unneessarily removing mathes ould ompromise the performane of the further proessing stages. In the rest of the paper the following definitions are assumed. Γ is the set of all region-traks. R = {R v } v vs is a region-trak, omposed by image regions in views vs. If R v R we say that region-trak R is present in view v. Φ vs = {R Γ R v R, v vs} is the set of regiontraks present simultaneously in eah view v vs.. Refinement Let R, R be two regions mathed between two views. In pratial situations R is not perfetly registered with R as they are independently extrated from the two views. It is desirable to refine this initial math, so as to obtain a more aurate orrespondene and to provide better input to the propagation stage (setion 4). Formally, R should be affinely transformed suh that the resulting region maximises the similarity () with R. This alls upon an algorithm that an browse a reasonably large range of the 6D affine spae searhing (an approximation of) the maxima while evaluating the similarity funtion as few times as possible (omputational ost). The affine spae is deomposed into its translation (t x, t y ), sale (s x, s y ), rotation (θ) and shear (h) omponents. We onsider searhing an hyperube Ω bounded in all its dimensions by predefined values. A point in Ω is denoted A = (t x, t y, s x, s y, θ, h). Let R be obtained by entering R around (0, 0). The algorithm starts from the identity transformation A = A 0 = (0, 0,,, 0, 0) and searhes for A max so that A max = argmax A Ω sim(r, AR ) In our experiments: t x [ 4,4], t y [ 4, 4], s x [0.6,.8], s y [0.6,.8], θ [ Pi 4, Pi ], h [,]. Disretization: 4 for translations, 0. for sales, Pi for rotation, 0. for shear 6

4 Figure : Example -dimensional non-onvex funtion to be maximized. Algorithms start from (0, 0). Gradient Desent reahes point (, 0), while the proposed algorithm iterates through (0, 9), (5, 9) and then reahes the global maximum (5, 5). Notie the numerous foldings in the landsape. Let A(dim, val) be point A with omponent dim set to val. Let i d, f d be the bounds of Ω along dimension d and v d max = max v [i d,f d ] sim(r, A(d, v)r ) () be the value of the highest similarity indued by affine transformations along the straight line segment passing from A, starting at A(d, i d ) and ending at A(d, f d ). Computing () for all dimensions yields 6 values vmax d, d...6, from whih only the absolute maximum v best is retained. This orresponds to evaluating similarity on 6 line segments onurrent in A (rays). Point A is now moved to the affine transformation induing v best. The proess iterates until stability. The proposed algorithm (RF) only searhes a predefined, immutable hyperube Ω entered at A 0, and an be seen as a partiular way of stepwise walking in Ω, where eah step ours in a diretion parallel to a oordinate axis and an be arbitrary large. RF is motivated by the observation that the similarity funtion generates spaes whih are smooth, but highly non-onvex, in whih they show frequent and diverse foldings. In suh a situation, Gradient Desent (GD) annot be relied on, as it gets stuk in the loal maxima losest to A 0. RF does not blindly limb the losest steepest hill, but instead its sight extends, on 6 rays, until the boundaries of Ω. Therefore, at every iteration, it gets another hane to notie a higher hill elsewhere and will eventually jump on it. The new hill is limbed until its tip, or until a searhing ray intersets an higher hill, and so on. The hanes of finding the absolute maximum are muh higher than for GD and are fairly high on absolute in the kind of spaes we onsider, whih are highly non-onvex, but also not very large (R roughly orresponds to R ). Figure exemplifies RF s behaviour in dimensions and shows a typial ase where it sueeds while GD fails. Figure : Top-left: The small white region has been artifiially deformed by simultaneous translation, rotation, sale and shear (A = ( 5, 0,.4,., Pi/7, 0.4)) to the large white one. The refined region (blak) omes very lose to the orret solution. Top-right: region in a view. Bottomleft: math before refinement, right: refined. In pratial situations RF iterates to 0 times. Given the searh spae boundaries and disretisation steps above, this amounts to 00 to 700 evaluations of the similarity measure. Thus, RF is about as fast as GD and is orders of magnitude faster than exhaustive searh (.5 mio for the same parameters) or simulated annealing. This makes RF omputationally affordable for our purposes, where thousands of refinements are required. The refinement algorithm is applied to all region-traks R Γ provided by the onflit-resolution algorithm (setion ) so as to obtain a globally optimal registration. To ahieve this, all regions of R are refined towards the same pivot-view. The pivot-view is seleted as the one maximising the sum of similarities to all others views where R is present. The sum of similarities for eah view is omputed after refinement to all other views. Note it is highly unlikely that a view where R is mismathed will be seleted as pivot. This simple approah is independent of the order in whih the views are onsidered and makes all regions within a region-trak well globally aligned. A high quality registration is important for the following propagation stage. 4. Propagation As pointed out in setion, many regions of a view V do not get mathed to another view V even though the feature is visible in the image (e.g.: the orresponding region has not been extrated, or maybe it has been extrated but the mathing failed). This setion desribes an approah for exploiting the information supplied by a orret math in order to generate many other orret mathes. Consider a region C in V without orrespondent in V (andidate

5 A Figure 4: The andidate region (spoon) is propagated to the right view via the affine transformation (A) of a support region (top). region) and a set Ψ = {S i } of mathed support regions in the neighborhood of C (figure 4). If S i and C lie on the same physial surfae (e.g.: a faet of an objet), then they will probably be mapped to V by similar affine and photometri transformations. For every S i Ψ do:. Compute the affine transformation A i mapping S i to S i in view V.. Compute the olor transformation TRGB i = {s R, s G, s B } between S i and Si. This is omposed by the sale fators on the three olorbands.. Projet C to view V via A i : C i = Ai C. 4. Evaluate the similarity between C i and C after applying the olor transformation T i RGB sim i = NCC(TRGB i C, C i ) + ( drgb(t RGB i C, C) i ) 00 Applying TRGB i allows us to use the unnormalized olordistane drgb on the raw image pattern, beause olor hanges (e.g.: darker light) are now ompensated for. This is an advantage as it provides maximal disriminative power. We retain C best, with best = argmax i sim i, the region that best mathes C and refine it by the algorithm of setion, yielding C ref. This refinement step adapts C best to the loal plane orientation and ounters perspetive effets (the affine approximation is valid only on a loal sale). C ref is onsidered orretly propagated if sim(trgb i C, C ref ) > t prop (t prop =.0 in our experiments). Note that refinement tends to raise the similarity of orretly propagated regions muh more than the similarity of mispropagated ones, bringing an important benefit: the inrease in the separation between the distributions of the similarity for mispropagated regions and for orretly propagated ones. Extensive experiments have shown the last thresholding step to be remarkably more effetive after refinement. In all experiments a irle of radius of the image width has been 5 used Note that this approah generates a new region in V whih might not have been originally extrated (differently than in [5], where the geometri transformation is used only to guide the searh for further mathes). This is important as it helps solving the main problem exposed in setion : the quik drop of the probability that a region is extrated simultaneously in several views. Indeed propagation strongly inreases the hanes that a region will be put in orrespondene, as it suffies that any nearby region undergoing a similar image transformation is orretly mathed. For every pair of views l, m, l m, the propagation algorithm is applied to all andidate regionsφ l \Φ m whih are present in l, but not in m, using as support the regions Φ lm already mathed between the two views. Everytime a region Rl i is suessfully propagated to view m, it is added to region-trak R i. Propagation doesn t generate new regiontraks, but extends urrently existing ones. As onsequene, the region-traks grow larger (i.e.: they are present in more views) and the onnetedness between any subset of views vs, i.e.: the amount Φ vs of region-traks present simultaneously in eah v vs, is strongly inreased (as shown in setion 6). Note also that a single propagation reates many new pairwise mathes, as all regions in the trak are impliitly assumed mathed to the newly added region. These transitive propagations ontribute atively to inreasing the inter-view onnetedness. 5. Topologial filter The region mathes along the views might still ontain some mismathes. These are due to original mismathes that survived the onflit-resolution stage and oasional mispropagations onstruted on them. A new method for removing mismathes based on a topologial onstraint for triples of regions is introdued. 5. The sidedness onstraint Consider a triple (R, R, R) of regions in V and their orresponding regions (R, R, R ) in V. Let i v be the enter of region Rv i. The funtion side(r v, R v, R v) = sign(( v v) v) () takes value if v is on the right side of the direted line v v from v to v, or value if it s on the left. The equation side(r, R, R ) = side(r, R, R ) (4) states that should be on the same side of the line in both views (figure 5). The sidedness onstraint (4) holds for all triples of oplanar regions, beause in this ase property () is viewpoint invariant. Equation (4) happens to be valid also for most non-oplanar triples. A triple for whih equation

6 Figure 5: Sidedness onstraint. should be on the same side of the direted line going from to in both views. (4) doesn t hold is said to violate the onstraint. This happens when at least one of the three regions is mismathed, or when the regions are not oplanar and there is important amera translation in the diretion perpendiular to the D sene plane ontaining their enters. The latter an reate a parallax effet strong enough to move to the other side (parallax-violation). However, this happens only to a small minority of triples, for any given pair of views, as also onfirmed by our experiments. Region-traks R i, R j, R k violate or respet equation (4) independently of the order in whih they appear in the triple. The three points should be ylially ordered in the same orientation (lokwise or anti-lokwise) in the two images in order to satisfy (4). 5. Removing mismathes A triple inluding one, or more, mismathed regions has higher hanes to violate the (4). When this happens we an only onlude that at least one of the regions in the triple is mismathed, but we do not know yet whih. While one triple is not enough to deide, this information an be reovered by onsidering all triples simulteneously. By integrating the weak information eah triple provides, it s possible to robustly disover mismathes. The key idea is that inorretly loated regions will be involved in a higher share of violations. [8] already noted the benefits of analysing topologial onfigurations of points and lines. Equation (4) is heked for all triples of regions (R i, R j, R k ), R i, R j, R k Φ, with Φ the set of all regions present in both V, V. Let Φ = {i R i Φ }. The algorithm starts by omputing h(i) = side(r, i R j, Rk ) side(r, i R j, Rk ) (5) j,k Φ\i,j>k the amount of violations R i is involved into, for all i Φ. h(i) is then normalized w.r.t. the total amount of possible violations any single region an be involved into h N (i) = h(i) (n ) (n ), n = Φ. The most violating region R w, with w = arg max i h N (i) is determined. If h N (w) > t topo, region R w is onsidered a mismath and removed from Φ. At eah iteration h N (i) is reomputed based on the remaining regions in Φ and eventually the most violating region is removed. The proess iterates until no more regions an be removed. It s wise to store the terms of the sum h(i) during the first iteration. In the other iterations, h(i) an be quikly reomputed by retrieving and adding up the neessary terms, making the omputational ost almost independent on the amount of iterations. During the first iterations, when several mismathes are still within Φ, even orretly mathed regions might have a high h N, beause of their partiipation in triples inluding a mismath. However, the mismathed regions will have an even higher h N, beause they will be involved in the very same triples, plus other violating ones. Thus the worst mismath R w, i.e.: the region whih is loated in V farthest from where it should be, has the highest hane to violate eah individual onstraint and therefore will have the highest h N. One R w is removed, all h N will derease, and the seond worst mismath will have the highest value. When only orret mathes are left, small error perentages due to oasional parallax-violations are still refleted by h N. However, these will probably be lower than t topo, ausing the algorithm to stop. Various experiments on artifial and real onfigurations onfirmed these theoretial onsiderations. The algorithm proved very robust, by withstanding large amounts of mismathes. In experiments with various senes, up to 65% of the regions where translated to uniformly distributed random loations. The algorithm suessfully removed all reloated regions, while losing at most a few orret mathes. If all region-traks of a triple are uniformly random distributed on the two images, then h N (i) has expeted value 0.5 for the three. This, together with experimental measurements on mismath-free onfigurations, helped us seleting t topo = 0.5. The topologial filter is applied to all pairs of views l, m, l < m yielding a set {(R l, R m )} of mismathes per pair. A single mismath still doesn t tell whih one of R l or R m is wrong. However, this information an be infered from the set of mismathes related to a single region-trak (i.e.: if R l is involved in 5 mismathes and R m in only ). Removing either R l or R m suffies to eliminate the mismath. We remove the minimum amount of regions from eah region-trak, suh that all mismathes are eliminated. This approah eliminates all deteted mismathes, while minimally reduing the region-traks, thus enhaning quality without sarifiing inter-views onnetedness. 6. Results and onlusion The multi-view mathing sheme has been tested on many sets of images (senes). All reported senes have been pro-

7 views traks CR views all 0 views traks CR Table. Number of traks for Valbonne. Figure 6: Valbonne. 0 -view mathes on views 9,, 4. essed with the same set of parameters reported in the paper. In the first example 0 images (ids,,,5,9,,,,4,5) from the Valbonne sene (6), were proessed. This image set poses several hallenges, like signifiant viewpoint differenes and uniform olors and textures. Table shows the number of region-traks present in various subsets of views. Note the high entries for 4 views and more, indiating strong onnetedness between views (e.g.: about 00 4-view mathes). The statistial problem stated in setion, namely the rapid derease of the hane of obtaining a N-view trak, with inreasing N, has been remedied: the amount of N-view mathes graefully dereases (lose to linearly) with inreasing N and orresponds well with failing visibility. The third olumn (CR) reports the amount of mathes just after forming the traks, via the algorithm proposed in setion. At this stage, the traks are still vulnerable, and their number deays exponentially (e.g.: 5 -view mathes in (9, ) roughly half in (9,, ) and only in (, 5, 9,,, )). However, the first stage s main goal is to solve the strutural problem, by formatting the data in region-traks. It s the ombined effet of refinement, propagation and topologial filtering that then ounters the statistial problem, as demonstrated in the seond olumn of the table. The inrease in the amount of N-view mathes is strong, and many of them now exist even when none ould be found in the original data (from 9 views on). The values in table ompare favorably against the ones reported in [5], orroborating the effetiveness of our approah. The orretness of the traks Φvs present simultaneously in a set of views vs is evaluated as Φvserrors ( vs ), where errors is the total amount of inorretly loated regions, omputed over all traks and views. This measure ranges from 0 (all mismathes) to (perfet traks), and takes into aount that a trak an be partially orret, if only some of its regions are mis-loated. Figure 6 shows 0 traks (96% orret), distributed over the whole hurh, in views (9,, 4). The Birthday sene features a more omplex geometry than Valbonne and more diverse textures. Eight very different viewpoints serve as input (ids to 8). Figure 7 shows 5 traks (95% orret) on views. Two of the views are Figure 7: Birthday. Left: top to bottom: views, 8, 7. Right: A 5-view trak. taken from almost opposite diretions (7, 8). Nevertheless, the mathes well over the ommonly visible parts of the sene. The telephone, and the piture of the girl above it, are affeted by strong out-of-plane rotation, but still have several orret mathes. 4 region-traks (98% orret) are present in another four images (figure 8). Note the quality of the traks on the mousepad, whih undergo strong image saling and rotation (views, 6), and the lak of 4-view mathes on the book. This is orret, as it s not visible in view 4. Robustness to sale hanges is demonstrated in figure 9, where viewpoint 5 is signifiantly loser to the sene than 4. Nevertheless, the two ameras see largely the same part of the sene, and the system produes 78 traks (97% orret), densely overing the images. Table summarizes the large inrease in N-view mathes

8 views traks CR views traks CR all 8 views Table. Number of traks for Birthday. Figure 8: Birthday. Top: views,. Bottom: views 4, 6. brought by the method. The amount of N-view mathes dereases lose to linearly with N. This is partiularly signifiant in this sene, where the parts visible in all N-views also derease signifiantly. The auray of the registration within a region-trak is exemplified in figure 7. The ellipse s shape and orientation over the same physial surfae in all views, aurately deforming to fit the viewpoints. This entails rotation, skew and anisotropi sale hanges, largely overing the affine spetrum. Figure 9 depits another interesting ase, where a region gradually rotates over almost 80 degrees out-ofplane during 5 views. These ome from a sene featuring a 9 views tour around a ereal box. The good alignment is made possible, and omputationally affordable, by the new optimization tehnique presented in setion. The experiments onfirm that the presented approah an generate a large amount of high quality region traks starting from an unordered set of images taken from substantially different viewpoints. The traks strongly onnet the views and provide a good overage of the ommonly visible parts of the sene. This is partiularly important in objetreognition, where more overage means a more omplete desription of the objet. Auray is more an issue in Dreonstrution and has also been improved. The method works in onjuntion with most WBS algorithms. Future developments inlude the appliation to objet-reognition and effiieny improvements. Figure 9: Top: view 4. Middle: view 5. Frame: 5 views from a ereal-box trak. Referenes [] T. Tuytelaars and L. Van Gool Wide Baseline Stereo based on Loal, Affinely invariant Regions British Mahine Vision Conferene, pp. 4-4, 000. [] K.Mikolajzyk and C.Shmid An affine invariant interest point detetor ECCV., vol., 8 4, 00. [] A.Baumberg Reliable Feature Mathing Aross Widely Separated Views Intl. Conf. on Comp. Vis., pp , 000. [4] J. Matas, O. Chum, M. Urban and T. Pajdla Robust Wide Baseline Stereo from Maximally Stable Extremal Regions British Mahine Vision Conf., pp. 44-4, 00. [5] F. Shaffalisky and A. Zisserman Multi-view mathing for unordered image sets European Conf. on Comp. Vis., 00. [6] D. Lowe Objet Reognition from Loal Sale-Invariant Features Intl Conf. on Comp. Vis., pp , 999. [7] D. Tell and S. Carlsson Wide baseline point mathing using affine invariants ECCV., vol., pp , 000. [8] S. Carlsson Combinatorial Geometry for Shape Representation and Indexing Obj. Repr. in Comp. Vis. II, Pone, Zisserman eds. 996