Multi-view object segmentation in space and time. Abdelaziz Djelouah, Jean Sebastien Franco, Edmond Boyer

Transcription

1 Multi-view object segmentation in space and time Abdelaziz Djelouah, Jean Sebastien Franco, Edmond Boyer

2 Outline Addressed problem Method Results and Conclusion

3 Outline Addressed problem Method Results and Conclusion

4 Addressed problem Automatic segmentation of a single object seen from multiple calibrated cameras

5 Outline Addressed problem Method Results and Conclusion

6 Method Graph cuts for segmentation Basic idea of this paper Overview Formulation Algorithm

7 Method Graph cuts for segmentation Basic idea of this paper Overview Formulation Algorithm

8 Graph cuts for segmentation Yuri Y. Boykov and Marie-Pierre Jolly ICCV 2001

9 Graph cuts for segmentation Advantages: 1.Clear defined cost function 2.Gloabally optimal solution

10 Graph cuts for segmentation Preliminaries: PP: the set of pixels NN: the set of pairs of neighboring pixels AA = (AA 1,, AA pp,, AA PP ): a binary vector defining a segmentation AA pp {"obj", "bkg"}: the assignment to pixel pp Cost function: EE AA = λλ. pp PP RR pp AA pp + {pp,qq} NN BB pp,qq. δδ(aa pp, AA qq ) where δδ AA pp, AA qq = 1 AA pp AA qq 0 ooooooooooooooooo

11 Graph cuts for segmentation Cost function: EE AA = λλ. pp PP RR pp AA pp + {pp,qq} NN BB pp,qq. δδ(aa pp, AA qq ) where And δδ AA pp, AA qq = 1 AA pp AA qq 0 ooooooooooooooooo RR pp "ooooooo = llllllll(ii pp OO) RR pp "bbbbbbb = llllllll(ii pp B) BB pp,qq exp II pp II qq σσ 2 dddddddd(pp,qq)

12 Graph cuts for segmentation Graph GG =< VV, E > : a set of nodes VV and a set of edges E ww ee : nonnegative weight of edge ee SS and TT : two terminal nodes t-link: edges between nodes and terminals n-links: other edges Cut ss tt cut: a subset of edges CC E making SS and TT become separated on GG(CC) =< VV, E\C > Cost of cut: CC = ww ee ee CC

13 Graph cuts for segmentation segmentation cut

14 Graph cuts for segmentation Feasible cut CC: 1. CC serves exactly one t-link at each pp 2. {pp, qq} CC iff pp and qq are t-linked to different terminals on GG(CC) Feasible cut CC Segmentation A(c) "bkg", iiii {pp, TT} CC AA pp cc = "obbbbb, iiii {pp, SS} CC Min cut CC on GG is feasible

15 Graph cuts for segmentation Edge Weights: Edge Weight(cost) for {pp, qq} BB {pp,qq} {pp, qq} NN {pp, SS} λλ. RR pp ("bbbbbbb) pp PP {pp, TT} λλ. RR pp ("ooooooo) pp PP Cost of feasible cut =cost function CC = ee CC ww ee = λλ. pp PP RR pp AA pp (cc) + {pp,qq} NN BB pp,qq. δδ(aa pp (cc), AA qq (cc)) = EE(AA(CC))

16 Graph cuts for segmentation Feasible cut CC Segmentation AA(CC) Cost of CC =Cost function EE(AA(CC)) Min cut CC on GG is feasible Minimize EE(AA) find a minimum s-t cut Min-cut/max-flow algorithms (Boykov and Kolmogorov PAMI 2004)

17 Graph cuts for segmentation Important notes Generally, directed graphs are used to solve energy minmization. Energies should satisfy the submodularity constrain. A general method of graph construction is available. More Information can be found in Kolmogorov and Zabih, PAMI 2004.

18 Graph cuts for segmentation Advantages: 1.Clear defined cost function 2.Gloabally optimal solution

19 Graph cuts for segmentation Advantages: 1.Clear defined cost function 2.Gloabally optimal solution better cost function better segmentation

20 Method Graph cuts for segmentation Basic idea of this paper Overview Formulation Algorithm

21 Basic idea Use multi-view coherence Create inter-view links with 3D samples

22 Method Graph cuts for segmentation Basic idea of this paper Overview Formulation Algorithm

23 Overview Divide each image into superpixels Initialize appearance model Iterate until convergence Label superpixels Update appearance model Label pixels

24 Method Graph cuts for segmentation Basic idea of this paper Overview Formulation Algorithm

25 Formulation Preliminaries II tt = II 1,tt,, II nn,tt : a set of input images at instant tt PP ii tt : the set of superpixels pp in II ii,tt xx pp {ff, bb}: the label of pp PP ii tt SS tt : the set of 3D samples ss uniformly sampled in the common visibility volume. xx ss {ff, bb}: the label of ss SS tt

26 Formulation Preliminaries II tt = II 1,tt,, II nn,tt : a set of input images at instant tt PP ii tt : the set of superpixels pp in II ii,tt xx pp {ff, bb}: the label of pp PP ii tt SS tt : the set of 3D samples ss uniformly sampled in the common visibility volume. xx ss {ff, bb}: the label of ss SS tt

27 Formulation Foreground and background models II rr ii : descriptor of pixel rr, an 11-dimension vector encoding gradient magnitude response for 4 scales, Laplacian for 2 scales, and RGB values HH ii BB and HH ii FF : background and foreground histograms of pixel descirptors in II ii tt, computed on clusters of pixel descirptors cluster 1 cluster 2 cluster 3 cluster 4

28 Formulation Foreground and background models II rr ii : descriptor of pixel rr, an 11-dimension vector encoding gradient magnitude response for 4 scales, Laplacian for 2 scales, and RGB values HH BB ii and HH FF ii : background and foreground histograms of pixel descirptors in II tt ii, computed on clusters of pixel descirptors HH ii : histogram of the whole image AA pp : descriptor of superpixel pp, histogram on clusters of pixel descriptors

29 Formulation Foreground and background models Model initialization

30 Formulation Energy principles Individual appearance The appearance of a superpixel should comply with image-wide background or forground models, depending on its label. Appearance continuity Neighbouring superpixels likely have the same labels. Appearance similarity Superpixels with similar color/texture likely have the same labels.

31 Formulation Energy principles Multi-view coherence 3D samples are considered object-consistent if they project to foreground regions with high likelihood. Projection constraint A superpixel should be foreground if it sees at least one object-consistent sample, otherwise it should be background. Time consistency Temporally linked superpixels likely have the same label.

32 Formulation Energy terms Individual appearance term EE cc xx pp = rr R llllhh ii BB (II ii pp rr ) iiii xx pp = bb rr R llllhh FF ii (II ii pp rr ) iiii xx pp = ff Appearance continuity term EE nn xx pp, xx qq = exp dd AA pp, AA qq iiii xx 2<dd AA pp,aa qq > 2 pp xx qq 2 0 ooooooooooooooooo

33 Formulation Energy terms Appearance similarity term EE aa xx pp, xx qq = exp dd AA pp, AA qq 2dd<AA pp,aa qq > 2 iiii xx pp xx qq 0 ooooooooooooooooo 2

34 Formulation Energy terms Sample objectness term EE ss xx ss = ln 1 PP ff ss ff ln PP ss iiii xx ss = bb iiii xx ss = ff PP ss ff = PP xx ss = ff II ss 1,, II ss nn ππ FF nn FF HH ii IIss ii ii=1 nn ii=1 HH ii (II ss ii ) samples from the object. = PP xx ss=ff PP(II ss 1,,II ss nn xx ss =ff) PP(II ss 1,,II ss nn ), where ππ FF is the proportion of 3D =

35 Formulation Energy terms Sample-pixel junction term EE jj xx ss, xx pp Sample projectin term = iiii xx ss = ff aaaaaa xx pp = bb 0 ooooooooooooooooo EE pp xx pp = ln 1 PP xx pp VV pp iiii xx pp = bb ln PP xx pp VV pp iiii xx pp = ff where PP(xx pp VV pp ) = max ss VV pp (PP ss ff )

36 Formulation Energy terms Time consistency terms EE ff xx pp tt, xx qq tt+1 = θθ ffexp dd AA pp tt, AA qq tt+1 2 2<dd AA pp tt, AA qq tt+1 > 2 iiiixx pp tt xx qq tt+1 0 ooooooooooooooo

37 Formulation Energy Function with submodularity being satisfied, minimization is solved by graph cuts

38 Algorithm

39 Outline Addressed problem Method Results and Conclusion

40 Results and conclusion

41 Results and conclusion

42 Results and conclusion Conclusion An unified framework dealing with intra-view, inter-view, and temporal cues Inter-view propagation of segmentation information using 3D samples

43 References [1] Abdelaziz Djelouah et al., Multi-View Object Segmentation in Space and Time, ICCV [2] Abdelaziz Djelouah et al., N-tuple Color Segmentation for Multi-View Silhouette Extraction, ECCV [3] Yuri Y. Boykov, Marie-Pierre Jolly, Interactive Graph Cuts for Optimal Boundary & Region Segmentation of Objects in N- D Images, ICCV [4] Boykov, Kolmogorov, An Experimental Comparison of Min-Cut/Max-Flow Algorithms for Energy Minimization in Vision, PAMI [5] Valdimir Kolmogorov, Ramin Zabih, What energy functions can be minimized via graph cuts?, PAMI 2004.

44 Thanks!