GrayCut Object Segmentation in IR-Images

Transcription

1 Department of Signal Processing and Communications Prof. Dr.-Ing. Udo Zölzer GrayCut Object Segmentation in IR-Images Christian Ruwwe & Udo Zölzer 2 nd International Symposium on Visual Computing (ISVC) Lake Tahoe, Nevada, USA November 6-8, 2006

2 Outline Motivation why IR-Images? Graph-based Segmentation GraphCut GrabCut GrayCut what is new? Gaussian Mixture Models Morphological postprocessing Don t-care Map Results and discussion Conclusion & Outlook GrayCut - Object Segmentation in IR-Images Page 2

3 Motivation - Ship Classification KEOD - an application for the German Navy: ship classification with optical sensors optical daylight camera (long wavelength) infrared sensor ( µm) gray-level images resolution 768 x 576 pixel (PAL) Currently two algorithms available distance-based comparison of feature points correlation-based contour comparison GrayCut - Object Segmentation in IR-Images Page 3

4 Motivation - Contour Correlation - Correlation of binary contour images high correlation value high probability for classification database of contour images made from 3D-models in different poses list of sorted classification result the user has to make the final descision! Contour line extraction contour sketch is error prone and highly user-dependent (semi-) automatic contour extraction needed GrayCut - Object Segmentation in IR-Images Page 4

5 Motivation - IR Images - Problems with IR-images limited dynamic range? bad SNR-ratio segmentation based on the gradient fails (e.g. Canny Detector) gray-levels can vary image can be inverted predefined gray-level or threshold fails ships have a homogenous bottom side, but an inhomogenous superstructure parametric curves (e.g. Snakes) cannot fullfill both GrayCut - Object Segmentation in IR-Images Page 5

6 Graph-based Segmentation - Basics - Segmentation is an energy minimization problem for all pixels minimize Combination of gradient and same-color approach 1. the fidelity term 2. the prior term from the likelihood of colors, from the gradient between neighbouring pixels 3. and a weighting factor for their relative influence. GrayCut - Object Segmentation in IR-Images Page 6

7 Graph-based Segmentation - S/T-Graph - S Building a graph pixels as nodes p a edge weights between neighbouring pixels n a,b from the gradient 2 special nodes (terminals) the object S the background T edge weights between pixels and terminals p S,a and p T,a from histogram-based probability mincut/maxflow results in the optimum cut between pixels of the object (connected to S) and background pixels (connected with T) the segmentation border / contour line p S,1 p p S,2 S,4 p S,5 P 1 P n 2 1,2 n 2,3 P 4 P n 5 4,5 n 5,6 P 7 P 8 P n 9 7,8 n 8,9 p T,7 p T,8 T P 3 n 1,4 n 2,5 n 3,6 n 4,7 n 5,8 P 6 p T,3 p T,6 p T,9 n 6,9 GrayCut - Object Segmentation in IR-Images Page 7

8 Graph-based Segmentation - GraphCut - GraphCut idea originally proposed by Greig/Porteous/Seheult in 1989 optimized with mincut/maxflow by Boykov/Jolly in Initialization user defined selection of seeds (or regions) sure object or sure background information for the fidelity term gradient of the image gives the prior term 2. Building the S/T-graph 3. Solving with mincut/maxflow-algorithm one-shot calculation repeat everything, if the result is not good enough GrayCut - Object Segmentation in IR-Images Page 8

9 Graph-based Segmentation - GrabCut - GrabCut (Rother/Kolmogorov/Blake, 2004) Extension to color images evaluate membership probability from Gaussian Mixture Models iterative optimization (due to the Expectation Maximization) 8-way neighbourhood (4-way in GraphCut) contrast/gradient is now calculated by alternating between mincut/maxflow solution of the graph EM updates to the Gaussian Mixture Models GrayCut - Object Segmentation in IR-Images Page 9

10 GrayCut -GrayCut- Combining the advantages of both iterative approximation using Gaussian Mixture Models Modifications using only 3 instead of 5 Gaussian components only one-dimensional due to gray-levels sufficient for one-dimensional histogram approximation introducing morphological operations between two iteration steps dilation, closing, increasing the number of iterations needed GrayCut - Object Segmentation in IR-Images Page 10

11 GrayCut - Gaussian Mixture Models - Using only 3 instead of 5 Gaussian components only one-dimensional due to gray-levels sufficient for one-dimensional histogram approximation Evolution of 3 components for object and background (left) and original two histograms of the image (right) GrayCut - Object Segmentation in IR-Images Page 11

12 GrayCut - Post-processing Observation color information in ship images is too low EM need more steps to adapt to color distribution high contrast leads to a fragmented segmentation result smooth segmentation to reflect a solid object Introducing morphological operations between iteration steps dilation widening the intermediate segmentation result closing filling holes and removing small fragments leading to a compact and robust segmentation result, but increasing the total number of iterations needed for a suitable result GrayCut - Object Segmentation in IR-Images Page 12

13 GrayCut - Don t-care Map Problem: overlay symbols from the camera system strong gradient, but not related to the image content segmentation border mostly along these symbols their color might influence the evolution of the GMMs Workaround: ignore some parts of the image 1. the overlays symbols (and their position) are known beforehand define these parts of the image as don t-care 2. clear the contrast term for these pixels (before each iteration) homogenous region 3. reset them to unknown-state (before each iteration) their color will not be used in the EM steps GrayCut - Object Segmentation in IR-Images Page 13

14 Results - Don t-care Map - Overlays symbol (black lines) are ignored contour line uses shortest path between the valid endpoints GrayCut - Object Segmentation in IR-Images Page 14

15 Results - Success! 23 suitable results out of 79 IR images (without user refinement) GrayCut - Object Segmentation in IR-Images Page 15

16 Results - Problems? 25 (of 79) results might be usable with additional user refinement Only in 12 images the algorithm gives no result at all GrayCut - Object Segmentation in IR-Images Page 16

17 Conclusion GrayCut uses ideas from GrabCut/GraphCut energy formulation and graph-based segmentation Gaussian Mixture Models for color description manual refinement (by user sketch) possible GrayCut introduces new features morphological processing between iteration steps don t-care map for overlays symbols Segmentation results are encouraging 60% success rate 25% failure GrayCut - Object Segmentation in IR-Images Page 17

18 Outlook Still there are 12 images, where the algorithms totally fails there is no result at all Why? Space for improvements problems with small superstructures (antennas, masts, ) reduction of user interaction (in 30% of the images) ìn 19 images the result is senseless Why? GrayCut - Object Segmentation in IR-Images Page 18

19 Thank You! Questions? GrayCut - Object Segmentation in IR-Images Page 19

20 Results Segmentation results on IR images GrayCut - Object Segmentation in IR-Images Page 20

21 Results Segmentation results on IR images GrayCut - Object Segmentation in IR-Images Page 21