Saliency Detection in Aerial Imagery using Multi-scale SLIC Segmentation Samir Sahli 1, Daniel A. Lavigne 2 and Yunlong Sheng 1 1- COPL, Image Science group, Laval University, Quebec, Canada 2- Defence R&D Canada - Valcartier, Quebec, Canada
Motivation Existing methods Multi-scale SLIC SLIC region over-segmentation Saliency map Selection of Salient regions Experiments Quantitative Comparison Discussion Conclusion
Intelligence, Surveillance, Recognition (ISR) Nadir view of a scene High-resolution large-scale image. Image taken by Wide Field of View (WFOV) cameras deployed on UAVs. During a campaign of surveillance Images are acquired in huge quantity. Imbalance between: Amount of data & Computational resources at disposal.
Saliency region detection Divide & Conquer strategy Focus attention ti on Salient regions; Process in Salient regions for object detection Method must be Computationally effective for processing large size images; No need for a priori knowledge on objects; No training i and parameter tuning Invariant to window size change.
Existing approaches Inspired by Human Vision Neurophysiological evidence for visual attention. Saliency map : Pixels representing the importance of the objects Estimate dissimilarity in terms of intensity, color and orientation Requires feature extraction, ti selection and combinations to create saliency map. Statistical approaches Adaptive Gaussian Mixed Model partitioning i image into 2 classes (background & salient regions) Combine salient features classified with a mixture of linear SVM Supervised learning or parameters Tuning. Intrinsic method Spectral Residual (SR), based on Fourier spectrum. Rosin method (ROS), based on density of edges in the image. Fast, require no parameter tuning, no a priori information, ease to implement.
Superpixels are coherent homogeneous local clusters of pixels, One parameter to input : Number of superpixels (K) Step 1: Initialization A grid of K seeds is set at equal distance S between the seeds S = N number of pixels Achanta, R. et al, EPFL Technical Report no. 149300, (2010). N K
Step 2: Perturb the seed locations Move away from edges in the image. Lowest gradient position in color space
Step 3: Competition for clustering Each pixel is associated to the nearest seed according to the Euclidian distance in the 5D space. Competition between neighboring seeds delimits the spatial extent of superpixels and forms boundaries.
Step 4: Move seeds to centroids Seeds move to the centroids of the newly formed clusters as shown by red arrows Re-clustering (Step 3) Iterate until all the clusters centers converge to stable positions.
Irregularity in Shape and Size of superpixels p Superpixels to over-segment the regions Superpixel contours are deformed to tightly adhere with boundaries of objects in the image. I l it i h d i Irregularity in shape and size, as a saliency measure.
Multi-scale SLIC SLIC segments images by combining region and edge information If Scale S >> object of interest structural details may be ignored by the superpixels. If Scale S < object of interest structural details can be described by superpixel of irregular shape and various size. As sizes of objects are unknown a priori As sizes of objects are unknown a priori We use multi-scale SLIC
Irregularity in Shape and Size (a) Strings of inner pixels (blue) and boundary pixels (red); (b) Minimum distance from an inner pixel to the boundary; (c) Hausdorff distance H (Red arrow) is the radius of the green disc inscribed in the superpixel.
Hausdorff distance Measure of the compactness of a shape. Properties Applicable to all geometric shapes; Independent of orientation; independent of extreme points; Refer to the intuitive notion of compactness. Dimensionless and Independent of scale.
Saliency measure Posterior Probability bilit Function f : Normalized Hausdorff distance Shape factor ShF is a monotone sigmoid function Each pixel is associated a probability that reflects its salience.
For each hscale Si For each superpixels (1) Compute the shape factor ShF; (2) set ShF to all pixels of the given superpixel. End saliency map for scale S End Sum saliency map from each scale final saliency map.
For each Scale Si For each superpixels (1) Compute the shape factor ShF; (2) set ShF to all pixels of the given superpixel. End saliency map for scale S End Sum saliency map from each scale final saliency map.
Spectral Residual Impossible to discern shapes Multi-scale edge map Presence of halos that spread beyond limits of the structure. Multiscale SLIC Better description of shape; High contrast, background /salient objects Separation between objects Vehicles appear disjoint like in input image
Pixel based probabilistic decision rule R where {S} is ensemble of the scale, εt the hidden scale. A pixel t in the image I Xt = 1 if t belong to the salient region Xt= 0, otherwise. Logic OR of all the salient regions detected at all scales gives the final salient region map
Database of 18 images larges-scale and high resolutions 1500x1500 pixels & 11.5 cm/pixel. Hand labeling for the Ground truth. Apply Multi-scale SLIC Produce salient regions; Compare with the ground truth; Evaluate using figure-of-merit: Precision, Recall, F-measure. Comparaison with Spectral Residual & Rosin [*] X. Hou and L. Zhang. CVPR, (2007) [**] P. L. Rosin. Pattern Recognition, 42(11):2363-2371, (2009).
F-measure is the overall performance measurement. The multi-scale SLIC has a F-measure at least 2 times higher than the compared methods. For attention detection, recall rate is not as important as precision
Multiscale SLIC vs. Multi-scale Edge maps Multi-Scale edge maps Method based on the edges, local gradient detect salient regions in the background. (local threshold computed on the square) The Multi-Scale SLIC Method based on the shape and size of superpixel boundaries. What is salient in the selected region? Nothing could be also a valuable response for Multi-scale SLIC.
Invariant to processing window size change Multi-Scale SLIC Pixels are detected independly of the rest of pixels image. Salient regions are independant of image size. Advantages: Crop or divide in sub-images; Apply Multi-Scale SLIC in parallele; Recombine. Mosaicking is possible suitable for Surveillance of Large-scale area Preserve capacity of global awarness while permit detection and recognition.
At each Scale: (1)Calcul of SLIC; (efficient) (2)Estimating the Hausdorff distance H by Distance Transformation (efficient) Total process, Multi Scale Slic with 7 scales took 4,36 s for a color image of 500x500 pixel size, (Matlab coding on Duo Core 2 PC at 2.5GHz). GPU version of SLIC exists Time given (ms)for one scale.m
We propose a new approach for detecting salient regions based on superpixels by multi-scale SLIC; Size and Shape irregularity of superpixels as Saliency measure. F-measure at least 2 times higher than the compared methods; Recall measure in the same order of magnitude. Advantages: No threshold are used; The decision is done independly for each pixel; Invariant to processing window size change: Mosaicking of large-scale l area under surveillance; Keep capacity of detection and recognition for tiny objects.