Computer Vision I - Basics of Image Processing Part 2 Carsten Rother 07/11/2014 Computer Vision I: Basics of Image Processing
Roadmap: Basics of Digital Image Processing Computer Vision I: Basics of Image Processing 07/11/2014 2 What is an Image? Point operators (ch. 3.1) Filtering: (ch. 3.2, ch 3.3, ch. 3.4) main focus Linear filtering Non-linear filtering Multi-scale image representation (ch. 3.5) (will be done in SS15 as part of Image Processing) Edges detection and linking (ch. 4.2) Line detection and vanishing point detection (ch. 4.3) (will be done in SS15 as part of Image Processing) Interest Point detection (ch. 4.1.1)
Reminder: Convolution (linear filter) Computer Vision I: Basics of Image Processing 07/11/2014 3 Replace each pixel by a linear combination of its neighbours and itself. 2D convolution (discrete) g = f h Centred at 0,0 f x, y h x, y g x, y g x, y = k,l f x k, y l h k, l the image f is implicitly mirrored
Reminder: Median Filter Computer Vision I: Basics of Image Processing 07/11/2014 4 Replace each pixel with the median in a neighbourhood: 5 6 5 5 6 5 4 20 5 4 5 5 4 6 5 4 6 5 median Median filter: order the values and take the middle one No strong smoothing effect since values are not averaged Very good to remove outliers (shot noise) Used a lot for post processing of outputs (e.g. optical flow)
Reminder: Motivation Bilateral Filter Computer Vision I: Basics of Image Processing 07/11/2014 5 Original + Gaussian noise Gaussian filtered Bilateral filtered Edge over-smoothed Edge not over-smoothed
Reminder: Bilateral Filter in pictures Computer Vision I: Basics of Image Processing 07/11/2014 6 Gaussian Filter weights Output (sketched) Centre pixel Noisy input Bilateral Filter weights Output
Reminder: Bilateral Filter in equations Computer Vision I: Basics of Image Processing 07/11/2014 7 Filters looks at: a) distance to surrounding pixels (as Gaussian) b) Intensity of surrounding pixels Linear combination Same as Gaussian filter Consider intensity Problem: computation is slow O Nw ; approximations can be done in O(N) See a tutorial on: http://people.csail.mit.edu/sparis/bf_course/
Bilateral filter is a non-linear filter Computer Vision I: Basics of Image Processing 07/11/2014 8 Linear Filter satisfy Additivity: T[X + Y] = T[X] + T[Y] Image: 0 1 0 1) Operator T is (linear) boxfilter: compute filter output for central element of the image: T X + Y = 2 3 T X + T Y = 2 3 1 3 1 3 1 3 2) Operator T is (non-linear) Bilteral filter with w(i, j, k, l) = exp( f i, j f k, l Compute filter output for central element of the image: 2 ) T X + Y = (0 0.04 + 2 1 + 0 0.04) / (0.04 + 1 + 0.04) = 1.92 T X + T Y = 2 * [ (0 0.36 + 1 1 + 0 0.36) / (0.36 + 1 + 0.36) ] = 2*(1/1.72) = 1.16 (Note: exp(-1) = 0.36; exp(-4) = 0.04)
Application: Bilateral Filter Computer Vision I: Basics of Image Processing 07/11/2014 9 Cartoonization HDR compression (Tone mapping)
Joint Bilateral Filter Computer Vision I: Basics of Image Processing 07/11/2014 10 ~ ~ Same as Gaussian Consider intensity f is the input image which is processed ~ f is a guidance image where we look for pixel similarity
Application: combine Flash and No-Flash input image f ~ Guidance image f Joint Bilateral Filter ~ ~ We don t care about absolute colors [Petschnigg et al. Siggraph 04] Computer Vision I: Basics of Image Processing 07/11/2014 11
Reminder: Application: Cost Volume Filtering Computer Vision I: Basics of Image Processing 07/11/2014 12 Reminder from first Lecture: Interactive Segmentation Goal z = R, G, B n x = 0,1 n Given z; derive binary x: Model: Energy function E x = i θ i x i + i,j θ ij (x i, x j ) Unary terms Pairwise terms Algorithm to minimization: x = argmin x E(x)
Reminder: Unary term New query image z i θ i (x i = 0) θ i (x i = 1) Dark means likely background Dark means likely foreground Optimum with unary terms only Computer Vision I: Introduction 07/11/2014 13
Cost Volumne for Binary Segmenation Label Space (here 2) Computer Vision I: Basics of Image Processing 07/11/2014 14 θ i (x i = 1) θ i (x i = 0) Image (x,y) For two labels, we can also look at the ratio Image f: f i = θ i (x i = 1) / θ i (x i = 0)
Application: Cost Volume Filtering Computer Vision I: Basics of Image Processing 07/11/2014 15 Ratio Cost-volume is the Input Image f Filtered cost volume ~ Guidance Input Image f (also shown user brush strokes) Results are very similar: This is an alternative to energy minimization! Cost Volume filtering (Winner takes all Result) Energy minimization [C. Rhemann, A. Hosni, M. Bleyer, C. Rother, and M. Gelautz, Fast Cost- Volume Filtering for Visual Correspondence and Beyond, CVPR 11]
Morphological operations Computer Vision I: Basics of Image Processing 07/11/2014 16 Two steps: 1. Perform convolution with a structural element : binary mask (e.g. circle or square) black is 1 white is 0 2. Threshold the continuous output f to recover a binary image:
Other non-linear operations on binary images Computer Vision I: Basics of Image Processing 07/11/2014 17 Binary Image Distance transform Skeleton Binary Input Image Connected components
Roadmap: Basics of Digital Image Processing Computer Vision I: Basics of Image Processing 07/11/2014 18 What is an Image? Point operators (ch. 3.1) Filtering: (ch. 3.2, ch 3.3, ch. 3.4) main focus Linear filtering Non-linear filtering Multi-scale image representation (ch. 3.5) (will be done in SS15 as part of Image Processing) Edges detection and linking (ch. 4.2) Line detection and vanishing point detection (ch. 4.3) (will be done in SS15 as part of Image Processing) Interest Point detection (ch. 4.1.1)
Goal: Find long edge chains Computer Vision I: Basics of Image Processing 07/11/2014 19 What do we want: Good detection: we want to find edges not noise Good localization: find true edge Single response: one per edge (independent of edge sharpness) Long edge-chains
Idealized edge types Computer Vision I: Basics of Image Processing 07/11/2014 20 We focus on this
What are edges? Computer Vision I: Basics of Image Processing 07/11/2014 21 correspond to fast changes in the image The magnitude of the derivative is large Image of 2 step edges Image of 2 ramp edges Slizce through the image Slizce through the image
What are fast changes in the image? Computer Vision I: Basics of Image Processing 07/11/2014 22 Texture or many edges? Scanline 250 Image Edges defined after smoothing Scanline 250 smoothed with Gausian
Edges and Derivatives Computer Vision I: Basics of Image Processing 07/11/2014 23 We will look at this first
Edge filters in 1D Computer Vision I: Basics of Image Processing 07/11/2014 24 We can implement this as a linear filter: Forward differences: 1/2-1 1 Central differences: 1/2-1 0 1
Reminder: Seperable Filters Computer Vision I: Basics of Image Processing 07/11/2014 25 1 4 1 2 1 This is the centralized difference-operator
Edge Filter in 1D: Example Computer Vision I: Basics of Image Processing 07/11/2014 26 Based on 1st derivative Smooth with Gaussian to filter out noise Calculate derivative Find its optima
Edge Filtering in 1D Computer Vision I: Basics of Image Processing 07/11/2014 27 Simplification: (saves one operation) Derivative of Gaussian
Edge Filtering in 2D Computer Vision I: Basics of Image Processing 07/11/2014 28
Edge Filter in 2D: Example Computer Vision I: Basics of Image Processing 07/11/2014 29 y x
Edge Filter in 2D Computer Vision I: Basics of Image Processing 07/11/2014 30 x-derivatives with different Gaussian smoothing
What is a gradient Computer Vision I: Basics of Image Processing 07/11/2014 31
What is a gradient Computer Vision I: Basics of Image Processing 07/11/2014 32
What is a gradient Computer Vision I: Basics of Image Processing 07/11/2014 33
What is a Gradient Computer Vision I: Basics of Image Processing 07/11/2014 34
Example Gradient magnitude image Computer Vision I: Basics of Image Processing 07/11/2014 35 First smoothed with Gaussian First smoothed with broad Gaussian Our goal was to get thin edge chains?
How to get edge chains Computer Vision I: Basics of Image Processing 07/11/2014 36 Rough Outline of a good edge detector (such as Canny) 1. Compute robust Gradient Image: ( (D x G) I, (D y G) I) 2. Find edge-points ( edgels ): non-maximum suppression 3. Link-up edge-points to get chains 4. Do hysteresis to clean-up chains edge-points or edgel
Non-maximum surpression Not Edge point Edge point Magnitude image Zoom on pixel-grid Wich pixel is an edge point? (non-max surpression) 1. Check if pixel is local maximum at gradient direction (+/- 180deg) (for this interpolate values for p,r) 2. Accept edge-point if above a threshold Computer Vision I: Basics of Image Processing 07/11/2014 37
Link up edge-points to get chains Computer Vision I: Basics of Image Processing 07/11/2014 38 Not Edge point Edge point 1. Link-up neighbouring pixels if both are edge-points.
Clean up chains with Hysteresis Computer Vision I: Basics of Image Processing 07/11/2014 39 Keep a chain: High start threshold Low threshold along the chain
Final Result Image Not much smoothing (fine scale) much smoothing (coarse scale) small threshold much smoothing (coarse scale) large threshold Computer Vision I: Basics of Image Processing 07/11/2014 40
Future Lecture: Segmentation Computer Vision I: Basics of Image Processing 07/11/2014 41 So far we looked at jumps in gray-scale images? Humans perceive edges very differently (edges depend on semantic) average human drawing Hard for computer vision methods without semantic reasoning [from Martin, Fowlkes and Mail 2004]
Try to learn semantically meaningful image edges Computer Vision I: Basics of Image Processing 07/11/2014 42 Features: brightness gradient; color gradient; texture gradient; weighted combination, etc.
Half-way break Computer Vision I: Basics of Image Processing 07/11/2014 43 3 minutes break
Roadmap: Basics of Digital Image Processing Computer Vision I: Basics of Image Processing 07/11/2014 44 What is an Image? Point operators (ch. 3.1) Filtering: (ch. 3.2, ch 3.3, ch. 3.4) main focus Linear filtering Non-linear filtering Multi-scale image representation (ch. 3.5) (will be done in SS15 as part of Image Processing) Edges detection and linking (ch. 4.2) Line detection and vanishing point detection (ch. 4.3) (will be done in SS15 as part of Image Processing) Interest Point detection (ch. 4.1.1)
What region should we try to match? Computer Vision I: Basics of Image Processing 07/11/2014 45 We want to find a few regions where this image pair matches: Applications later Look for a region that is unique, i.e. not ambiguous
Goal: Interest Point Detection Computer Vision I: Basics of Image Processing 07/11/2014 46 Goal: predict a few interest points in order to remove redundant data efficiently Should be invariant against: a. Geometric transformation scaling, rotation, translation, affine transformation, projective transformation etc. b. Color transformation additive (lightning change), multiplicative (contrast), linear (both), monotone etc.; c. Discretization (e.g. spatial resolution, focus);
Points versus Lines Computer Vision I: Basics of Image Processing 07/11/2014 47 Apeture problem Lines are not as good as points?
Harris Detector Computer Vision I: Basics of Image Processing 07/11/2014 48 Local measure of feature uniqueness: Shifting the window in any direction: how does it change [Szeliski and Seitz]
Harris Detector Computer Vision I: Basics of Image Processing 07/11/2014 49 How similar is the image Autocorrelation function: to itself? Δy (x, y) Δx is a small window around is a convolution kernel and used to decrease the influence of pixels far from, e.g. the Gaussian For simplicity we use w(u, v) = 1
Harris Detector Computer Vision I: Basics of Image Processing 07/11/2014 50 One is interested in properties of at each position Let us look at a linear approximation of Taylor expansion around + ε(δx, Δy) Gradient at (u, v)
Harris Detector Computer Vision I: Basics of Image Processing 07/11/2014 51 Put it together: with Q is call the Structure Tensor We compute this at any image location (x, y)
Harris Detector Computer Vision I: Basics of Image Processing 07/11/2014 52 The autocorrelation function Function c is (after approximation) a quadratic function in Isolines are ellipses ( is symmetric and positive definite); Eigenvector x 1 with (larger) Eigenvalue λ 1 is the direction of fastest change in function c Eigenvector x 2 with (smaller) Eigenvalue λ 2 is direction of slowest change in function c and x 2 Δx Δy x 1 Function c Note c = 0 for Δx = Δy = 0
Computer Vision I: Basics of Image Processing 07/11/2014 53 Harris Detector Some examples isolines for : (a) Flat (b) Edges (c) Corners a. Homogenous regions: both -s are small b. Edges: one is small the other one is large c. Corners: both -s are large (this is what we are looking for!)
Harris Detector Computer Vision I: Basics of Image Processing 07/11/2014 54 Image λ 1 (larger eigenvalue) λ 2 (smaller eigenvalue) Zoom in
Computer Vision I: Basics of Image Processing 07/11/2014 55 Harris detector Cornerness is a characteristic of Proposition by Harris: Downweights edges where λ 1 λ 2 Harris Value Smallest eigenvalue
Harris Corners - example Computer Vision I: Basics of Image Processing 07/11/2014 56
H-score (red- high, blue - low) Computer Vision I: Basics of Image Processing 07/11/2014 57
Threshold (H-score > value) Computer Vision I: Basics of Image Processing 07/11/2014 58
Non-maximum suppression Computer Vision I: Basics of Image Processing 07/11/2014 59
Harris corners in red Computer Vision: Algorithms and Applications -- 07/11/2014 60
Other examples Computer Vision I: Basics of Image Processing 07/11/2014 61
Maximally stable extremal regions Computer Vision I: Basics of Image Processing 07/11/2014 62 Invariant to affine transformation of gray-values Both small and large structures are detected
Literature Computer Vision I: Basics of Image Processing 07/11/2014 63 There is a large body of literature on detectors and descriptors (later lecture) A comparison paper (e.g. what is the most robust corner detectors): K. Mikolajczyk, T. Tuytelaars, C. Schmid, A. Zisserman, J. Matas, F. Schaffalitzky, T. Kadir: A Comparison of Affine Region Detectors (IJCV 2006)