Image Sampling and Quantisation Introduction to Signal and Image Processing Prof. Dr. Philippe Cattin MIAC, University of Basel 1 of 46 22.02.2016 09:17
Contents Contents 1 Motivation 2 Sampling Introduction and Motivation Sampling Example Quantisation Example 2.1 Tessellation Tessellation Tessellation Examples by M.C. Escher (1) Tessellation Examples by M.C. Escher (2) Tessellation Basics Tessellation Claim How Many Tessellations Exist with Regular Polygons? Combinatorial Analysis All Semi-Regular Tessellations All Regular Tessellations Tessellation Rules Advantages of Square Tessellation 2.2 A Sampling Model A Sampling Model The Neighbourhood Function Fourier Transform of the Neighbourhood Function Filtering with the Neighbourhood Function Sampling of a Continuous 1D Function Sampling of a Continuous 1D Function (2) Sampling of a Discrete 1D Function An Alternative Reasoning for Periodicity in the DFT Sampling of Two-Dimensional Functions 3 4 5 8 9 10 11 12 13 14 15 16 17 18 20 21 22 23 24 25 26 27 28 2 of 46 22.02.2016 09:17
(Images) Summary Sampling Theorem Aliasing Example 1 Aliasing Example 2 Aliasing Example 3 Remark on the Discrete Fourier Transform Linear, Shift-Invariant Operators Linear, Shift-Invariant Operators (2) Liner, Shift-Invariant Operators (3) Liner, Shift-Invariant Operators (4) 3 Quantisation Quantisation Lloyd-Max Quantisation Quantisation Example Quantisation Example (2) Quantisation Example (3) 29 30 31 32 33 34 35 36 37 39 40 41 42 43 3 of 46 22.02.2016 09:17
Motivation Introduction and Motivation (3) In order for computers to process an image, this image has to be described as a series of numbers, each of finite precision This calls for two kinds of discretisation: Sampling, and Quantisation By sampling is meant that the brightness information is only stored at a discrete number of locations. Quantisation indicates the discretisation of the brightness levels at these positions. 4 of 46 22.02.2016 09:17
Motivation Sampling Example (4) Sampling is the process of measuring the brightness information only at a discrete number of locations Fig 4.1: Hight profile of Switzerland Fig 4.2: Sampled hight profile 5 of 46 22.02.2016 09:17
Motivation Quantisation Example (5) Quantisation is the process of discretising the brightness at a finite number of positions Height map with grey values with grey values with grey values with grey values Fig 4.3: 6 of 46 22.02.2016 09:17
Sampling Tessellation Tessellation (8) Definition Tessellations are patterns that cover a plane with repeating figures so there is no overlapping or empty spaces Sampling is best performed following a regular tessellation of the image: 1. Brightness is integrated over cells of same size 2. Cells should cover the whole image These cells are usually referred to as picture elements or pixels. 7 of 46 22.02.2016 09:17
Tessellation Tessellation Examples by M.C. Escher (1) (9) Fig 4.4: Sample Escher images 8 of 46 22.02.2016 09:17
Tessellation Tessellation Examples by M.C. Escher (2) (10) Fig 4.5: Sample Escher images 9 of 46 22.02.2016 09:17
Tessellation Tessellation Basics (11) Three types of tessellations with polygons exist 1. regular tessellations (using the same regular polygon) 2. semi-regular tessellations (using various regular polygons) 3. hyperbolic tessellations (they use non-regular polygons) They are formed by translating, rotating, and reflecting polygons Fig 4.6: regular Fig 4.7: semi-regular Fig 4.8: hyperbolic 10 of 46 22.02.2016 09:17
Tessellation Tessellation Claim (12) There exist only 11 possible tessellations with regular polygons that can cover the entire image 11 of 46 22.02.2016 09:17
Tessellation How Many Tessellations (13) Exist with Regular Polygons? Observation 1: Since the regular polygons in a tessellation must fill the plane at each vertex, the interior angle must be an exact divisor of Observation 2: A regular -gon has an internal angle of degrees Fig 4.9: Of the regular polygons, only triangles ( ), squares ( ), pentagons ( ), hexagons ( ), octagons ( ), decagons ( ) and dodecagons ( ) can be used for tiling around a common vertex - again because of the angle value 12 of 46 22.02.2016 09:17
Tessellation Combinatorial Analysis (14) A combinatorial analysis of these base polygons produces the following 14 solutions Regular Tessellations 4.6. Semi-regular Tessellations 3.3.3.3.4.3.4. Semi-regular Tessellations that can not be extended infinitely 3. 5. Fig 4.10: Tessellations 13 of 46 22.02.2016 09:17
Tessellation All Semi-Regular Tessellations (15) Eight semi-regular tessellations exist Snub hexagonal Trihexagonal Prismatic trisquare Snub square Small rhombitrihexagonal Truncated square Fig 4.11: Truncated hexagonal Great rhombitrihexagonal 14 of 46 22.02.2016 09:17
Tessellation All Regular Tessellations (16) But only three regular tessellations exist Triangular tiling Square tiling Hexagonal tiling Fig 4.12: 15 of 46 22.02.2016 09:17
Tessellation Tessellation Rules (17) For practical applications in computer vision the tessellation has to adhere to the following rules The tessellation must tile an infinite area with no gaps or overlapping Each vertex must look the same The tiles must all be the same regular polygon This leaves us with the following three regular tessellations Regular Tessellations 4. 6. Although the hexagonal tessellation offers some substantial advantages (e.g. no ambiguities in defining connectedness, closer spatial organisation as found in mammalian retinas), the square tessellation is the most commonly used. 16 of 46 22.02.2016 09:17
Tessellation Advantages of Square Tessellation (18) They directly support operations in the Cartesian coordinate frame Most algorithms (FFT, Image pyramids) are based on square tessellations The resolution is often a power of 2: e.g. 16x16, 32x32,..., 256x256, 512x512 17 of 46 22.02.2016 09:17
A Sampling Model A Sampling Model (20) As we have seen, The intensity value attributed to a pixel corresponds to the integration of the incoming irradiance over a cell of the tessellation The cells are only located at discrete locations The sampling process can thus be modeled in a 2-step scheme: 1. Integrate brightness over regions of the pixel size, 2. Read out values only at the pixel positions. 18 of 46 22.02.2016 09:17
A Sampling Model The Neighbourhood Function (21) First a neighbourhood function has to be defined, that is 1 inside a region with the shape of a pixel/cell and 0 outside. Integrating the incoming intensity region then yields rewriting this expression as over such a (4.1) Fig 4.13: Neighbourhood function for square pixels (4.2) we recognise it as the convolution of with which can also be written as. Since is symmetric we can equally well write. 19 of 46 22.02.2016 09:17
A Sampling Model Fourier Transform of the Neighbourhood Function (22) To gain a deeper understand of the sampling model we need its Fourier Transform : (4.3) Fig 4.14:, the Fourier Transform of the neighbourhood function (notice the negative values) Because is real and even its Fourier Transform is too the neighbourhood filter will not change the phase but only their amplitude. Since becomes negative for some some frequencies undergo a complete phase reversal (shift over - see next slide). 20 of 46 22.02.2016 09:17
A Sampling Model Filtering with the Neighbourhood Function (23) As the Fourier Transform of the neighbourhood function has negative amplitudes for some frequencies, complete phase reversals can be observed for higher frequencies: Fig 4.15: Star pattern that increases its frequency towards the centre Fig 4.16: Complete phase reversals occur at higher frequencies 21 of 46 22.02.2016 09:17
A Sampling Model Sampling of a Continuous 1D Function (24) As the second step after filtering with the neighbourhood function we have to select values only at discrete pixel positions. This is modelled as a multiplication with a 1D or 2D pattern (train) of Dirac impulses at these discrete positions. Consider the real neighbourhood function filtered Suppose its Fourier Transform is band limited and thus vanishes outside the interval To obtain a sampled version of simply involves multiplying it by a sampling function, which consists of a train of Dirac impulses apart Its Fourier Transform is also a train of Dirac impulses with a distance inversely proportional to, namely apart By the convolution theorem multiplication in the image domain is equivalent to convolution in the frequency domain The transform is periodic, with period, and the individual repetitions of can overlap aliasing!!! The centre of the overlap occurs at To avoid these problems, the sampling interval has to be selected so that, or (4.4) 22 of 46 22.02.2016 09:17
Once the individual are separated a multiplication with the window function yields a completely isolated The inverse Fourier Transform then yields the original continuous function Complete recovery of a band-limited function that satisfies the above inequality is known as the Whittaker- Shannon Sampling Theorem 23 of 46 22.02.2016 09:17
A Sampling Model Sampling of a Continuous 1D Function (2) (25) All the frequency domain information of a band-limited function is contained in the interval If the Whittaker-Shannon Sampling Theorem or Nyquist Sampling Theorem (4.5) is not satisfied, the transform in this interval is corrupted by contributions from adjacent periods. This phenomenon is frequently referred to as aliasing. 24 of 46 22.02.2016 09:17
A Sampling Model Sampling of a Discrete 1D Function (26) The preceding example applies to functions of unlimited duration in the spatial domain. For practical examples only functions sampled over a finite region are of interest. This situation is shown graphically below Consider a real neighourhoodfunction-filtered function Suppose its Fourier Transform is band limited and thus vanishes outside the interval The sampling function fulfils the Whittaker-Shannon Theorem As the Whittaker-Shannon Sampling Theorem (aka Nyquist Criterion) is fulfilled, the are well separated and no aliasing is present The Sampling Window and its Fourier Transform has Frequency components that extend to infinity Because has frequency components that extend to infinity, the convolution of these functions introduces a distortion in the frequency domain representation of a function that has been sampled and limited to a finite region by 25 of 46 22.02.2016 09:17
These considerations lead to the important conclusion that No function of finite duration can be band limited Conversely, A function that is band limited must extend from in the spatial domain to These important practical results establish fundamental limitations to the treatment of digital functions. 26 of 46 22.02.2016 09:17
A Sampling Model An Alternative Reasoning for Periodicity in the DFT (27) So far, all the results in the Fourier domain have been of a continuous nature. To obtain a discrete Fourier Transform simply requires to sample it with a train of Dirac impulses that are units apart. Consider the signals and as the results of the operation sequence on the previous slide To sample we multiply it with a train of Dirac impulses that are units apart The inverse Fourier Transform of yields, an other train of Dirac impulses with inversely spaced pulses The graph shows the result of sampling As the equivalent of a multiplication in the Fourier domain is a convolution in the spatial domain, it yields a periodic function, with period If samples of and are taken and the spacings between samples are selected so that a period in each domain is covered by uniformly spaced samples, then in the spatial domain and in the frequency domain. The latter equation is based on the periodic property of the Fourier Transform of a sampled function, with period, as shown earlier. The Sampling Theorem for discrete signals can thus be formulated as 27 of 46 22.02.2016 09:17
(4.6) 28 of 46 22.02.2016 09:17
A Sampling Model Sampling of Two-Dimensional Functions (Images) (28) The preceding sampling concepts (after some modifications in notation) are directly applicable to 2D functions The sampling process for these functions can be formulated making use of a 2D train of Dirac impulses For a function, where and are continuous, a sampled function is obtained by forming the product. The equivalent operation in the Frequency domain is the convolution of and, where is a train of Dirac impulses with separation and. If is band limited it might look like shown on the right Let and represent the widths in and direction that completely enclose the band-limited function No aliasing is present if and The 2D sampling theorem can thus be formulated as (4.7) and (4.8) A periodicity analysis similar to the discrete 1D case shown previously would yield a 2D Sampling Theorem of (4.9) and 29 of 46 22.02.2016 09:17
(4.10) 30 of 46 22.02.2016 09:17
A Sampling Model Summary Sampling Theorem (29) The One-Dimensional Sampling Theorem states that If the Fourier Transform of a function is zero for all Frequencies beyond, i.e. the Fourier Transform is band-limited, then the continuous function can be completely reconstructed as long as. The Two-Dimensional Sampling Theorem states that If the Fourier Transform of a function is zero for all Frequencies beyond, i.e. the Fourier Transform is band-limited, then the continuous function can be completely reconstructed as long as and. 31 of 46 22.02.2016 09:17
A Sampling Model Aliasing Example 1 (30) The input image contains regions with clearly different frequency content. Going from the centre to boundary, the frequency increases. It can be seen that once the Nyquist rate is higher than the actual sampling, aliasing occurs. (a) Original pattern (b) Sinc size 5 (a) the 256x256 sample pattern (b) the sinc function for a sampling rate of (grey is zero, brighter is positive, and darker is negative) (c) the original pattern is sampled with (d) the reconstructed pattern. In regions where the Nyquist rate is higher strong aliasing artefacts are present (c) Sampled pattern (d) Reconstruction Fig 4.17 Aliasing example 32 of 46 22.02.2016 09:17
A Sampling Model Aliasing Example 2 (31) This example shows the reconstruction of the rolling pattern for a sampling rate ( ) that is well above the Nyquist rate. (a) the 128x128 sample rolling pattern (b) the sinc function for a sampling rate of. The grey background is zero, brighter is positive, and darker is negative (c) the original pattern is sampled with (d) the reconstructed rolling pattern. The reconstruction is perfect (except for boundary effects) (a) Original pattern (b) Sinc of size 5 (c) Sampled pattern (d) Reconstruction Fig 4.18 Aliasing example 2 33 of 46 22.02.2016 09:17
A Sampling Model Aliasing Example 3 (32) In this example the sampling rate ( ) is below the Nyquist rate. (a) the 128x128 sample rolling pattern (b) the sinc function for a sampling rate of. The grey background is zero, brighter is positive, and darker is negative (c) the original pattern is sampled with (d) the reconstructed rolling pattern is no longer valid. It is interesting that not only the frequency changed, but even the orientation of the pattern. (a) Original pattern (b) Sinc size 15 (c) Sampled pattern (d) Reconstruction Fig 4.19 Aliasing example 3 34 of 46 22.02.2016 09:17
A Sampling Model Remark on the Discrete Fourier Transform (33) As already noted, Sampling in one domain implies periodicity in the other If both domains are discretised and thus should both the original image and its Fourier Transform be interpreted as periods of periodic signals. The discrete Fourier Transform is therefore not the Fourier Transform of the image as such, but rather of the periodic signal created by repeating the image data both horizontally and vertically Periodically repeated image Flipped images 35 of 46 22.02.2016 09:17
A Sampling Model Linear, Shift-Invariant Operators (34) Convolution theory is not only important in image acquisition but plays an important role at several other occasions. To fully benefit from the convolution theorem a little bit more background theory is required. In fact, it will be explained that Every linear, shift-invariant operation can be expressed as a convolution and vice versa. Definition: Consider a 2D system that produces output and when given inputs and respectively. The system is called linear if the output is produced when the input is Fig 4.20: Linear system The system is called shiftinvariant if the output is produced when the input is Fig 4.21: Shift-invariant system 36 of 46 22.02.2016 09:17
A Sampling Model Linear, Shift-Invariant Operators (2) (35) Suppose a process, e.g. camera with lens system, can be modeled as a linear, shift-invariant operation. As we have seen, any image can be considered as a sum of point sources (Dirac impulses). The output of for a single point source is called Point spread function (PSF) of which we denote as. Fig 4.22: Point spread function Knowledge of the PSF can be used to determine the output for Assuming shift-invariance implies that the output to such a Dirac pulse is always the same irrespective of its position. In terms of image acquisition, we assume that the light comming from a point source will be distributed over the image following a fixed spatial pattern. The projection of such a point will therefore always be blurred in the same way independent of its position in the image. 37 of 46 22.02.2016 09:17
A Sampling Model Liner, Shift-Invariant Operators (3) (36) Let us consider an input picture linear combination of point sources. It can be written as a (4.11) For the linear and shift-invariant operation we obtain (4.12) The linear, shift-invariant operation has led to a convolution operation. This is true in general and every LSI operation can be written as a convolution and vice versa. A simple variable substitution shows that the above expression can also be written as (4.13) so that (4.14) i.e. convolution is commutative (convolution is also associative). 38 of 46 22.02.2016 09:17
39 of 46 22.02.2016 09:17
A Sampling Model Liner, Shift-Invariant Operators (4) (37) Suppose we would like to process an image by first convolving with, followed by a convolution with, thus (4.15) the global operation can therefore be interpreted as applying a single (generally larger) filter. The reverse analysis might be useful too, i.e. if a filter (separable) can be decomposed as a convolution of two simpler filter efficiency can be increased by applying the smaller filters sequentially. Example The Figures on the right show a 2D Gauss kernel and a 1D Gauss kernel of size and respectively. Fig 4.23: 2D Gauss kernel It can be easily shown numerically that the kernel can be separated into two 1D kernels and thus (4.16) Convolving the image sequentially with the 1D kernels is computationally more efficient than convolving the entire image with the 2D kernel. Fig 4.24: 1D Gauss kernel 40 of 46 22.02.2016 09:17
Quantisation Quantisation (39) The subjective image quality depends on (1) the number of samples and (2) the number of grey-values. Figure 4.26 shows this relation. The key point of interest is, that isopreference curves tend to become more vertical as the detail in the image increases images with large amount of detail require fewer grey levels. Fig 4.25: (a) Low detail face image, (b) Cameraman with mid detail, and (c) crowd with high detail content Fig 4.26: Isopreference curves for the three sample images 41 of 46 22.02.2016 09:17
Quantisation Lloyd-Max Quantisation (40) In the Introduction of this Lecture we have already shortly explained the effect of using more or less quantisation levels. This part is concerned with the optimal placement of these quantisation levels Suppose we create intervals in the range of possible intensities, defined by the decision levels. Fig 4.27: Principle of the Lloyd-Max quantiser We therefore assign to all intensities in the interval the new grey level. The mean-square quantisation error between the input and output of the quantiser for a given choice of boundaries and output levels is thus (4.17) where is the probability density function for the input sample value. For a given number of output levels, we would like to determine the output levels and interval boundaries that minimise. The partial derivatives of with respect to and must thus vanish: (4.18) 42 of 46 22.02.2016 09:17
For not equal to zero we obtain the Lloyd-Max Quantiser Equations (4.19) We see that the decision levels are located halfway between the output levels whilst each is the centroid of the portion of between and If the sample values occur equally frequently, the optimal quantised will spread the values and uniformly, and the Lloyd-Max Quantiser Equations can be simplified to (4.20) As can be seen from the following examples, improvement can be disputed. The main problem is, that Lloyd-Max quantisation does not take local image structure or interpretation into account. 43 of 46 22.02.2016 09:17
Quantisation Quantisation Example (41) Original image with 256 grey values 32 equally spaced grey values 32 Lloyd-max quantised grey values Fig 4.28: Quantisation example with 32 grey values 44 of 46 22.02.2016 09:17
Quantisation Quantisation Example (2) (42) Original image with 256 grey values 16 equally spaced grey values 16 Lloyd-max quantised grey values Fig 4.29: Quantisation example with 16 grey values 45 of 46 22.02.2016 09:17
Quantisation Quantisation Example (3) (43) Original image with 256 grey values 8 equally spaced grey values 8 Lloyd-max quantised grey values Fig 4.30: Quantisation example with 8 grey values 46 of 46 22.02.2016 09:17