11. Image Data Analytics. Jacobs University Visualization and Computer Graphics Lab

Transcription

1 11. Image Data Analytics

2 Motivation Images (and even videos) have become a popular data format for storing information digitally. Data Analytics 377

3 Motivation Traditionally, scientific and medical imaging techniques were considered. Data Analytics 378

4 Motivation Nowadays, photographs form the majority. 1 pixel Data Analytics 379

5 Image Analytics? Similarity computation is difficult. Similar images may look very different. E.g., two photographs of cars may have completely different colors and shapes. Digital image processing allows for the detection of features in the images. Semantic queries on images is difficult without metadata. Data Analytics 380

6 Semantic gap Image processing allows for the detection of regions of certain properties (typically homogeneous regions). Heuristics can be used to combine homogeneous regions to larger structures. Larger structures can be considered as low-level semantics. E.g., detect human hands. The problem is to bridge the low-level semanstics to high-level semantics as in queries: E.g., show me all images, where presidents are shaking hands. The gap between the low-level and high-level semantics is called the semantic gap. Data Analytics 381

7 11.1 Digital Image Processing

8 Digital Image Processing Image Restoration Morphological Processing Image Enhancement Segmentation Image Acquisition Object Recognition Problem Domain Colour Image Processing Image Compression Representation & Description Data Analytics 383

9 Image Aquisition Image Restoration Morphological Processing Image Enhancement Segmentation Image Acquisition Object Recognition Problem Domain Colour Image Processing Image Compression Representation & Description Data Analytics 384

10 Image Enhancement Image Restoration Morphological Processing Image Enhancement Segmentation Image Acquisition Object Recognition Problem Domain Colour Image Processing Image Compression Representation & Description Data Analytics 385

11 Image Restoration Image Restoration Morphological Processing Image Enhancement Segmentation Image Acquisition Object Recognition Problem Domain Colour Image Processing Image Compression Representation & Description Data Analytics 386

12 Morphological Processing Image Restoration Morphological Processing Image Enhancement Segmentation Image Acquisition Object Recognition Problem Domain Colour Image Processing Image Compression Representation & Description Data Analytics 387

13 Segmentation Image Restoration Morphological Processing Image Enhancement Segmentation Image Acquisition Object Recognition Problem Domain Colour Image Processing Image Compression Representation & Description Data Analytics 388

14 Object Recognition Image Restoration Morphological Processing Image Enhancement Segmentation Image Acquisition Object Recognition Problem Domain Colour Image Processing Image Compression Representation & Description Data Analytics 389

15 Representation & Description Image Restoration Morphological Processing Image Enhancement Segmentation Image Acquisition Object Recognition Problem Domain Colour Image Processing Image Compression Representation & Description Data Analytics 390

16 Image Compression Image Restoration Morphological Processing Image Enhancement Segmentation Image Acquisition Object Recognition Problem Domain Colour Image Processing Image Compression Representation & Description Data Analytics 391

17 Colour Image Processing Image Restoration Morphological Processing Image Enhancement Segmentation Image Acquisition Object Recognition Problem Domain Colour Image Processing Image Compression Representation & Description Data Analytics 392

18 11.2 Image Segmentation Data Analytics 393

19 Definition Image segmentation is the operation of partitioning an image into a collection of connected sets of pixels. Data Analytics 394

20 Definition A segmentation is a partition of an image I into a set of regions S satisfying the following conditions: 1. Partition covers the whole image. 2. No regions intersect. 3. Each region is homogeneous within itself. 4. Adjacent regions form no homogeneous region when united. Data Analytics 395

21 Region Growing Region growing techniques start with one pixel of a potential region and try to grow it by adding adjacent pixels till the pixels being compared are too disimilar. The first pixel selected can be just the first unlabeled pixel in the image or a set of seed pixels can be chosen from the image. Usually a statistical test is used to decide which pixels can be added to a region, e.g., threshold on (N-1) * N T = (y - X) / S (N+1) to decide whether pixel with intensity y is added, where X denotes the mean, S standard deviation, and N is the number of pixels in the region /2 Data Analytics 396

22 Region Growing Result: image segmentation Data Analytics 397

23 Clustering Apply a clustering approach on the pixel intensities. Histogram-based approaches try to automatically split the range of the intensities by looking into minima of the histogram. Data Analytics 398

24 Clustering We can use the clustering methods we have been looking into. For example, we can use k-means with a guessed number of segments k. There exist modifications to k-means that look into local statistics to consider spatial distribution of pixels. Data Analytics 399

25 Results for k-means: Clustering Data Analytics 400

26 Clustering Graph cut algorithms are widely used. Let G = (V,E) be a graph. The vertices V represent the pixels. Each edge (u,v) has a weight w(u,v) that represents the similarity between u and v. The goal is to partition the vertices into disjoint sets with high similarity within each set and low similarity across sets. Graph G can be broken into 2 disjoint graphs by removing edges that connect these sets. The segmentation is obtained by finding minimal cuts of G. Data Analytics 401

27 Clustering Graph cuts with normalized cut: cut(a,b) = uεa, vεb w(u,v). A cut(a, B) cut(a,b) Ncut(A,B) = asso(a,v) asso(b,v) B asso(a,v) = w(u,t) u A, t V 3 3 Ncut(A,B) = Data Analytics 402

28 Clustering Graph cut results: Data Analytics 403

29 11.3 Image Collection Data Analytics Data Analytics 404

30 Image collections Image segmentation helps to identify certain features in the image. They do not work perfectly. However, if we are trying to perform a data analytics approach on a collection of images, the data segmentation results are only of use for low-level semantics queries. Is there a way to analyze a collection of images on a higher level? Data Analytics 405

31 Data analytics Assume that we can characterize an image by a number of image descriptors, we can try to compute similarities based on those descriptors. Then, we can build a similarity (or distance) matrix of pairwise (dis-)similarities of images. Based on the similarity (or distance) matrix, we can apply the data analytics approaches: Cluster approaches, Classification approaches, Interactive visual analysis based on MDS projections. Data Analytics 406

32 Data analytics example Corel data set includes 1,000 photographs on 10 different themes, described by 150 dimensions (SIFT descriptors). The Medical data set is of magnetic resonance (MRI) images and has 540 objects and 28 dimensions (Fourier descriptors and energies derived from histograms, plus mean intensity and standard deviation). Data Analytics 407

33 Projection-based visualization of labeled data Data Analytics 408

34 11.4 Image Descriptors Data Analytics 409

35 Image descriptors Fourier analysis Wavelet analysis SIFT features Color statistics Data Analytics 410

36 Image Transforms Many times, image processing tasks are best performed in a domain other than the spatial domain. Key steps (1) Transform the image (2) Carry the task(s) in the transformed domain. (3) Apply inverse transform to return to the spatial domain. Data Analytics 411

37 Fourier Series Theorem Any periodic function f(t) can be expressed as a weighted sum (infinite) of sine and cosine functions of varying frequency: is called the fundamental frequency Data Analytics 412

38 Fourier Series α 1 α 2 α 3 Data Analytics 413

39 Continuous Fourier Transform (FT) Transforms a signal (i.e., function) from the spatial (x) domain to the frequency (u) domain. where Data Analytics 414

40 Example: Removing undesirable frequencies noisy signal frequencies To remove certain frequencies, set their corresponding F(u) coefficients to zero! remove high frequencies reconstructed signal Data Analytics 415

41 How do frequencies show up in an image? Low frequencies correspond to slowly varying pixel intensities (e.g., continuous surface). High frequencies correspond to quickly varying pixel intensities (e.g., edges) Original Image Low-passed Data Analytics 416

42 Example of noise reduction using FT Input image Spectrum (frequency domain) Bandreject filter Output image Data Analytics 417

43 Extending FT in 2D Forward FT Inverse FT Data Analytics 418

44 Discrete Fourier Transform Assume that f(x,y) is M x N. Forward DFT Inverse DFT: Data Analytics 419

45 Extending DFT to 2D 2D cos/sin functions Interpretation: Data Analytics 420

46 Magnitude and Phase of DFT only phase only magnitude phase (woman) magnitude (rectangle) phase (rectangle) magnitude (woman) Data Analytics 421

47 Wavelet transform An alternative to Fourier transforms are wavelet transforms. They have the advantage that they represent the image at multiple levels of details. Data Analytics 422

48 B-spline representation coarsening details Data Analytics 423

49 B-spline representation Haar wavelet transform Data Analytics 424

50 Haar wavelets Basis function Wavelet function Data Analytics 425

51 Haar wavelets Data Analytics 426

52 Multiresolution representation Data Analytics 427

53 Multiresolution representation Object is represented as a sequence of resolutions. The resolutions are called levels (levels of detail, LOD) The differences are called detail coefficients. The levels build a multiresolution hierarchy: The level is the base level. The base level does not need to be represented by a regular mesh. All levels use then semi-regular meshes. Data Analytics 428

54 Multiresolution representation with Haar wavelets Data Analytics 429

55 2D Haar wavelet transform 2D basis and wavelet functions are tensor products of 1D basis and wavelet functions. Data Analytics 430

56 2D Haar wavelet transform Basis: Data Analytics 431

57 2D Haar wavelet transform Data Analytics 432

58 2D Haar wavelet transform Alternative construction: Use 2D basis function and three 2D wavelet functions Data Analytics 433

59 2D Haar wavelet transform Basis: Data Analytics 434

60 2D Haar wavelet transform Advantage: One obtains undistorted downscaled versions of the 2D image. Data Analytics 435

61 2D wavelet transform in RGB space Data Analytics 436

62 Image compression Haar wavelets: 100% 21% 4% 1% Error: 0% 5% 10% 15% Data Analytics 437

63 Image compression JPEG 2000: Cohen-Daubechies-Feauveau wavelets Data Analytics 438

64 Image compression JPEG 2000: lossy compression leads to blurring. Data Analytics 439

65 SIFT features The scale-invariant feature transform (SIFT) is another transform that can be used to describe image characteristics. Data Analytics 440

66 Canonical Frames Data Analytics 441

67 Multi-Scale Oriented Patches Extract oriented patches at multiple scales Data Analytics 442

68 Application: Image Stitching Data Analytics 443

69 Multi-Scale Oriented Patches 1. Detect an interesting patch with an interest operator. Patches are translation invariant. 2. Determine its dominant orientation. 3. Rotate the patch so that the dominant orientation points upward. This makes the patches rotation invariant. 4. Do this at multiple scales, converting them all to one scale through sampling. 5. Convert to illumination invariant form Data Analytics 444

70 Idea of SIFT Image content is transformed into local feature coordinates that are invariant to translation, rotation, scale, and other imaging parameters Data Analytics 445

71 Claimed Advantages of SIFT Locality: features are local, so robust to occlusion and clutter (no prior segmentation) Distinctiveness: individual features can be matched to a large database of objects Quantity: many features can be generated for even small objects Efficiency: close to real-time performance Extensibility: can easily be extended to wide range of differing feature types, with each adding robustness Data Analytics 446

72 Overall Procedure at a High Level 1. Scale-space extrema detection Search over multiple scales and image locations. 2. Keypoint localization Fit a model to determine location and scale. Select keypoints based on a measure of stability. 3. Orientation assignment Compute best orientation(s) for each keypoint region. 4. Keypoint description Use local image gradients at selected scale and rotation to describe each keypoint region. Data Analytics 447

73 Using SIFT for Matching Objects 11/15/ Data Analytics 448

74 Color statistics We have been looking at greyscale images (application: medical imaging data). We have been using histograms on color distribution. However, photographs are typically color images. How can we process color statistics? Data Analytics 449

75 11.5 Color Models Data Analytics 450

76 Electromagnetic spectrum purple blue green yellow orange red Data Analytics 451

77 Visible light spectrum Data Analytics 452

78 Relative sensitivity of human eye The ability of the human eye to distinguish colors is based on the sensitivity in the retina to light of different wavelength. Data Analytics 453

79 Tristimulus The retina contains 3 types of receptor cells, which is called tristimulus. The receptors are most responsive to light of wavelengths 420nm, 534nm, and 564nm. Data Analytics 454

80 RGB color model Idea: Use three wavelengths R, G, and B that reflect monochromatic light and represent a tristimulus. Other colors are obtained by combining/mixing the three components R, G, and B. Data Analytics 455

81 RGB color model Implementation: Choose three colors: R = red (700 nm) G = green (546.1 nm) B = blue (435.8 nm) Arrange them in a 3D Cartesian coordinate system such that Data Analytics 456

82 RGB color model This model allows the generation of colors c with where. Data Analytics 457

83 RGB color cube All colors c that can be generated are represented by the unit cube in the 3D Cartesian coordinate system. green yellow cyan grey white black red blue magenta Data Analytics 458

84 Additive color scheme RGB color model is additive, i.e., adding colors makes the resulting color brighter. Application: color monitors. Data Analytics 459

85 Composition example Data Analytics 460

86 Compositing in the RGB color cube Data Analytics 461

87 Caveat It is often believed that the RGB color model reflects the tristimulus of the human eye. Thisiswrong. In particular, the large wavelength of the human eye s tristimulus is 565 nm, which is not red but rather yellow-green. Data Analytics 462

88 Choice of RGB wavelength The choice of the wavelength in the RGB model has historical and practical reasons. When first monitors were developed, generating monochromatic light was a difficult task. The chosen wavelength were those that could be generated most easily. Data Analytics 463

89 Reconstruction of visible spectral light In order to reconstruct all wavelengths of the visible spectral light, we have to add the R, G, and B components with the following weighting factors: Data Analytics 464

90 Reconstruction of visible spectral light The negative values indicate that not all visible colors can be produced with the RGB color model. Nevertheless, close approximations can be achieved. Data Analytics 465

91 11.6 Assignment

92 Assignment 9 Download and extract the archive of an image collection data set from The data set consists of around 5,000 images of different buildings in Oxford. If processing of the entire data set takes too long, you may use a subset, but make sure the subset contains images of different buildings. Then, convert the color images to grayscale images. Approach 1: Interpreting the grayscale images as 1D vectors of pixel intensities, run a clustering algorithm with an appropriate distance metric. What do the clusters represent? Approach 2: Transform the greyscale images into the space of SIFT descriptors. How many dimensions does this space have? Hint: You may use the Python bindings of OpenCV. Here is a tutorial: py_feature2d/py_sift_intro/py_sift_intro.html. Apply a clustering approach to the SIFT descriptors and visualise the result in an MDS projection (in 2D). What do the clusters represent? p.t.o Data Analytics 467

93 Assignment 9 Approach 3: Exchange the order of the two analysis steps in Approach 2, i.e., first project the SIFT descriptors to a 2D space using an MDS approach and then cluster the 2D points using the same clustering approach as above. Again, visualise the result. What do the discovered clusters represent? How do the three approaches compare? Data Analytics 468