Image Processing. Overview. Trade spatial resolution for intensity resolution Reduce visual artifacts due to quantization. Sampling and Reconstruction

Transcription

1 Image Processing Overview Image Representation What is an image? Halftoning and Dithering Trade spatial resolution for intensity resolution Reduce visual artifacts due to quantization Sampling and Reconstruction Key steps in image processing Avoid visual artifacts due to aliasing 1

2 What is an Image? An image is a 2D rectilinear array of pixels Continuous image Digital image What is an Image? An image is a 2D rectilinear array of pixels Continuous image Digital image 2

3 What is an Image? An image is a 2D rectilinear array of pixels Continuous image Digital image A pixel is a sample, not a little square!! Image Acquisition Pixels are samples from continuous function Photoreceptors in eye CCD cells in digital camera Rays in virtual camera 3

4 Image Display Re-create continuous function from samples Example: cathode ray tube Image is reconstructed by displaying pixels with finite area (Gaussian) Image Resolution Intensity resolution Each pixel has only Depth bits for colors/intensities Spatial resolution Image has only Width x Height pixels Temporal resolution Monitor refreshes images at only Rate Hz 4

5 Sources of Error Intensity quantization Not enough intensity resolution Spatial aliasing Not enough spatial resolution Temporal aliasing Not enough temporal resolution E 2 = ( x, y ) ( I( x, y) P( x, y) ) 2 Overview Image Representation What is an image? Halftoning and Dithering Trade spatial resolution for intensity resolution Reduce visual artifacts due to quantization Sampling and Reconstruction Key steps in image processing Avoid visual artifacts due to aliasing 5

6 Quantization Artifact due to limited intensity resolution Frame buffers have limited number of bits per pixel Physical devices have limited dynamic range Blue channel Green channel Red channel Uniform Quantization P ( x, y) = trunc( I ( x, y) + 0.5) I(x, y) P(x, y) 2 bits per pixel 6

7 Uniform Quantization Image with decreasing bits per pixel: 8 bits 4 bits 2 bits 1 bit Notice contouring Reducing Effects of Quantization Halftoning Classical halftoning Dithering Random dither Ordered dither Error diffusion dither 7

8 Classical Halftoning Use dots of varying size to representation intensities Area of dots proportional to intensity in image I(x, y) P(x, y) Classical Halftoning Newspaper image From New York Times 9/21/99 8

9 Halftone Patterns Use cluster of pixels to represent intensity Trade spatial resolution for intensity resolution Halftone Patterns How many intensities in a n x n cluster? 9

10 Dithering Distribute errors among pixels Exploit spatial integration in our eye Display greater range of perceptible intensities Original (8 bits) Uniform Quantization (1 bit) Floyd-Steinberg Dither (1 bit) Random Dither Randomize quantization errors Errors appear as noise P ( x, y) = trunc( I ( x, y) + noise( x, y) + 0.5) 10

11 Random Dither Original (8 bits) Uniform Quantization (1 bit) Random Dither (1 bit) Ordered Dither Pseudo-random quantization errors Matrix stores pattern of thresholds j = x mod n i = y mod n e = I(x, y) trunc(i(x, y)) if( e > D(i, j) ) P(x, y) = ceil(i(x, y)) else P(x, y) = floor(i(x, y)) D 2 3 =

12 Ordered Dither Original (8 bits) Uniform Quantization (1 bit) Ordered Dither (1 bit) Error Diffusion Dither Spread quantization error over neighbor pixels Error dispersed to pixels right and below α β γ δ α + β + γ + δ =

13 Error Diffusion Dither Original (8 bits) Random Dither (1 bit) Ordered Dither (1 bit) Floyd-Steinberg Dither (1 bit) Overview Image Representation What is an image? Halftoning and Dithering Trade spatial resolution for intensity resolution Reduce visual artifacts due to quantization Sampling and Reconstruction Key steps in image processing Avoid visual artifacts due to aliasing 13

14 Sampling and Reconstruction Sampling Reconstruction Sampling and Reconstruction 14

15 Aliasing In general: Artifacts due to under-sampling or poor reconstruction Specifically, in graphics: Spatial aliasing Temporal aliasing Under-sampling Spatial Aliasing Artifacts due to limited spatial resolution 15

16 Spatial Aliasing Artifacts due to limited spatial resolution Jaggies Temporal Aliasing Artifacts due to Limited Temporal Resolution Strobing Flickering 16

17 Temporal Aliasing Artifacts due to Limited Temporal Resolution Strobing Flickering Temporal Aliasing Artifacts due to Limited Temporal Resolution Strobing Flickering 17

18 Temporal Aliasing Artifacts due to Limited Temporal Resolution Strobing Flickering Antialiasing Sample at higher rate Not always possible Doesn t always solve problem Pre-filter to form bandlimited signal Form bandlimited function (low-pass filter) Trades aliasing for blurring Must consider sampling theory! 18

19 Sampling Theory How many samples are required to represent a given signal without loss of information? What signals can be reconstructed without loss for a given sampling rate? Sampling Theorem A signal can be reconstructed from its samples, if the original signal has no frequencies above ½ the sampling frequency Shannon The minimum sampling rate for bandlimited function is called Nyquist rate A signal is bandlimited if its highest frequency is bounded. The frequency is called the bandwidth. 19

20 Image Processing Quantization Uniform quantization Random dither Ordered dither Floyd-Steinberg dither Pixel operations Add random noise Add luminance Add contrast Add saturation Filtering Blur Detect edge Warping Scale Rotate Warps Combining Morphs Composite Image Processing Quantization Uniform quantization Random dither Ordered dither Floyd-Steinberg dither Pixel operations Add random noise Add luminance Add contrast Add saturation Filtering Blur Detect edge Warping Scale Rotate Warps Combining Morphs Composite 20

21 Image Processing Quantization Uniform quantization Random dither Ordered dither Floyd-Steinberg dither Pixel operations Add random noise Add luminance Add contrast Add saturation Filtering Blur Detect edge Warping Scale Rotate Warps Combining Morphs Composite Adjusting Brightness Simply scale pixel components Must clamp to range (e.g., 0 to 255) Original Brighter 21

22 Adjusting Contrast Compute mean luminance L for all pixels Luminance = 0.30*r *g *b Scale deviation from L for each pixel component Must clamp to range (e.g. 0 to 255) L Original More contrast Image Processing Quantization Uniform quantization Random dither Ordered dither Floyd-Steinberg dither Pixel operations Add random noise Add luminance Add contrast Add saturation Filtering Blur Detect edge Warping Scale Rotate Warps Combining Morphs Composite 22

23 Adjusting Blurriness Convolve with a filter whose entries sum to one Each pixel becomes a weighted average of its neighbors Original Blur 1 16 Filter = Edge Detection Convolve with a filter that finds differences between neighbor pixels Original Edge Detection 1 Filter =

24 Image Processing Quantization Uniform quantization Random dither Ordered dither Floyd-Steinberg dither Pixel operations Add random noise Add luminance Add contrast Add saturation Filtering Blur Detect edge Warping Scale Rotate Warps Combining Morphs Composite Image Warping Move pixels of image Mapping Resampling Warp Source Image Destination Image 24

25 Overview Mapping Forward Reverse Resampling Point sampling Triangle filter Gaussian filter Mapping Define transformation Describe the destination (x, y) for every location (u, v) in the source (or vice-versa, if invertible) v y u x 25

26 Example Mappings Scale by factor : x = factor * u y = factor * v v y Scale 0.8 u x Example Mappings Rotate by θ degrees: x = u cos θ v sin θ y = u sin θ + v cos θ y v Rotate 30 u x 26

27 Example Mappings Shear in X by factor : x = u + factor * v y = v v Shear X 1.3 y Shear in Y by factor : x = u y = v + factor * u v u y Shear Y 1.3 x u x Other Mappings Any function of u and v : x = f x (u, v) y = f y (u, v) Fish-eye Swirl Rain 27

28 Image Warping Implementation I Forward mapping : for(int u=0; u<umax; u++) { for(int v=0; v<vmax; v++) { float x = f x (u,v); float y = f y (u,v); dst(x,y) = src(u,v); } } (u, v) f (x, y) Source Image Destination Image Forwarding Mapping Iterate over source image v y Rotate -30 u x 28

29 Forwarding Mapping NOT Iterate over source image Many source pixels can map to same destination pixel y v Rotate -30 u x Forwarding Mapping NOT Iterate over source image v Many source pixels can map to same destination pixel y Some destination pixels may not be covered Rotate -30 u x 29

30 Image Warping Implementation II Reverse mapping for(int x=0; x<xmax; x++) { for(int y=0; y<ymax; y++) { float u = f x -1 (x,y); float v = f y -1 (x,y); dst(x,y) = src(u,v); } } (u, v) f (x, y) Source Image Destination Image Reverse Mapping Iterate over destination image Must resample source May oversample, but much simpler! y v Rotate -30 u x 30

31 Resampling Evaluate source image at arbitrary (u, v) (u, v) does not usually have integer coordinates (u, v) (x, y) Source Image Destination Image Overview Mapping Forward Reverse Resampling Point sampling Triangle filter Gaussian filter 31

32 Point Sampling Take value at closest pixel int iu = trunc(u+0.5); int iv = trunc(v+0.5); dst(x, y) = src(iu, iv); v y This method is simple, but it causes aliasing Rotate -30 Scale 0.5 u x Triangle Filtering Convolve with triangle filter Input Output 32

33 Triangle Filtering Bilinearly interpolate four closest pixels a = linear interpolation of src(u 1, v 2 ) and src(u 2, v 2 ) b = linear interpolation of src(u 1, v 1 ) and src(u 2, v 1 ) dst(x, y) = linear interpolation of a and b a (u 1, v 2 ) (u 2, v 2 ) (u, v) (u 1, v 1 ) b (u 2, v 1 ) Gaussian Filtering Convolve with Gaussian filter Input Output Width of Gaussian kernel affects bluriness 33

34 Gaussian Filtering Compute weighted sum of pixel neighborhood : Weights are normalized values of Gaussian function (u, v) Filtering Methods Comparison Trade-offs Aliasing versus blurring Computation speed Point Bilinear Gaussian 34

35 Image Warping Implementation III Reverse mapping for(int x=0; x<xmax; x++) { for(int y=0; y<ymax; y++) { float u = f x -1 (x,y); float v = f y -1 (x,y); dst(x,y) = resample_src(u,v,w); } } (u, v) f (x, y) Source Image Destination Image Image Warping Implementation III Reverse mapping for(int x=0; x<xmax; x++) { for(int y=0; y<ymax; y++) { float u = f x -1 (x,y); float v = f y -1 (x,y); dst(x,y) = resample_src(u,v,w); } } (u, v) w f (x, y) Source Image Destination Image 35

36 Example: Scale Scale (src, dst, sx, sy) : float w max(1/sx, 1/sy) for(int x=0; x<xmax; x++) { for(int y=0; y<ymax; y++) { float u = x/sx ; float v = y/sy; dst(x,y) = resample_src(u,v,w); } v y } (u, v) Scale 0.5 (x, y) u x Example: Rotate Rotate (src, dst, theta) for(int x=0; x<xmax; x++) { for(int y=0; y<ymax; y++) { float u = x*cos(-θ)-y*sin(-θ) float v = x*sin(-θ)+y*cos(-θ) dst(x,y) = resample_src(u,v,w); } y } v (u, v) (x, y) Rotate 30 u x 36

37 Example: Fun Swirl (src, dst, theta) for(int x=0; x<xmax; x++) { for(int y=0; y<ymax; y++) { float u = rot(dist(x,xcenter)*θ) float v = rot(dist(y,ycenter)*θ) dst(x,y) = resample_src(u,v,w); } } v (u, v) y (x, y) Swirl 45 u x Image Processing Quantization Uniform quantization Random dither Ordered dither Floyd-Steinberg dither Pixel operations Add random noise Add luminance Add contrast Add saturation Filtering Blur Detect edge Warping Scale Rotate Warps Combining Morphs Composite 37

38 Image Processing Quantization Uniform quantization Random dither Ordered dither Floyd-Steinberg dither Pixel operations Add random noise Add luminance Add contrast Add saturation Filtering Blur Detect edge Warping Scale Rotate Warps Combining Morphs Composite Image Processing Quantization Uniform quantization Random dither Ordered dither Floyd-Steinberg dither Pixel operations Add random noise Add luminance Add contrast Add saturation Filtering Blur Detect edge Warping Scale Rotate Warps Combining Morphs Composite 38

39 Overview Image compositing Blue-screen mattes Alpha channel Porter-Duff compositing algebra Image morphing Specifying correspondences Warping Blending Image Compositing Separate an image into elements Render independently Composite together Applications Cel animation Chroma-keying Blue-screen matting Dobkin meets Elvis 39

40 Blue-Screen Matting Composite foreground and background images Create background image Create foreground image with blue background Insert non-blue foreground pixels into background Alpha Channel Encodes pixel coverage information α = 0 : no coverage (or transparent) α = 1 : full coverage (or opaque) 0 < α < 1 : partial coverage (or semi-transparent) Example : α = 0.3 or Partial coverage Semi-Transparent 40

41 Compositing with Alpha Controls the linear interpolation of foreground and background pixels when elements are composited α = 0 0< α < 1 α = 1 Pixels with Alpha Alpha channel convention : (r, g, b, α) represents a pixel that is α covered by the color C=(r/α, g/α, b/α) Color components are premultiplied by α Can display (r, g, b) values directly Closure in composition algebra What is the meaning of the following? (0, 1, 0, 1) = (0, ½, 0, 1) = (0, ½, 0, ½) = (0, ½, 0, 0) = 41

42 Pixels with Alpha Alpha channel convention : (r, g, b, α) represents a pixel that is α covered by the color C=(r/α, g/α, b/α) Color components are premultiplied by α Can display (r, g, b) values directly Closure in composition algebra What is the meaning of the following? (0, 1, 0, 1) = full green, full coverage (0, ½, 0, 1) = half green, full coverage (0, ½, 0, ½) = full green, half coverage (0, ½, 0, 0) = no coverage Semi-Transparent Objects Suppose we put A over B over background G A B G How much of B is blocked by A? 42

43 Semi-Transparent Objects Suppose we put A over B over background G A B G α A How much of B is blocked by A? How much of B is shows through A? Semi-Transparent Objects Suppose we put A over B over background G A B G α A How much of B is blocked by A? How much (1 α of B is shows through A? A ) How much of G shows through both A and B? 43

44 Semi-Transparent Objects Suppose we put A over B over background G A B G α A How much of B is blocked by A? How much (1 α of B is shows through A? A ) How much of G shows through both A and B? (1 α A ) (1 α B ) Opaque Objects How do we combine 2 partially covered pixels 3 possible colors (0, A, B) 4 regions (0, A, B, AB) A AB B A 0 B 44

45 Composition Algebra 12 reasonable combinations clear A B A over B B over A A in B B in A A out B B out A A atop B B atop A A xor B Example: C = A over B For colors that are not premultiplied : C = α A A + (1 α A ) α B B α = α A + (1 α A ) α B For colors that are premultiplied : C = A + (1 α A ) B α = α A + (1 α A ) α B A over B 45

46 Image Composition Example Jurassic Park Overview Image compositing Blue-screen mattes Alpha channel Porter-Duff compositing algebra Image morphing Specifying correspondences Warping Blending 46

47 Image Morphing Animate transition between two images Cross-Dissolving Blend image with over operator Alpha of bottom image is 1.0 Alpha of top image varies from 0.0 to 1.0 blend ( i, j) = ( 1 t) src( i, j) + t dst( i, j) ( 0 t 1) 47

48 Image Morphing Combines warping and cross-dissolving Image Morphing The warping step is the hard one Aim to align feature in images How specify mapping for the warp? 48

49 Feature Based Warping Beier & Neeley use pairs of lines to specify warp Given p in destination image, where is p in source image? Beier & Neeley SIGGRAPH 92 Warping with One Line Pair What happens to the F? Translation 49

50 Warping with One Line Pair What happens to the F? Scale Warping with One Line Pair What happens to the F? Rotation 50

51 Warping with One Line Pair What happens to the F? In general, similarity transformations Warping with Multiple Line Pairs Use weighted combination of points defined by each pair of corresponding lines 51

52 Warping with Multiple Line Pairs Use weighted combination of points defined by each pair of corresponding lines P is a weighted average Weighting Effect of Each Line Pair To weight the contribution of each line pair : weight [ i] = p ( length[ i] ) a + dist[ i] b Where length[i] is the length of the i-th line dist[i] is the distance from a point P to the i-th line a, b, p are constants that control the wrap 52

53 Warping Pseudocode WarpImage(Image, L [ ], L[ ]) Begin for each destination pixel p do psum = (0, 0) wsum = 0 for each line L[i] in destination do p [i] = p transformed by (L[i], L [i]) psum += p [i] * weight[i] wsum += weight[i] end p = psum / wsum Morphing Pseudocode GenerateAnimation(Image 0, L 0 [ ], Image 1, L 1 [ ]) Begin for each intermediate frame time t do for i=1 to number of line pairs do L[i] = line t-th of the way from L 0 [i] to L 1 [i] end Warp 0 = WarpImage(Image 0, L 0, L) Warp 1 = WarpImage(Image 1, L 1, L) for each pixel p in FinalImage do Result(p) = (1-t)*Warp 0 + t*warp 1 end 53

54 Beier & Neeley Example Image 0 Image 1 Warp 0 Warp 1 Result Beier & Neeley Example Image 0 Image 1 Warp 0 Warp 1 Result 54

55 Image Processing Quantization Uniform quantization Random dither Ordered dither Floyd-Steinberg dither Pixel operations Add random noise Add luminance Add contrast Add saturation Filtering Blur Detect edge Warping Scale Rotate Warps Combining Morphs Composite 55