Non-Differentiable Image Manifolds

Transcription

1 The Multiscale Structure of Non-Differentiable Image Manifolds Michael Wakin Electrical l Engineering i Colorado School of Mines Joint work with Richard Baraniuk, Hyeokho Choi, David Donoho

2 Models for Image Structure Not all N-pixel images are created equal Models capture concise structure few degrees of freedom permit effective denoising, compression, registration, detection, classification, segmentation, estimation,

3 Geometry: Where are the Images? concise models low-dimensional geometry

4 Linear Subspace Models e.g., 2D Fourier basis with bandlimited it d images

5 Many Image Families are Highly Nonlinear + =

6 Sparse Models: Unions of Subspaces e.g., wavelet bases with piecewise smooth images What more can we say about nonlinear signal families?

7 Manifold Models K-dimensional parameter captures degrees of freedom in signal f R N f Signal class F = {f : } forms a K-dimensional manifold also nonparametric collections: faces, handwritten digits, shape spaces, etc. Generally nonlinear Surprise: Often non-differentiable

8 Overview Motivating application: parameter estimation Non-differentiability from edge migration Parameter estimation (revisited) Non-differentiability from edge occlusion Manifolds in Compressive Sensing

9 Application: Parameter Estimation Given an observed image I = f, can we recover the underlying articulation parameters efficiently, and with high precision? (?,?) Given a noisy image I f, can we do the same? Relevant in pose estimation, image registration, computer vision, edge detection,

10 Optimization problem Newton s Method For a differentiable manifold, project onto tangent planes tangent

11 Newton s Method

12

13

14

15

16

17 Overview Motivating application: parameter estimation Non-differentiability from edge migration Parameter estimation (revisited) Non-differentiability from edge occlusion Manifolds in Compressive Sensing

18 Non-differentiability from Edge Migration Problem: movement of sharp edges example: shifted disk [Donoho,Grimes] Tangents do not exist Visualization: Local PCA experiment

19

20 Local PCA to approximate tangent space

21 span

22

23 Multiscale Tangent Structure Family of approximate tangent planes T(, ) scale, location on manifold If manifold F were differentiable: Does not happen when edges exist: Tangent spaces do not converge twisting into new dimensions But we can study and exploit this multiscale structure ~ wavelets for non-differentiable functions

24 Shortcut to Multiscale Structure via Regularization Smoothing the images smoothes the manifold more smoothing gives smoother manifold Example: convolution with Gaussian, width s Alternate family of multiscale tangent planes tangent planes well defined, analogous to PCA

25 Wavelet-like Characterization Family T(s, ) like continuous wavelet transform discretization: T(s i, j ) (i,j) (log) scale Fixed angle of twist between samples space Sampling is manifold-dependent

26 Overview Motivating application: parameter estimation Non-differentiability from edge migration Parameter estimation (revisited) Non-differentiability from edge occlusion Manifolds in Compressive Sensing

27 Recall Newton s Method

28 Recall Newton s Method

29 Multiscale Newton Algorithm Construct a coarse-to-fine sequence {F s } of manifolds that converge to F Take one Newton step at each scale

30

31

32

33

34

35

36

37

38

39 New Perspective on Multiscale Techniques Image Registration & Coarse-to-Fine Differential Estimation Irani/Peleg, Belhumeur/Hager, Keller/Averbach, Simoncelli & many others all suggested by the geometry of the manifold

40 Experiments: Translating Disk

41 s = 1/2

42 s = 1/4

43 s = 1/16

44 s = 1/256

45 MSE iteration

46 Experiments: Rotating 3-D Cube

47 s = 1/2

48 s = 1/4

49 s = 1/16

50 s = 1/256

51

52 Overview Motivating application: parameter estimation Non-differentiability from edge migration Parameter estimation (revisited) Non-differentiability from edge occlusion Manifolds in Compressive Sensing

53 Occlusion-based Non-differentiability Sudden appearance/disappearance of edges Tangent spaces changing g dimension different left, right tangents Occurs at every scale pitch, roll, yaw

54 Occlusion-based Non-differentiability

55 Occlusion-based Non-differentiability

56 Occlusion-based Non-differentiability

57 Occlusion-based Non-differentiability

58 Occlusion-based Non-differentiability

59 Occlusion-based Non-differentiability

60 Occlusion-based Non-differentiability

61 Occlusion-based Non-differentiability

62 Occlusion-based Non-differentiability

63 Occlusion-based Non-differentiability

64 Occlusion-based Non-differentiability

65 Five Relevant Tangent Vectors left right head-on view roll (1) pitch (2) yaw (2) Can explicitly consider such points in parameter estimation

66 Overview Motivating application: parameter estimation Non-differentiability from edge migration Parameter estimation (revisited) Non-differentiability from edge occlusion Manifolds in Compressive Sensing

67 Compressive Sensing Signal x is K-sparse in basis/dictionary Ψ Collect linear measurements y = Φx measurement operator Φ incoherent with elements from Ψ not adapted to signal x random Φ will work measurements signal or image [Candès, Romberg, Tao; Donoho]

68 Single Pixel CS Camera single photon detector random pattern on DMD array 4096 pixels 1600 measurements (40%) [with R. Baraniuk + Rice CS Team]

69 Why CS Works: Stable Embeddings

70 One Challenge: Non-Differentiability Many image manifolds are non-differentiable no embedding guarantee difficult to navigate Solution: multiscale random projections Noiselets [Coifman et al.]

71 Example: Ellipse Parameters original initial guess initial error N = 128x128 = K = 5 (major & minor axes; rotation; up & down) M = 6 per scale (30 total): M = 20 per scale (100 total): 57% success 99% success

72 Multi-Signal Recovery: Manifold Lifting 200 images N = 64 2 = 4096?

73 Final Reconstruction image-by-image reconstruction without using manifold structure joint reconstruction using manifold structure PSNR 15.4dB PSNR 23.8dB

74 Conclusions Image manifolds contain rich geometric structure Image appearance manifolds non-differentiable, due to sharp edges edge migration global non-differentiability wavelet-like multiscale structure accessible by regularizing each image edge occlusion local non-differentiability Can exploit multiscale structure in algorithms proxy for standard calculus new interpretation for image registration, etc. applications in Compressive Sensing