Unwarping paper: A differential geometric approach

Transcription

1 Unwarping paper: A differential geometric approach Nail Gumerov, Ali Zandifar, Ramani Duraiswami and Larry S. Davis March 28, 2003

2 Outline Motivation Statement of the Problem Previous Works Definition of Applicable Surface Full Integrability of the PDE s Forward Problem Inverse Problem Future Works References 2

3 Motivation (I) To develop a Seeing-Eye video-based interface for the visually impaired to access environmental information which leads to a way of independence. We are concerned about those daily activities which involved with interpreting Environmental texts or Scene Texts There are vast types of texts in outdoor and indoor scene located not necessarily on flat surfaces Off-the-shelf OCR fails to interpret them!! 3

4 Motivation (II) Mann-Brook Brook-Fogarty[ Fogarty[ 99] Table Various Scene text 4

5 Motivation (III) 5

6 Statement of problem 3D structure understanding and unwarping scene texts from single views for OCR purposes 6

7 Previous work: Understanding 3D structure (I) Multiple or Single views Multiple views Stereopsis Structure from Motion; optical flow or epipolar constraints Poleman-Kanade[ Kanade[ 97], Dellaert-Seitz[ Seitz[ 00], 00], Challenges: good reliable features to track, correspondence problem and occlusions 7

8 Previous works: Understanding 3D structure (II)- Single views Shape from Shading Oliensis[ 91], Forsyth-Ponce[ Ponce[ 00], 00], Light source model and reflection model in outdoor scenes Shape from Structured Light Wang-Mitchie[ Mitchie[ 87 ], Not useful in outdoor scene texts and sensitive to other light sources in indoor scene Shape from Occluding contours Koenderink[ 84][ 84][ 90] No exact solution but gives qualitatively good results Shape from Texture Rosenholtz-Malik[ Malik[ 97], Aloimonos[ 88], Riberio[ 89], Garding[ 92] Texture identification and modeling of scene texts and occlusions 3D Structure from surface properties Differential geometric properties of a surface is sufficient for the exact 3D structure solution 8

9 Previous works: Unwarping Interpolation Rigid mapping Wolberg[ 90] Deformable mapping and elastic matching Terzopolus[ 96], Blongie[ 02], Need two images, features and correspondence throughout whole patch Reconstruction Unwarping using 3D range Data Pilu[ 01], Brown[ 01], 01], Metric rectification Zissermann[ 98][ 98][ 00], Pilu[ 01], 01], Assumption: Planarity of an object Unwarping using surface properties Only boundaries of a patch in image and parametric plane 9

10 Parametric surface representation Consider a smooth and parametric surface S in 3D: r= ruv (, ) = ( Xuv (, ), Yuv (, ), Zuv (, )) Assume 1 st and 2 nd nd derivatives of r exist w.r.t u and v: ru, rv, ruu, ruv, rvv 10

11 Definitions First Fundamental form of a surface: E = r, F = r r, G = r Second Fundamental form of a surface: L= ru. nu = ruu n, M = ru. nv= ruv n N = r. n = r n v v vv where n is normal vector of a surface: n = 2 2 u u v v ru rv r r u v 11

12 Applicable Surface (I) Isometric ( preserved length) and conformal (preserved angle) with flat surface E = r = F = r r = G = r = Zero Gaussian Curvature: At least in one principal direction the curvature is zero; e.g. cylindrical surface K 2 2 u 1, u v 0, v 1 LN M = k k = = EG F

13 Applicable Surface (II) We can show from derivatives of isometric and conformal properties and zero Gaussian curvature that: n= ar = br = cr b 2 uu uv vv ac= 0 13

14 Inverse Problem 14

15 Theorema Egregium (Gauss) If surface S is obtained from surface S 0 by smooth bending (without stretching), at which the first quadratic form (E,F,G) is preserved, the Gauss curvature of the surface S is the same as the Gauss curvature of S 0. 15

16 Equations: Basic PDE s These are high order nonlinear PDEs Conventional Methods: Finite Difference, Spectral Methods, 16

17 Full Integrability of the PDE s Reformulation: We found a general analytical solution and showed Full integrability of these equations. Jacobian of transform (u,v) (W u,w v ) is zero! Thus, 17

18 Full Integrability of the PDE s Further, we have degenerativity of the following Jacobians: This shows that the mapping function, t=t(u,v), is universal for all coordinates: 18

19 Full Integrability of the PDE s n r v r u z y x 19

20 Full Integrability of the PDE s Implicit definition of t(u,v). 20

21 Relation to Wave (Hyperbolic) Equations 21

22 Linear Wave Equation v Characteristic plane t u = 3 u = 2 u = 1 u = 0 Perturbations propagate v 0-3c 0 v = v 0 -c 0 u v 0-2c 0 v 0 -c 0 v 0 v v + c 0 u = const u 22

23 (Riemann-Hopf equation): Wave Equation t v v = v 0 -c(t)u v We have: breaking wave v 0 form of Riemann-Hopf equation v + c 0 u = const 23

24 Characteristic Plane B C Characteristic Plane = Undistorted Flat Plane Along characteristics: t=const v A D u 24

25 Full Integral for Applicable Surface Other equations If functions are found, the entire solution is known 25

26 Forward (Warping) Problem Given a 3D curve, specified by equation Given a 2D curve in the (u,v)-plane, specified by equations Determine applicable surface, such that 26

27 Forward (Warping) Problem Z v Γ A Ω A Γ X Y Ω u 27

28 Forward (Warping) Problem Solution: 1). Use natural parametrization Length of the Curves is the same! 2). Solve ODEs (analytical solution is also available) 28

29 Forward (Warping) Problem 29

30 Inverse Problem 30

31 Inverse Problem Given a closed 2D curve in the image plane, specified by equations Given a 2D curve in the (u,v)-plane, specified by equations Determine applicable surface, such that Here F x and F y are given camera equations. Also a few correspondence points are known. 31

32 Inverse Problem B C B A C y v A D x D u 32

33 Inverse Problem t = t 2 B Γ 13 Γ 23 Patch Γ 2 Γ 1 t = t 1 C Γ 13 B Γ 12 Γ 23 Γ 12 Γ 22 Γ 11 A C v Γ 11 u A Γ 21 D y x Γ 21 D Γ 22 33

34 Inverse Problem 1). Guess initial break point (for rectangular page this is one of the corners 2). Specify initial conditions (currently we need 2 free parameters) 3). Solve system of ODE s with these initial conditions 4). Find the difference of solution and data in the correspondence points: try to find proper initial conditions. 5). If impossible change guess 1). 6). Use other available information for determination of initial conditions by minimization the error function. 34

35 Current state: Inverse Problem 1). ODE solver works. Tests are performed. 2). For synthetic initial data obtained from the forward problem produces acceptable results. Problems (working on): Number of necessary feature points and initial conditions. Algorithm stability. Accuracy of the solver near the corner points. Feature point matching technique. Incorporation of apriori information to minimize the error 35

36 Ambiguities Method relies on an image of the boundary Some deformations can lead to same images of boundary Consider a rectangular sheet of paper Both deformations lead to same image in cross-section Deformation can only be determined by inside data More generally, we need to derive necessary and sufficient conditions for the applicable surfaces to identify non-uniqueness of the solution. 36

37 Z v X Y u + 3D Curve Warping B C + B 2D Patch Warping Image Acquisition v A u D A y D x C Boundary Detection 2D Patch Unwarping 37

38 Sensitivity and Error Analysis Choice of boundary values Error in Boundary extraction Error in unwarping problem Error in Model (Camera or Surface property) Future Works (I) 38

39 Future Works (II) Boundary Extraction Problem 39

40 Future Works (III) Shape from Texture is an alternative to understanding 3D structure and unwarping scene texts using differential geometric Properties Challenges: Texture identification and modeling for a font style Hypothesized slant and tilt as the gradient of texture deformation Shape from texture problem 40

41 Conclusions Developed a method to determine structure from differential geometry constraints and some image information Correspondence free The solution is new General approach may also be promising for other types of surfaces Subject of research 41

42 References 1. N. Gumerov, A. Zandifar, R. Duraiswami and L.S. Davis, Structure of Applicable Surfaces from Single Views, sent to ICCV J.J. Koenderink, Solid Shape, MIT press M. Do Carmo, Differential Geometry of Curves and Surfaces, Prentice Hall

43 THANK YOU! 43