Supplementary Material: Geometric Calibration of Micro-Lens-Based Light-Field Cameras using Line Features

Size: px

Start display at page:

Download "Supplementary Material: Geometric Calibration of Micro-Lens-Based Light-Field Cameras using Line Features"

Estella Watts
5 years ago
Views:

1 Supplementary Material: Geometri Calibration of Miro-Lens-Based Light-Field Cameras using Line Features Yunsu Bok, Hae-Gon Jeon and In So Kweon KAIST, Korea As the supplementary material, we provide detailed derivation of the projetion model in Set. 3 and the linear form in Set. 4. We also present the whole proess of generating sub-aperture images using our alibration result. Projetion model of miro-lens-based light-field ameras Miro-lens-based light-field ameras ontain two layers of lenses; main lens and miro-lens array. We apply thin lens model to main lens and pinhole model to miro-lenses, similar to. X, Y, Z Projeted Point Projeted Center Miro Lens Center Image Path of Ray Point X, Y, Z L l x, x, y y CCD Array Miro Lens Array Main Lens F Fig.. Projetion model of miro-lens-based light-field ameras. Projeted loation x, y) of the image in Fig. is omputed by extending line onneting the image and a miro-lens enter as follows: L + l) x y X Y + L + l Z L x y X Y Z Z L + l Z ) X Y + L + l Z Z l X Y Z + L + l Z L x y. L x y )

2 2 Y. Bok, H.-G. Jeon and I. S. Kweon The third term of ) is simplified as l Z+ L + l Z L lz + L2 + Ll LZ )L + l) L+l. 2) The first and seond terms of ) divided by L + l) are x l X LL + l Z) x +. 3) y )L + l) Y )L + l) y Equation 3) is simplified by subtrating projeted miro-lens enter x, y ) from it: ) x x l X LL + l Z) + y y )L + l) Y )L + l) x y l X LL + l Z) )L + l) x + )L + l) Y )L + l) y l X + L2 + Ll LZ) L 2 + Ll LZ lz) x 4) )L + l) Y )L + l) y l X lz x + )L + l) Y )L + l) y ) l X x + Z. )L + l) Y y Being substituted by X Y F Z Z F X Y, 5) Z whih is omputed using the thin-lens model, 4) beomes x x y y l L F Z F Z )L + l) lf LZ F ) F Z )L + l) F Z F X X Y Y X + F + Z x y Z F Z ) ) LF L F )Z LF l x + Z LL + l) Y y /F /L)Z LL/l + ) ) X x + Z K 2 K Z ) Y, y x y ) X x + Z Y y ) 6) where K /F /L and K 2 LL/l + ).

3 Supplementary Material 3 2 Calibration of miro-lens-based light-field ameras As we have mentioned in the paper, we use a simple intrinsi model i.e., zero skew, single foal length and zero enter) to ompute an initial solution: u x f, 7) v y where f is the foal length of intrinsi parameters. Atually it indiates the number of pixels in one measurement unit millimeter in this paper). Adopting 7) to the projetion model 6), it beomes u u v v K 2 K Z ) f ) X u + Z Y. 8) v We apply 8) to line features extrated from raw images of a hekerboard pattern. Projetions of orners are lose enough to orresponding line features to use an approximation that they lie on the features. Let ax + by + 0 a 2 + b 2 ) be the equation of a line feature, setting miro-lens enter u, v ) as origin. Substituting orner projetions into the line equation, au u ) + bv v ) + 0 9) afx + Z u ) + bfy + Z v ) + K 2 K Z ) 0. 0) Let X w, Y w, Z w ) be one of two orners whih define a line segment in world oordinate system i.e., hekerboard oordinate system). It must be transformed into amera oordinate system by an unknown transformation matrix with a 3 3 rotation matrix R and a 3 translation vetor t: X Y Z R X w Y w Z w r X w + r 2 Y w + t + t r 2 X w + r 22 Y w + t 2, ) r 3 X w + r 32 Y w + t 3 where r ij and t i are elements of R and t at i-th row and j-th olumn. Without loss of generality, the z-oordinate of the hekerboard pattern is set to zero Z w 0 for all orners). Substituting X, Y, Z ) by ), 0) beomes afr X w + r 2 Y w + t ) + r 3 X w + r 32 Y w + t 3 )u ) + bfr 2 X w + r 22 Y w + t 2 ) + r 3 X w + r 32 Y w + t 3 )v ) + K 2 K r 3 X w + r 32 Y w + t 3 ) ) afr X w afr 2 Y w aft + au r 3 X w + au r 32 Y w + au t 3 bfr 2 X w bfr 22 Y w bft 2 + bv r 3 X w + bv r 32 Y w + bv t 3 2) + K K 2 r 3 X w + K K 2 r 32 Y w + K K 2 t 3 K 2 ax w fr af r 2 Y w a ft bfr 2 X w bfr 22 Y w bft 2 + au + bv ) r 3 X w + au + bv ) r 32 Y w + au + bv ) t 3 + X w K K 2 r 3 + Y w K K 2 r 32 + K 2 K t 3 ) 0,

4 4 Y. Bok, H.-G. Jeon and I. S. Kweon whih an be expressed in Ax 0 form: ax w ay w a bx w by w b au + bv )X w au + bv )Y w au + bv ) X w Y w fr fr 2 ft fr 2 fr 22 ft 2 r ) r 32 t 3 K K 2 r 3 K K 2 r 32 K 2 K t 3 ) 3 Generating sub-aperture images A sub-aperture image is a olletion of amera rays whih pass through a ommon point. Using the alibration result by the proposed method, we an ompute a amera ray orresponding to eah point in a raw image. The generalized version of the projetion model is desribed in the paper: ) u u fx X u + Z x v v K 2 K Z ) f y Y. 4) v y A amera ray orresponding to u, v) in a miro-lens image entered at u, v ) passes through a 3D point X, Y, Z ). Aording to 4), the 3D point is independent of miro-lens enter u, v ) if Z is equal to zero. It means that any ray orresponding to a pixel with same displaement d u, d v ) u u, v v ) from miro-lens enter passes through a ommon point X s, Y s, 0): Xs Y s Z 0 u u v v Z 0 fx X s K 2 f y Y s 5) u u )/f K x du /f 2 K x v v )/f 2, 6) y d v /f y so that X s, Y s, 0) beomes the enter of a sub-aperture image at d u, d v ). Diretion of the ray x d, y d, z d ) is also omputed using 4). Sine the ray passes both 3D points X, Y, Z ) and X + x d, Y + y d, Z + z d ), u u v v ) fx X u + Z x K 2 K Z ) f y Y v y ) 7) fx X + x d ) u + Z f y Y + y d ) + z d ) x v y K 2 K Z + z d ) )

5 K 2 K Z + z d ) ) K 2 K Z ) fx X K K 2 z d f y Y fx X f y Y fx X + x d ) f y Y + y d ) Supplementary Material 5 u + Z x v y ) + Z + z d ) u + K K 2 z d Z x v y fx x K 2 K Z ) d f y y d + K 2 K Z )z d u x v y u x v y ) 8). 9) Dividing 9) by K 2 z d, fx x K Z ) d fx X K f y y z u d z x d f y Y d. 20) v y From 4), fx X u Z x u u f y Y K v 2 K Z ). 2) y v v Substituting f x X, f y Y ) by 2), 20) beomes fx x K Z ) d f y y d ) u K z d Z x u u u K v 2 K Z ) z x y v v d v y u u u K K 2 z d K Z ) + K v v Z )z x d v y 22) xd /z d u u )/f K y d /z K x u 2 + x )/f x. 23) d v v )/f y v y )/f y We set the size of sub-aperture images to pixels beause that of raw images is pixels and the average distane between neighboring miro-lens enters is around 0 pixels. The parameters in their intrinsi matrix K sub are set to /0 of the alibration result f x, x, y : f x /0 0 x /0 K sub 0 f x /0 y /0. 24) 0 0 For eah miro-lens image, we ompute the orrespondenes between pixels in a raw image and those in a sub-aperture image. Let us assume that we want to generate a sub-aperture image at d x, d y ). We ollet pixels with displaement of d x, d y ) from miro-lens enters, and ompute their orresponding rays using 23). We remove radial distortion from the rays using k and k 2 omputed in the alibration proess. Finally they are projeted onto sub-aperture image by multiplying K sub.

6 6 Y. Bok, H.-G. Jeon and I. S. Kweon Using Lytro amera, miro-lens enters are plaed in a triangular shape as shown in Fig. 2a). We onnet pixels with displaement of d x, d y ) from adjaent miro-lens enters to generate triangular meshes. They are transformed into a sub-aperture image see Fig. 2b)). For eah pixel of the sub-aperture image, we find a mesh whih ontains the pixel and ompute its intensity value by a triangle interpolation also known as baryentri interpolation). As shown in Fig. 2), let u, v ), u 2, v 2 ) and u 3, v 3 ) be the points in a sub-aperture image whih define a mesh, and y, y 2 and y 3 be their intensity value omputed by bilinear interpolation using raw image. The intensity value y orresponding to u, v) inside the mesh is omputed as follows: u 0 v 0 u v u v u 2 v 2 u 3 v 3 25) l 0 u u 0 ) 2 + v v 0 ) 2 26) l u u ) 2 + v v ) 2 27) l 2 u 0 u 2 ) 2 + v 0 v 2 ) 2 28) l 3 u 0 u 3 ) 2 + v 0 v 3 ) 2 29) y S y + S 2 y 2 + S 3 y 3 S + S 2 + S 3 l 0 l 0 + l y + l l 0 + l l3 y 2 + l ) 2 y 3, l 2 + l 3 l 2 + l 3 30) where u 0, v 0 ) is the intersetion of two lines: one onneting u, v ) and u, v), and the other onneting u 2, v 2 ) and u 3, v 3 ).

7 Supplementary Material u, v ) : y u, v) : y 68.5 S 3 S S u 2, v 2 ) : y 2 u 3, v 3 ) : y 3 a) b) ) Fig. 2. a) An example of triangular meshes blue line) generated by onneting pixels with displaement of 2, ) pixels from adjaent miro-lens enters red dots). b) The meshes are transformed into a sub-aperture image to ompute intensity value of eah pixel green dots) of the sub-aperture image. Grey lines indiate pixel boundaries of the sub-aperture image. ) Notation for triangle interpolation. The intensity value y at pixel u, v) is omputed by a weighted sum of three verties of a mesh whih ontains the pixel. Weights of the verties are proportional to the area of triangles at their opposite side.

8 8 Y. Bok, H.-G. Jeon and I. S. Kweon Referenes. Dansereau, D.G., Pizarro, O., Williams, S.B.: Deoding, alibration and retifiation for lenselet-based plenopti ameras. In: IEEE Conferene on Computer Vision and Pattern Reognition. pp )

Lecture 02 Image Formation

Lecture 02 Image Formation Institute of Informatis Institute of Neuroinformatis Leture 2 Image Formation Davide Saramuzza Outline of this leture Image Formation Other amera parameters Digital amera Perspetive amera model Lens distortion