3D Camera for a Cellular Phone. Deborah Cohen & Dani Voitsechov Supervisor : Raja Giryes 2010/11

Transcription

1 3D Camera for a Cellular Phone Deborah Cohen & Dani Voitsechov Supervisor : Raja Giryes 2010/11 1

2 Contents Why 3D? Project definition and goals Projective model of a structured light system Algorithm (3 phases) System structure Implementation steps Advantages and drawbacks Summary Future work 2

3 Why 3D? Nowadays, more and more applications need real-time, accurate, low cost 3D scanning : Entertainment (video games, movies) Objects recognition Medical imaging Robot navigation systems 3

4 3D Techniques Some techniques that exist today : Time-of-flight 3D laser scanner Triangulation Passive stereometric scanners Active stereometric scanners Our technique : 4 Active stereometric Structured Light Timemultiplexed

5 Project definition and goals Main goal : Miniaturization of the camera for a cell phone Steps : implementation of a control system for the camera and the projector synchronization (between the camera and the projector) patterns projection and images capture off-line calibration reconstruction optimizations Objectives : real-time, accuracy, low cost 5

6 Projective model of a structured light system Assuming a pin-hole model for the camera and the projector Figure 1 Pin-hole model Camera : 2 coordinates - line Projector : 1 coordinate - plane Figure 2 Camera and projector planes vs. object plane 6

7 Projective model of a structured light system (cont.) Forward projection : Backward projection : T : X X, X w c p T 1 : X, X X c p w X X X C C c w c p p homogenous world coordinate system homogenous camera coordinate system homogenous projector coordinate system camera perspective projection matrix projector perspective projection matrix The matrices describe the intrinsic and extrinsic properties of the camera and the projector. 7

8 Projective model of a structured light system (cont.) Transformation of world coordinates to camera coordinates : X C X c c w where f kf x 0 x y c 0 c 0 y c c c C f y R t Transformation of world coordinates to projector coordinates : X C X p p w 0 f p 0 xp where Cp Rp t p

9 Algorithm Phase 1 Calibration Determining the PPMs: C c and C p Phase 2 Decoding Computing x p Phase 3 Reconstruction Finding the world coordinates x w 9

10 Input : Output : Algorithm - Decoding I fully illuminated image, I fully darkened image, I,..., I N coded images Stages : 1. Normalization : H L 1 N xp 2. Binarization : projector coordinate for each pixel B k J k x, y x, y k H, L,,, I x y I x y I x y I x y 1 Jk x, y threshold 0 else L 3. Decoding : N k xp 2 Bk x, y k1 Fixed point algorithm : Figure 4 Projected coded light patterns o look-up table for normalization o shift operation instead of division (power of 2) 10

11 Input : Algorithm - Reconstruction Output : Back projection : xw c xp, xc, yc projector and camera coordinates 1 R s xw world coordinates T x x, y, z non-homogenous world coordinates w w w w where Q ycc3 c2 R, s, p k-th row of C and C respectively k k c p xcc3 c 1 xp p2 p 1 11 Fixed point algorithm : o using homogeneous coordinates (fractions, scaling)

12 Scheme of object reconstruction Figure 5 Scheme of three-dimensional object reconstruction in a coded light scanner system 12

13 Algorithm - Calibration Input : Output : set of measured x, x and corresponding known 1 T calibration data c p x w N n1 N n1 Figure 3 Corner detection Minimize squared error between known fiducial points in World Coordinate System (WCS) and the backward projected measurements : Solving by numerical global optimization methods 13 N 2 1 arg min c, p w k k i1 2 T T x x x

14 BeagleBoard 3 x 3 POP (package on package) o CPU/Memory chip Processor TI OMAP3530 : o ARM cortex-a8 CPU (600 MHz) o TMS320C64x+ DSP (up to 430 MHz) o Imagination Technologies PowerVR SGX530 GPU Memory chip : o External : 128 MB LPDDR RAM memory & 256 MB NAND Flash memory o Internal : L1-112 KB (DSP),32 KB (ARM), L2-96 KB (DSP),256 KB (ARM) Peripheral connections o DVI-D, USB, SD/MMC, S-Video, Stereo audio Development o Boot from NAND memory, SD/MMC, USB o Using Angstrom Linux 14 Figure 9 BeagleBoard described

15 I/O devices Camera Logitech QuickCam Pro 9000 Projector TI DLP Pico projector Figure 10 Camera Figure 11 Projector 15

16 System Figure 6 System size compared to iphone Figure 7 Top view 16 Figure 8 Front view

17 Tasks division in the BeagleBoard ARM controls DSP & GPU controls the communication with the camera and the projector DSP runs fixed point reconstruction algorithm GPU runs fixed point reconstruction algorithm 17

18 Tasks division in the BeagleBoard (cont.) GPU uses the projector to illuminate the scene with 1-dimensional light patterns (9 patterns, Gray code) ARM uses the camera to capture the patterns Captured data is sent to the DSP DSP performs 3D reconstruction DSP returns the results to the ARM 18

19 Tasks division Figure 12 - BeagleBoard 19 Figure 13 Division of tasks between the ARM and the DSP

20 Step 1 - Patterns projection for Embedded Systems: premier environment for developing portable, interactive 2D and 3D graphics applications Draw triangle arrays (2 triangles per stripes): void gldrawarrays(glenum mode, GLint first, GLsizei count) 20

21 Step 1 - Patterns projection (cont.) 21 Figure 14 Coded light projected by the pico projector

22 Step 2 - Pictures capture SDK for UVC based cameras : video for Linux to UVC (v4l2uvc) Grab frame in yuyv format through the ARM : int uvcgrab (struct vdin *vd) Store the y coordinate (grayscale picture) 22

23 Checking - Image projection ES Figure 16 Projection of a captured picture Using interleaved vertex data to match the texture coordinates with the triangle coordinates Bind texture (picture) to 2 triangles which cover the whole display (projected picture) void glvertexattribpointer (Gluint index, GLint size, GLenum type, GLboolean normalized, GLsizei stride, const GLvoid * pointer) 23

24 Step 3 - ARM to DSP / DSP to ARM 1. Write pictures to DSP memory 2. Data available 3. Reconst. 4. Reconstructed data available 5. Read reconstructed picture from DSP memory 24 Figure 15 Communication flow between ARM and DSP

25 Step 3 (2) - ARM tasks Inputs : calibration matrices, 9 pictures 1 2 Write pictures to read buffer Transform calibration matrices to calibration parameters (written in message) 3 Send message 25

26 Step 3 (2) - DSP tasks Inputs : calibration parameters, 9 pictures 1 2 Perform decoding & reconstruction Write 3D homogeneous coordinates to write buffer (X,Y,Z,W) 3 Send message 26

27 Result Using surf function from Matlab 27

28 Time analysis TASK Patterns projection + pictures capture TOTAL TIME (sec) (*) EFFECTIVE TIME (sec) 5.78 (**) (*) Sync in software (sleep) (**) Without sleep time TASK ARM reconstruction DSP reconstruction TOTAL TIME (sec) RECONSTRUCTION TIME (sec)

29 29 Demo

30 Advantages and drawbacks Advantages Drawbacks Accuracy Real time (tasks division between the processors) Compatibility (Linux) Suitable only for scenes with low to medium motion Calibration relatively long (but performed once) Low cost Reflective or transparent surfaces raise difficulties Low power 30

31 Summary 3D Structured light Calibration Decoding Decoding & Reconstruction Reconstruction Calibration Embedded system Embedded system BeagleBoard Camera and projector Patterns projection Structured light 3D Camera for Cell Phone ARM, DSP, GPU Picture capture DSP ARM GPU Image projection 31

32 Future work Accelerate the projections and captures using devices with hardware synchronization Consider other real-time platforms Use cellular phone camera Optimizations 32

33 Thanks We want to thank the SIPL and GIP staffs, and particularly our supervisor Raja, and also Pavel and Alon who were always ready to help us even after dark. 33

34 34 QUESTIONS?