DEPTH, STEREO AND FOCUS WITH LIGHT-FIELD CAMERAS

Size: px

Start display at page:

Download "DEPTH, STEREO AND FOCUS WITH LIGHT-FIELD CAMERAS"

Kevin McKinney
5 years ago
Views:

1 DEPTH, STEREO AND FOCUS WITH LIGHT-FIELD CAMERAS CINEC 2014 Frederik Zilly, Head of Group Computational Imaging & Algorithms Moving Picture Technologies Department, Fraunhofer IIS Fraunhofer, Frederik Zilly

2 CONTENT Introduction Lightfield Acquisition Concept of Lightfields Lightfield Capturing with Microlenses Lightfield Capturing with Camera Arrays Lightfield Processing Visual Effects Results Data Processing for Cluster Eye cameras Conclusion Outlook Fraunhofer 2

3 Introduction What is driving us? The transition from analog to digital movie production created a huge potential for image manipulations Special effects Corrections of image flaws Still lots of manual work. Furthermore, huge effort on set. Todays computers and embedded systems provide lots of computation power and storage capacities Research question How can image processing algorithms improve image quality? How can image processing simplify video capturing and editing? What needs to be changed in video acquisition such that image processing manipulations can be performed more easily?

Moving Picture Technologies Research Topics Computational Imaging Frederik Zilly Systems and Devices Wolfgang

Imaging Lightfield Technologies, Rendering Hardware based Image Compr. (Low complex, J2K,.

Topics Post Production & Distribution Technologies and Tools Archive Package Creation / Conforming

4 Moving Picture Technologies Research Topics Computational Imaging Frederik Zilly Systems and Devices Wolfgang Heppner Digital Cinema Heiko Sparenberg New Acquisition Workflows Image Data Compression Algorithms HDR Imaging Lightfield Technologies, Rendering Hardware based Image Compr. (Low complex, J2K,..) HighSpeed Interfaces (10GE, 3G SDI) Multicamera and Cameraarray Setups HighEnd Recording Technology Today s Topics Post Production & Distribution Technologies and Tools Archive Package Creation / Conforming Technologies Image / Audio / Metadata- Transcoding Technologies Digital Cinema Technologies Parallelisation Technologies on GPUs and Multicore-CPUs

5 Computational Imaging Lightfield Technology Development of Lightfield acquisition workflows and devices based on Plenoptic cameras Camera arrays Cluster Eye cameras Design of algorithms for refocusing / synthetic aperture depth estimation view rendering

6 Concept of Lightfields Repositioning the Camera Virtual Camera Traditional Camera Lightfield Camera

7 Concept of Lightfields Depth Of Field Traditional cameras: Lenses capture rays and focus them into one direction Parameters: Aperture size Focus Goal: Capture not only pixel value, but also direction of light Then we can simulate lens operations by computers

8 Lightfield Acquisition Plenoptic camera using micro lenses Microlens array in front of image sensor allows multiple optical imaging of same scene Prototype camera: Sensor MLA Main Lens

9 Lightfield Acquisition Plenoptic camera using micro lenses

10 Lightfield Acquisition Refocus Demonstrator Use of a multicore computer for computation and display

11 Lightfield using Micro Lenses Lessons Learned Lightfield capture permits to better manipulate images after acquisition in post producion Several challenges to overcome (assembling of microlenses and sensor, calibration, real-time computation) Achievement of a high quality image over a wide range of settings requires dense sampling or sophisticated reconstruction algorithms High resolution video is a challenge because of the redundant scene capture Strategy: Use camera arrays instead or in combination

12 Lightfield Acquisition Using Camera Arrays Motivation Every individual camera is able to deliver high resolution Challenges Dense sampling required -> needs lots of cameras Cameras need to be small Limitations in optics, sensor sizes,... Still a need for high quality Handling the huge amount of data Obtain a reasonable cabling effort

View rendering Visual Effects Camera array used for capturing

13 Lightfield Acquisition Using Camera Arrays Originally Sparse Lightfield Acquisition Rectification Disparity estimation View rendering Visual Effects Camera array used for capturing Apply multi-camera image processing to create a dense lightfield

14 Lightfield Processing Rectification Corresponding Image Points shall be in the same line or column

15 Lightfield Processing Multi-focal Disparity Estimation Dense Disparity maps are estimated

16 Lightfield Processing Dense Lightfield Generation Intermediate Views are generated using Depth Image Based Rendering

17 Lightfield Processing Visual Effects Virtually Reposition the camera Rendering in Z-Direction To be nearer at the scene (e.g. football stadium) To be more far away (e.g. to simulate helicopter flight) Create Vertigo-Effect / Dolly-Zoom Create Matrix-Effect / Camera path in freeze frame Create Stereo Pairs with configurable inter-axial distance Create Content for Auto-Stereoscopic Displays Reposition the Depth-of-Field Change position Change width of the DOF

18 Lightfield Processing View Rendering Depth Image Based Rendering Forward Warping of Depth Map: Post-Processing the result using morphological filters Backwarp Warping the RGB images

19 Visual Effects Rendering in Z-Direction Camera at original position

20 Visual Effects Rendering in Z-Direction Camera moved forwards

21 Visual Effects Rendering in Z-Direction

22 Visual Effects Vertigo / Dolly-Zoom Background has normal magnification factor

23 Visual Effects Vertigo / Dolly-Zoom Background appears magnified

24 Visual Effects Vertigo / Dolly-Zoom

25 Visual Effects Synthetic Aperture / Focus Depth-of-Field in the background

26 Visual Effects Synthetic Aperture / Focus Depth-of-Field in the foreground

27 Visual Effects Synthetic Aperture / Focus

28 Visual Effects Inter-axial distance Small Baseline

29 Visual Effects Inter-axial distance Large Baseline

30 Visual Effects Inter-axial distance

Stop-Motion-Video Capture with Lightfields Camera Arrays remains

31 Content Acquisition with 16 cameras Plot Scene setup Camera setup Shooting More Parallax with larger array Create Workflow for Stop-Motion-Video Capture with Lightfields Camera Arrays remains untouched, all effects in Post-Production Larger Camera Arrays for Live-Action Scenes Postproduction

32 Lightfield Stop Motion Capture Stop Motion Demo Clip shot with 16 Cameras

33 Lightfield Stop Motion Capture Stop Motion Demo Clip shot with 16 Cameras

34 Lightfield Stop Motion Capture Stop Motion Demo Clip shot with 16 Cameras

35 Lightfield Stop Motion Video

36 Lightfield in Post Production One Depth-Map generated per camera view and video frame

37 Lightfield in Post Production Viewing Perspective and Focus can be chosen in Post-Production

38 Demo-Clip Viewing Perspective and Focus can be chosen in Post-Production

39 Conclusion Sparse Lightfield Acquisition with Camera Array works The higher the number of cameras the higher the image quality the higher is the required processing power The larger the camera array the larger is the creative leeway to reposition the virtual camera Huge amount of data needs clever workflows and data handling Next steps: Further improvements of the image quality Increase processing speed

40 Thank you for your attention! Have a nice day! Frederik Zilly, Fraunhofer IIS, Erlangen frederik.zilly@iis.fraunhofer.de

41 Data Processing for the Cluster Eye Camera Arrayƒofƒminiaturized ƒcamerasƒusingƒa ƒmicrolens ƒarrayƒonƒimageƒsensor ƒ(cmos) Eachƒmicrolensƒimages ƒdifferent ƒpartƒofƒfieldƒof ƒview

42 All-In-Focus-Rendering for Cluster Eye camera Example of an All-in-focus Rendering using Images from the Cluster Eye

43 Generation of Depth Maps Depth Map generated from images from the Cluster Eye

Computational Photography: Real Time Plenoptic Rendering

Computational Photography: Real Time Plenoptic Rendering Andrew Lumsdaine, Georgi Chunev Indiana University Todor Georgiev Adobe Systems Who was at the Keynote Yesterday? 2 Overview Plenoptic cameras Rendering