Designing a Self-Calibrating Pipeline for Projection Mapping Application Kevin Wright Kevin Moule 2
Who we are Kevin Wright Director of the Application Software group at Christie responsible for developing video control and display calibration software Kevin Moule Focused on developing machine vision based system for automatically warping and blending multi-projector displays. 3
Christie at a Glance 4
Christie at a Glance Cinema Projection Business Displays Visualization Simulation 5
Projection Mapping Projection Mapping also known as video mapping, spatial augmented reality or shader lamps is a projection technology used to turn objects, often irregularly shaped, into a display surface for video projection 6
Projection Mapping Today - Staged Events 7
The Potential for Projection Mapping 8 Optional
Christie s C7 Automotive Kiosk 9
Manual Projection Mapping Workflow 10
How can we bring Projection Mapping to the Masses? Integration Barriers Design constraints Integration expertise Maintenance requirements Alignment Changing models 11
Automatic Projection Mapping Workflow 12
Christie s Digital Sandbox 13
Automatic Projection Mapping Starting with a known 3D object Using structured light from the projectors plus a number of machine vision cameras Derive all the parameters of the 3D physical world Projector/camera/model pose Projector lens parameters 14
Automatic Projection Mapping Replicate the physical world in the virtual world Render images from projector viewpoints Content creation happens in the virtual world, the system maps content to projectors 15
Automatic Projection Mapping Without the proper geometry, the images do not line up The content looks correct only through the combination of geometry and projection 16
Automatic Projection Mapping Projector locations need to be accurately evaluated Being off by a 1cm leads to a significant misalignment 17
Automatic Projection Mapping Content creation happens in the virtual world, the system maps content to projectors The content looks correct only through the combination of geometry and projection 18
Issues With 3D Workflow Viewport Limitations Not all visual pipelines provide a full 6 degrees of freedom or the lensing parameters to properly correlate viewport with projector location Pre-rendered content also forces a fixed viewport 19
Issues With 3D Workflow Perspective Correction When creating the illusion of depth (as in through the car window) the viewport must be drawn from the perspective of the viewer rather than the projector 20
Issues With 3D Workflow Perspective Correction Something as simple as specular highlights fall into this category A single light source may appear as many on a shiny object when using multiple projector viewpoints. 21
Issues With 3D Workflow Perspective Correction Something as simple as specular highlights fall into this category A single light source may appear as many on a shiny object when using multiple projector viewpoints. 22
Issues With Workflow Projecting Depth Geometry required for good projection can often differ from the actual geometry Recreating the illusion of the actual geometry requires an eye point that matches the viewer s position 23
Improvements to Workflow - Warping Image warping may be applied to correct for the difference between eyepoint and projector viewports 24
Improvements to Workflow - Warping 25
Improvements to Workflow - Warping NVIDIA Warp + Intensity API enables the warping and blending The API is a low level warping engine that needs additional software support Ideal for us because we can layer our existing projector based warping tools (Twist/AutoCal) on top Ideal for the application because the warping support is baked into the driver, no need for software changes 26
Improvements to Workflow Rendering Alternatives A two-pass rendering process can capture an eye-point correct image for use from the perspective of the projector Start with an image rendered from the desired view 27
Improvements to Workflow Rendering Alternatives Naively generating images from the projector locations lead to incorrect and inconsistent results 28
Improvements to Workflow Rendering Alternatives Use the desired viewpoint image as a texture when generating image from the projectors 29
Future work Formalizing workflow, enabling broader use Improve automation of warping and blending Investigate other rendering techniques and platforms Increase creative control 30