Course #25 Computer Graphics for Large Scale Immersive Theaters Immersive Rendering Basics and Aesthetics Brad Thompson Lead Animator Spitz, Inc. Bthompson@spitzinc.com
Introducing the Frame
This is not the frame Hemisphere Represented in Polar Coordinates
It s only an Abstracted Representation of it. Hemisphere Represented in Polar Coordinates
Example of Polar Format Frame Hemisphere Represented in Polar Coordinates
A Viewer is only Likely to see This much of the Image If you re sitting in a good seat, this is all you can see!
Unfortunately, this is how we see the frame as we work. Hemisphere Represented in Polar Coordinates
This View of the Canvas is Deceptive! Implies a false notion of a round frame in which to compose Implies a false sense of 2-dimensionality Enormousness of image is misrepresented on ANY flat monitor.
This Composition is ok But only for a severely tilted dome. For the majority of existing dome theaters, this view would not look good because the horizon is too high.
This Composition is Hard to read But much better for a slightly tilted dome because of it s gravity level horizon.
The dome is a 3-D 3 environment! The polar fisheye format is just a way of mapping a 3D environment with a 2D bitmap.
The picure is BIG! If you re looking over here you won t see here Bigger than your eye s FOV
The picure is BIG! An image this large will be undecipherable by a viewer
Polar Mapping Same Image, Different Mappings Cubic mapping Cylindrical Mapping
Most Common Mappings Equidistant cylindrical projection Polar projection
Most Common Mappings Equidistant cylindrical projection Polar projection
Most Common Mappings Placement of objects on dome is easily understood Polar distortion makes image comprehension challenging Polar projection Nonliniarity makes post animation Challenging
Most Common Mappings More difficult to understand placement of objects on dome Less distortion of the lower panorama, but more at the top Equidistant cylindrical projection Post animation is easier unless objects cross the zenith
Most Common Mappings Theater view Polar view Cylindrical panorama view
Most Common Mappings Conversion from one format to another is possible with minor degradation.
The Screen is not a window on the world! The Framless dome screen takes us through the window. Many of the rules of conventional cinema don t apply The dome necesitates it s own cinemagraphic language
Apparent motion increase Motion that looks ok on your monitor can be overwhelming in the theater. Can cause cybersickness
Apparent motion increase The apparent motion increase is around 4-5 times. Can cause cybersickness
Avoid Hard Cuts Real life is a continuous experience Cuts are artificial and have the potential to jerk the viewer out of the immersive experience If immersiveness is your goal, use Dissolves, crossfades, and dips to black instead Immersive environments demand longer continous shot lengths. A 60 second continous shot is not unusual.
Avoid Abrupt Changes Abrupt changes in view or environment can disorient the audience Because we can see the entire environment, nearly any edit/cut that doesn t take us somewhere else is percieved as a jump cut Create a stage upon which the characters act, then move the viewer through the space.
Size and detail requirements 4K frame = 16million pixels High resolution combined with huge FOV = more detail required Appropriate detail level is very difficult to judge on SD or even HD monitors.
Frame rate 30fps progressive scan is generally acceptable. 30fps interlaced scan gives smoother motion but is less flexible and requires rendering at 60 full fps anyway for full hemisphere formats. 60fps progressive scan may be desirable in some circumstances
Rendering Spherically (The Nitty Gritty)
Rendering spherically Most scanline type renderers can only render parallel or perspective type views. These rendering algorithms are highly optimized and efficient More general transformations can get more and more expensive as they are composed. This doesn t happen with 4x4 matrices Easier to create alternative camera projections within a raytracer.
Rendering spherically Flat plane projections carry polygons to polygons. Perspective Projection
Rendering spherically More general projections do not work this way. Polar (fisheye) projection
Two approaches to rendering Render directly to the desired format Render multiple perspective views, then stitch them together into the desired format
Approach 1: Rendering Directly to the Format Image quality is generally higher because no post warping is involved. Immediate feedback. No post processing required. Efficiency. Resulting renders are nearly projection ready.
Approach 1: Rendering Directly to the Format Smaller storage requirements since finished frames are returned no need to store unstitched partials. Textures and effects that rely on the camera normal usually work better with this method Integrated 3D post effects like glows or flares don t show seaming artifacts but may not align properly unless they are aware of the unique geometry
Approach 1: Rendering Directly to the Format Many rendering engines now support alternative camera projections directly FinalRender, VRAY, Brazil, RayMax, and VirtualLight have the best implementations I ve personally seen Mental Ray, BMRT, and Prman are capable, but less elegant POVray claims this ability.
Approach 2: The Hemicube Solution -5 cameras -90deg FOV -Post process required
Approach 2: The Hemicube Solution Universality. Any rendering engine that can render perspective projections can be used. Faster rendering speed, depending on scene/renderer. Same idea now implemented in Apple s QTVR 5+ as Cubic Rendering.
Approach 2: The Hemicube Solution Partials are not warped and can more easily be re-purposed for flat screens or quick review (in theory) Conceptually familiar to many 3D artists because it s similar to a cubic environment map.
Approach 2: The Hemicube Solution Hemicube solution seaming dangers: Texture effects that rely on camera normal Particles that rely on camera normal (facing particles)
Approach 2: The Hemicube Solution Hemicube solution seaming dangers: Post Glows based on an object channel Lensflares
Eyepoint Offset Geometry only looks absolutely correct from one position in the dome, typically dome center. In many applications, there are no seats at this position in the theater. An eyepoint offset function can be applied so that geometry looks correct from other seats. Can be implemented during 3D rendering or during the hemicube stitching process
Eyepoint Offset Focal convergence moved away from the projection surface
Eyepoint Offset Polar mapping Eyepoint shifted forward
2D Processing HD Video Film Scan Immersive Animation SD Video Still Photos
2D Processing Adjusting, compositing, adding to imagery that is already in a spherical format Reformatting standard flat plane imagery Preparing spherical material for projection 2D warping is faster than 3D equivalent
2D Processing Adjusting, compositing, adding to imagery that is already in a spherical format For many operations, any off the shelf, resolution independent compositing/fx package can be used. Uniform image adjustments like color or gamma correction, don t require software to be aware of unique format.
2D Processing Adding motion effects may require special handling Generally, geometry correction isn t needed if the effect is small in the frame and doesn t run into a high distortion zone. Effects that cover a large portion of the dome will require geometry correction Animation trajectories are effected by the unique format.
2D Processing 60 deg -100 deg 100 deg 0 deg Trajectory of an object moving in a visually straight line through an equidistant cylindrical mapping
2D Processing Processing required to reformat standard flat imagery to work on the dome.
2D Processing Reformatting standard imagery Billboard filter creates a virtual flat plane Plane can be positioned anywhere on dome Theoretical 180 degree limit to horizontal FOV
2D Processing Source video Billboard processed video
2D Processing Conglomerator tool Constantly evolving, multifunction, standalone, batch image warping tool written by Spitz, Inc. Mapping conversions, hemicube stitching, image cropping Simple GUI or command line execution for scripting.
Editing and finishing Most current immersive projection systems rely on multiple edge blended projectors. Master frames must be broken out into separate video streams for each projector These separate streams are then played back synchronously under the control of a theater automation system.
Editing and finishing Two approaches to editing Edit First method: Create a low-resolution edit from proxy clips, then conform your high resolution source to the proxy edit Split First method: Create a proxy edit, then conform your projector splits to this edit. This method is no longer viable if you plan to distribute content to more than 1 theater and doesn t apply to single lens type theaters.
Storage requirements Some common resolutions for Master frames are 8 or 10 bits per color channel at: 1800x486 Panoramic/partial dome 1536x1536 Single lens full dome QXGA 2200x2200 SD full dome projection 2800x2800 HD full dome edge blended 3600x3600 HD full dome edge blended 4000x4000 HD full dome edge blended
Storage requirements Worst case calculation: 4000x4000 = 16,000,000 pixels per image 10bits of color info per pixel = 480,000,000 bits per image or about 60 megabytes. at 60fps = 3,600 megabytes per second 20 minute show = 4,320,000 megabytes double storage for editing = over 8.5tb
Storage requirements Of course these requirements can be reduced by: using 8 bit per channel color depth using lossless compression on frames (RLE) dropping to 30 fps good planning and understanding of process.
Thinking toward the future Production tool advancements: Hope to see arbitrary/programmable camera projections available in more off the shelf 2D and 3D packages and rendering engines Real-time preview of alternate camera projections in 3D packages for more WYSIWYG and less render and pray.
Thinking toward the future Production tool advancements: More robust editing solutions that fit the needs of immersive video producers. Truly resolution independent digital editor capable of working in real-time, using low resolution proxies for offline conforming of either full resolution clips or projector sub-frames.
Thinking toward the future Better visualization tools: Not every animation house has a dome theater to view dailies. A QTVR-like dome movie previewing software would help those without a dome be able to visualize what their images will look like prior to projection.
Thinking toward the future Better visualization tools: A cheaper single lens projector/portable dome system that could be purchased or rented by a production facility for low cost viewing of dailies. (SciDome) Generally easier to use, less academic interfaces for our tools (getting much better now)
Thinking toward the future More invisible integration between software and theater hardware. Standardized/unified interfaces for all the tools from production through theater automation. Better hardware abstraction so that you don t need to know the rack to make a show.
Thinking toward the future Standardization of theater systems: This would allow easy content sharing between theaters. Easy, less expensive distribution for immersive producers. Theaters themselves should not NEED to do any production work. Opens a larger market for content providers.
References: Rendering: Hemispherical video production: http://astronomy.swin.edu.au/pbourke/projection/skyvision/ Fast Environment mapping: http://www.sgi.com/software/performer/brew/envmap.html Spherical Rendering info: http://www.cs.unc.edu/~zimmons/cs238/maps/environment.html A Realistic camera model for CG: http://graphics.stanford.edu/papers/camera/ Makers of RayMax and FinalRender: http://www.cebas.com Makers of Mental Ray renderer: http://www.mentalimages.com/ Makers of BMRT: www.exluna.com (no longer available) POV-Ray raytracer: http://www.povray.org/
References: Rendering: Brazil Rendering System: http://www.splutterfish.com VRAY Renderer: http://www.vrayrender.com Fisheye lens shader for renderman: http://terpsichore.stsci.edu/~summers/viz/fisheye/ Image based rendering of realistic lens systems: http://www9.informatik.unierlangen.de/eng/research/rendering/lensmodel/ Cubic environment mapping: http://www.iqtvra.org/defish/fisheye.html
References: Image processing: Panorama tools: http://www.fh-furtwangen.de/~dersch/ Realvis multi image stitching tools: http://www.realviz.com Guide to panorama stitching software: http://www.panoguide.com/ International QuicktimeVR assoc.: http://www.iqtvra.org/ Spherical imagery aquisition: Panoscan: http://www.panoscan.com Enroute: http://www.fpvideo.com SpheronVR: http://www.spheron.com/