Is 3-D TV preparing the way for holographic TV?

Transcription

1 Is 3-D TV preparing the way for holographic TV? V. Michael Bove, Jr., Quinn Y. J. Smithwick, James Barabas, and Daniel E. Smalley Object-Based Media Group, MIT Media Laboratory, Room E15-368B, 20 Ames St., Cambridge, MA USA ABSTRACT Holographic television, or holo-video, has been seen by many as the ultimate development not only of holography but of electronic visual communication generally. To make widespread, successful holo-video, four things are needed: 1) content, 2) a distribution mechanism, 3) sufficient processing at the receiver, and 4) suitable electro-optics at the display, and all of these must be available at prices suitable for consumers. In the past one to two years, there has been a great interest in 3-D television, but few researchers seem to have noted that many of the recent developments in 3-D TV are also solving or at least pointing the way to solving problems associated with holo-video. We examine particularly relevant developments in content capture/creation, content representation (including standardization activities), and the increased suitability of graphics processors for 3-D applications, and connect these with work at the MIT Media Laboratory in developing a holo-video display suitable for consumer use. Keywords: synthetic holography, 3-D display, holographic video, computer graphics, graphics processors 1. INTRODUCTION The technological history of 3-D television dates back more than sixty years (see, e.g., [1]) however despite several generations of attempts at commercialization, 3-D TV has so far failed to take hold as a widespread consumer product. Over the past several years, significant engineering, content-creation, standardization, and marketing effort has been focused on home 3-D TV, a situation which seems to have at least the following causes: the transition of broadcasting in many countries to digital and in many cases high-definition leaves manufacturers and content creators searching for the next form of premium television the current generation of consumers is not familiar with the long, often-tawdry, and largely unsuccessful history of 3-D TV and cinema display technology is undergoing rapid technological change, which means that displays are being replaced rapidly and a window of opportunity exists for including 3-D functionality new delivery mechanisms such as internet video and Blu-ray can potentially carry 3-D content and are more flexible and extensible than previous media lowered cost of various technologies such as shutter glasses, digital interconnects, and high-refresh-rate displays much current content such as CGI programs and video games is already suitable for 3-D display, while reduced size and cost of high-quality video cameras makes it easier to build stereoscopic camera rigs in the gaming and internet video domains, off-the-shelf graphics processing units (GPUs) and video decoders are now fast enough to handle stereo imagery Some notable recent events include the adoption by MPEG of the multiview video coding (MVC) extension to the H.264 advanced video coding (AVC) standard, the creation of the 3D@Home Consortium, the launching by the Society of Motion Picture and Television Engineers of a 3-D Home Entertainment Task Force, and the evaluation by the Bluray Disc Association of a 3-D extension proposed by Panasonic. An EU-funded consortium called MOBILE3DTV is developing standards and visual optimizations for transmitting stereoscopic video over the DVB-H system to mobile devices equipped with autostereoscopic screens.

2 All the efforts discussed in this section involve stereoscopic television meaning that they use only the depth cues of binocular parallax and convergence (though there have from time to time been experiments and even commercial trials of home television employing less-well-known psychophysical effects such as the Pulfrich Effect [2]). But they have relevance to holographic television whose ultimate goal is less about tricking the eye and more about reconstruction of light wavefronts identical to those that would have originated from a real scene as they not only ready the market for high quality dynamic 3-D imagery but also may allow easier construction of a successful holo-video system. The notion of holographic television still seems to have a powerful hold on the public s imagination, as demonstrated by recent excitement over technologies that are called holographic but don t use holographic principles, such as CNN s holographic interview system which is really just a 360-degree camera array combined with a video compositor. The fundamental question at the heart of designing a stereoscopic display system is, How do we get the correct view to the correct eye? while design of a transmission system also brings in questions of backward compatibility (Will viewers with 2-D systems see normal TV? Will already-deployed 2-D codecs, bitstream formats, and other legacy hardware and software be able to handle the 3-D content?) and efficiency (Can we take advantage of the significant redundancy between the stereo views?) With respect to making each eye see an appropriate parallax view, the fundamental approaches have been the same for many years (1950 reference [1] discusses and evaluates most of these principles, and Okoshi s 1976 book [3] explores them in depth), but changes in technology affect the relative practicality and cost of each. While it may not have been true even in the recent past, current electronic display systems generally contain sufficient memory and processing that the 3-D display process can be decoupled from the data representation (e.g. temporally-alternating transmitted views can be used on a display that requires them to be shown simultaneously on alternating columns of pixels). Well-known methods for controlling the views that each eye sees include: Color multiplexing: Monochrome views can be reproduced in complementary colors (commonly red/cyan) to create an anaglyph image and viewed through correspondingly-colored eyeglass lenses. Such a system does not, of course, permit color imagery, and can be fatiguing over long periods of time. Time multiplexing: If a video display has sufficiently high temporal refresh rate to avoid visible flicker, left and right views can be alternated in time and viewed through glasses with an active synchronous shutters. The cost of these glasses has dropped significantly in recent years but they are still somewhat bulky and require batteries. Liquid-crystal monitors are now available with refresh rates of 120Hz and higher, adding to the recent interest in such an approach. If the glasses can track the viewer s position and adjust the shutter phasing appropriately, there is no reason that a monitor with high enough refresh rate (LG has recently shown a 480Hz LCD panel) couldn t support more than two parallax views, though the author is not aware of any demonstrations of this sort. Polarization multiplexing: Projectors can be made to alternate their polarizations temporally, two orthogonally polarized projectors can be used simultaneously, or direct-view displays can be built such that columns of pixels have differing polarizations; in each case inexpensive passive glasses can be worn by viewers, though at the cost of increased display complexity. Angular multiplexing (autostereo): If the display itself has the ability to steer views, the viewer needn t wear special eyewear to perceive a stereoscopic image. Parallax barriers and 1-D or 2-D lenticular arrays are the usual mechanisms for performing this function, but holographic video displays (to be discussed in the next section) have this property as well. The parallax views also need to be represented in a suitable fashion for transmission and storage. In order of increasing complexity, approaches include: Simulcasting: Two or more parallel streams can be transmitted using standard codecs. Such an approach is compatible with existing standards and 2-D displays (which can simply pick out one stream), but it is inefficient as it takes no advantage of the statistical and perceptual redundancy between views. Interleaving in one stream: Views can be anamorphically squeezed into one image frame for an existing codec or can be alternated temporally (H.264 AVC supports a flag that identifies L/R and predicts L views and R views separately). This approach can be compatible with existing codecs but not with 2-D displays. Multiview coding: In a predictive coder (such as the multiview extension to H.264) frames for each view can be predicted not just from temporal neighbors but also from neighboring views. This technique is more effi-

3 cient than the preceding methods (though there are limits to how much redundancy such a coder can find), and a decoder for a 2-D display can simply pick out a single view. 2-D stream plus depth map: A very efficient and display-independent representation of 3-D images can be made by sending a single 2-D view and a depth map which is used to create differential parallax shifts of image regions at the display. The depth map might even be transmitted in MPEG private data, so that ordinary 2- D displays can ignore it. As might be imagined, such a representation has the disadvantage that occlusions cause missing regions that must somehow be synthesized [4], and the complications of computing the depth map during image capture and resynthesizing the parallax views at the display. Full 3-D representation: Most display-independent of all is transmitting a full 3-D texture-mapped model of the scene, from which a GPU at the display can synthesize parallax views appropriate for the display. There isn t currently a broadly adopted standard for transmission of such content in real time though on-line video games have demonstrated the possibility of doing so and creating 3-D models for real scenes continues to be impractical so this approach is suitable mostly for synthetic imagery. 2. RELEVANCE TO HOLO-VIDEO Many research groups are currently exploring the creation of dynamic 3-D electronic displays based on diffraction; the Object-Based Media Group at the MIT Media Laboratory has added the explicit requirement of making a display that is suitable for consumers in the relatively near term. In particular, this means that the display must be compact and rugged, manufacturable for a few hundred dollars, and able to be driven by standard hardware (i.e. the graphics processor in a PC or video game system) over standard video signal interfaces. As a consequence, we need to bear in mind developments and trajectories in related consumer technologies as we hope to make use of as many of these as possible. A consumer holo-video system (as indeed any consumer video system) requires for success the availability of four key elements: content a distribution mechanism sufficient processing at the receiver suitable electro-optics at the display To the degree that industry is already creating some of these, it makes the development of an end-to-end holo-video system easier as it removes the need for the researcher to build all parts of the chain. Recall that a holo-video display system, because of its ability to synthesize lightfields, is able to reproduce imagery captured for a two-or-more-view stereoscopic display, or for an integral display. Thus there is certain to be imagery in distribution that can be shown on a holo-video display, though even better imagery is available from content such as games that is already distributed as 3- D models assuming that a receiver can convert parallax views or 3-D models into diffraction patterns in real time. An example of real-time conversion of integral images into a holographic lightfield is given in reference [5]. Cinematographic training and technical literature are also beginning to reemphazise the craft of 3-D cinematic storytelling (see for instance [6]), and the scene composition and editing lessons apply equally well to holographic video. To be a bit more specific about the holographic display of stereo video sources, since a holographic video display allows very fine directional control of light, it is quite simple to cause a left-eye view and a right-eye view to come from the display at symmetrical angles on each side of the optical center of the screen. To see the images, a viewer would have to be directly centered on the screen and at a specific distance. But it s also possible to cause the views to come out at a range of directions on each side, which relaxes the distance restriction. It is further possible to support multiple viewers (or to give a single viewer the option to be elsewhere than centered on the screen) by repeating ranges of directions L/R/L/R/L/R, et cetera. However in that case half the viewing positions cause the viewer to see a pseudoscopic image, and it would perhaps be better to have a blank image at a narrow range of directions after each L/R pair, i.e. L/R/blank/L/R/blank/L/R, such that for a reasonable range of viewing distances if a viewer can see a view in each eye the result will be correct stereo (here presumably the viewer will move his or her head slightly to such a position). See Fig. 1.

4 Our research has for many years taken the viewpoint that end-to-end transmission of diffraction patterns for holo-video is neither efficient nor necessary if they can be generated at the receiver. While our desire to do so led us to develop specialized and often architecturally unusual computational systems, [7] more recently we have instead concentrated on tracking developments in GPU hardware and corresponding software pipelines and trying to achieve sufficient performance and quality from off-the-shelf infrastructure. Thus our target is a holo-video monitor that can plug into a standard PC or game machine using standard electrical interfaces and yet provide holographic images. Fig. 1. Viewing a stereo pair on a holo-video display: (Top) showing each parallax view at a single angle requires the viewer to be at the center of the screen and at a specific distance. (Center) showing each view at a range of angles loosens the distance restriction. (Bottom) repeating the view pairs supports multiple horizontal positions or multiple viewers; gaps prevent pseudoscopic images. 3. MORE SPECIFICS ABOUT OUR RESEARCH The Mark III display [8] builds on lessons from previous generations of MIT Media Laboratory holo-video displays. Like them it is horizontal-parallax-only and based on the Scophony electro-optical architecture, however it uses a different sort of light modulator (a very-high-bandwidth 2-axis lithium niobate guided-wave device), and a simpler and cheaper optical architecture that eliminates the horizontal scanning mirrors and some other components, and permits the optical path to be folded to fit into a CRT-monitor-like box. The initial version of this display is for proof-of-concept purposes and has a small monochrome screen whose specifications are 440 scan lines, 30 Hz

5 24 view angle 80 mm 60 mm 80 mm (W H D) view volume Some of these numerical specs are limited by our requirement that this display be drivable by a single standard dualhead GPU over its monitor interfaces. Likewise the HPO nature of the display is a consequence of the desire to compute the diffraction patterns on-the-fly and also the frame buffer size limitations of the GPU. Reference [9] provides a detailed description of our rendering method and how we map it onto the GPU hardware and software pipelines, but we shall summarize the ideas here. It is currently impractical to compute dynamic holographic imagery for scenes of any significant visual complexity using physically-based algorithms such as interferometric point-cloud methods. Instead we aim to construct lightfields by a diffraction-specific method, superposing precomputed basis diffraction fringes modulated by views of the scene rendered from a 3-D model. This approach handles complex surface textures, occlusions, and transparent objects correctly, and (more relevant to the focus of this paper) provides a point in the process into which stereoscopic real-world images captured by multiple cameras can be placed (instead of images rendered from 3-D models). The large amounts of precomputation and table lookup make this an inherently fast approach, but we have put significant effort over the past five years into mapping the algorithmic steps onto the operations that GPUs do well. Current GPUs are approaching teraflop performance, but can deliver that amount of computation only if the algorithm can be cast into a form that makes use of the fact that most GPUs are vector processors for four-component (nominally RGBA) vectors, and have certain functions that are optimized for certain vector data types. It is also essential to consider how to maximize parallelism and to keep all parts of the hardware pipeline busy as much as possible. Formerly the hardware pipeline and the corresponding software interface involved a largely fixed set of functions connected in a fixed way. GPU hardware and software now use programmable shaders whose mathematical transformations or texture and lighting calculations can be changed. As more programmers use GPUs to handle a variety of stream calculations even ones that have no connection to graphics this flexibility should continue to increase. Our method of diffraction-specific computation treats the holographic stereogram as a summation of overlapping amplitude modulated chirped gratings. This superposition of gratings on the hologram plane produces a set of viewdirectional emitters on an emitter plane. Each chirp focuses light to create a point emitter, while the angle-dependent brightnesses of the views are encoded in the amplitude modulation (Fig. 2). We perform three main steps: 1) precomputing the basis fringe chirp vectors into a chirp texture, 2) multi-view rendering of the scene directly into the modulation vectors stored in the modulation texture using a double frustum camera, and then 3) assembling the final hologram by gathering chirp and modulation vectors via texture fetches and then performing a dot product. Each basis chirp vector is independent of the particular scene being displayed but depends on its horizontal position in the hololine. Thus these can be computed once for a given display and stored as a 1-D texture to be used as a lookup table. Then the brightness of a point emitter from a given view direction is set by modulating some portion of the chirp making that emitter: the view ray is projected back through the emitter to the hologram plane (behind the emitter plane) and then the part of a chirp that contributes to that emitter s view from that direction can be found and multiplied by the modulation value (which is calculated by rendering the scene from a large number of viewpoints). We capture the scene with a virtual camera with its centers of projection at the positions of the emitters; it uses a double frustum to capture pixels in front of and behind the emitter plane. The novel projection matrix for this camera is handled by the vertex shader of the GPU. A common technique in video games to achieve high visual quality at reduced computational cost is to use a low-polygon-count model with higher-complexity texture and surface-normal maps. We follow this trend as well and use the fragment shader of the GPU to handle lighting, normal mapping, and texture mapping. Earlier this year we reported rendering performance results for the Mark III holo-video display with an NVIDIA Quadro 4500FX GPU (circa 2006). A 336 pixel x 440 line x 96 views hologram runs at 10 fps for a texture mapped model with 500 polygons with per-pixel normal mapped lighting (equivalent to a polygon model). If the modulation texture is prerendered, or the multi-view rendering and modulation vector assembly is performed once and the texture reused, the hologram is generated at 15 fps. While this is not up to the 30 fps refresh rate of the screen it is sufficient to provide

6 dynamic display capabilities for entertainment or graphical-user-interface purposes. We have also applied this rendering method to our old Mark II display (Fig. 3). As of this writing we are undertaking tests using newer GPU hardware and expect to release better performance figures shortly. We are also working on relaxing the restriction that emitters must lie on a plane, making our renderer more of a hybrid between a holo-stereogram and a physically-based model. Fig. 2. Diffraction specific holographic stereogram: parallax views modulate chirps and sum to form hologram made up of viewdependent point emitters. Views are captured from emitter plane using a double frustum geometry. As a comparision with our earlier work with our Mark II display and three circa-2004 GPUs, taking into account the differing polygon count, frame rate, and number of GPUs, we have now increased computational performance by roughly a factor of This increase is not simply from GPU architectural improvements but also from increased optimization of our code.

7 Fig. 3. Still image from an animated sequence (on our old Mark II display) of a texture-mapped dinosaur running past a grove of trees. 4. CONCLUSIONS AND OBSERVATIONS The era of interactive consumer holographic displays has nearly arrived. We have developed a rendering algorithm for commodity hardware which can render holographic stereograms of useable dimensions and large view numbers from modestly-sized texture-mapped models at interactive frame rates. Our use of standard 3-D graphics APIs makes our system compatible with much existing graphics content and software, and our pipeline has compatibility with real-world stereographic TV imagery as well as synthetic graphics. While we have demonstrated interactive-rate holo-video rendering at nearly SDTV resolution using a standard GPU, a collection of factors will continue to throttle the scale-up of our work. Recall that in a holo-video display (unlike a normal video display) the physical size of the pixels ideally much less than one micron is set by the physics of diffraction rather than by perceptual considerations. As a consequence, the pixel count increases with the display size (and with the square of the display size in full-parallax displays). We already need hololines that are significantly longer than the maximum line length supported by GPUs, and thus our display has to treat several scan lines from the GPU as a single hololine; a much larger screen would exceed the total pixel count per frame supported by current or planned GPUs of which we re aware. So even though computational speed of a GPU might grow to permit computing larger holo-video images in real time, there may not be enough framebuffer memory to hold the larger image. And should the framebuffer size increase, the electrical interconnect may still limit the ability to get the pixels out of the GPU and into the display (we re currently using all six channel outputs of a dual-head GPU just to achieve enough bandwidth to get a monochrome hologram into Mark III). MIT s first generation holo-video display, the Mark I, in one configuration demonstrated full-color holo-video. Now that compact solid-state RGB laser sources have been developed for consumer video projection applications, converting the Mark III apparatus to full-color would be relatively simple. Modifying our rendering algorithm to produce color holograms for Mark III is likewise straightforward, though different basis chirps will have to be used for red, green, and blue light so that each diffracts through the same angle. Perhaps the easiest way to handle color would be linesequential multiplexing, where the first hololine would be the red channel, the second hololine the green channel, and the third hololine the blue channel. The color-alternation rate would be high enough to avoid the rainbow effect that frame-sequential color displays exhibit for moving objects (or moving eyes). There is less likelihood of near-term availability of real-world imagery captured by 2-D camera arrays than of stereoscopic video, but given a high-enough-resolution 2-D light modulator and sufficient GPU framebuffer size and output

8 bandwidth, our rendering method is applicable to full-parallax displays by using 2-D zone plate basis functions instead of 1-D chirps, rendering the scene using a double frustum camera from a 2-D array of emitter positions rather than from a 1-D line, and summing dot products for zone plate vectors and modulation vectors. While we search for solutions to the scale-up problems above that still fit within the consumer-electronics space, and while we refine the image brightness and sharpness of the Mark III system and improve the performance of the rendering pipeline, there remain other tasks to perform. Chief among these are understanding the perceptual and artistic properties of dynamic 3-D content on our display, and evaluating the performance and image quality of rendering stereoscopic real-world imagery from various sources. 5. ACKNOWLEDGMENTS The authors gratefully acknowledge the late Steve Benton and the many alumni of the Spatial Imaging Group who started the holo-video project at the MIT Media Lab that led to our current research. This work has been supported by the Digital Life, Things That Think, and CELab consortia and the Center for Future Storytelling at the MIT Media Laboratory. Thanks also to NVIDIA for the donation of graphics hardware used in this research. REFERENCES 1. H. R. Johnston, C, A Hermanson, and H. L. Hull, Stereo-Television in Remote Control, Electrical Engineering, 69, pp , (1950). 2. A. Lit, The Magnitude of the Pulfrich Stereo-Phenomenon as a Function of Binocular Differences of Intensity at Various Levels of Illumination, Am. J. Psychol., 62, pp , (1949). 3. T. Okoshi, Three-Dimensional Imaging Techniques, Academic Press, New York, K.-T. Kim, M. W. Siegel, and J.-Y. Son, Synthesis of a High Resolution 3D-Stereoscopic Image from a High Resolution Monoscopic Image and a Low Resolution Depth Map, Proc. SPIE Stereoscopic Displays and Virtual Reality Systems V, 3295A, pp , (1998). 5. M.-S. Kim, G. Baasantseren, N. Kim, J.-H. Park, M.-Y. Shin, and K.-H. Yoo, Fourier Hologram Generation of 3D Objects Using Multiple Orthographic View Images Captured by Lens Array, Proc. SPIE Practical Holography XXIII, 7233, p , (2009). 6. B. Mendiburu, 3D Moviemaking: Stereoscopic Digital Cinema from Script to Screen, Focal Press, Burlington MA, USA, (2009). 7. J. A. Watlington, M. Lucente, C. J. Sparrell, V. M. Bove, Jr., and I. Tamitani, A Hardware Architecture for Rapid Generation of Electro-Holographic Fringe Patterns, Proc. SPIE Practical Holography IX, 2406, pp , (1995). 8. D. Smalley, Q. Smithwick, and V. M. Bove, Jr., Holographic Video Display Based on Guided-Wave Acousto- Optic Devices, Proc. SPIE Practical Holography XXI, 6488, p L, (2007). 9. Q. Y. J. Smithwick, J. Barabas, D. E. Smalley, and V. M. Bove, Jr., Real-Time Shader Rendering of Holographic Stereograms, Proc. SPIE Practical Holography XXIII, 7233, p , (2009).