3D-TV Rendering from Multiple Cameras

Transcription

1 Master s Thesis 3D-TV Rendering from Multiple Cameras Thesis ommittee: Prof. dr. ir. Jan Biemond Dr. C.P. Botha Dr. Emile A. Hendriks Dr. ir. Andre Redert Author Anastasia Manta amantamio@hotmail.om Student number Thesis supervisor Dr. Emile A.Hendriks Dr.ir. Andre Redert Date November 1, 008

2 - -

3 Prefae This thesis is the final work of my Master study at the Information and Communiation Theory group, Faulty of Eletrial Engineering, Mathematis and Informatis, at Delft University of Tehnology. It serves as doumentation of my work during the graduation, whih has been made from February 008 until November 008. This graduation has been preeded by a researh assignment on Multiview Imaging and 3DTV during the period Otober January 008. Anastasia Manta Delft, November 008 Aknowledgements One year ago, deiding my thesis topi was not an easy task. There were so many things in my mind and a wide range of different interesting topis to hoose from. One year after, I would sinerely like to thank my supervisors, Emile and Andre, for being really good teahers. I would like to thank them for being supportive and reative, for our fruitful disussions and for teahing me the way to overome problems and ome up with new fresh ideas. It was a pleasure working with them. The ICT group provided a friendly and pleasant environment to work. I would like to thank everybody who ontributed to the nie atmosphere of the group. Finally, I have to admit that spending two years in TUDelft was an unepetedly nie eperiene. I am grateful to all the people that were part of my eperiene. Espeially my family in Greee, my friends, the old and the new ones

4 - 4 -

5 Contents 1. Introdution a. Three - Dimensional Television (3D TV) b. Three-Dimensional Television; a hain of proesses Problem Statement Related work a. Model-based rendering (MBR) rendering with epliit geometry b. Image-based rendering (IBR) rendering with no geometry Hybrid tehniques rendering with impliit geometry d. Compatibility with 3DTV appliations Rendering a. Basi approah b. Weighted approah Outlier removal approah Eperimental setup Evaluation Disussion and Future work b. Future Researh Appendi A. Results B. Clustering eperiment C. Depth maps D. Mean Shift algorithm Referenes

6 1. Introdution a. Three - Dimensional Television (3D TV) As early as the 190s, TV pioneers dreamed of developing high-definition three-dimensional (3D) olor TV, as only this would provide the most natural viewing eperiene. During the net eighty years, the early blak-and-white prototypes evolved into high-quality olor TV, but the hurdle of 3D-TV still remains. The term three-dimensional television (3DTV) is understood as a natural etension of two-dimensional television whih produes a flat image on a sreen into the third dimension. This means that the impression of the viewer also involves the pereption of depth. Our two eyes view the world from slightly different positions and the two slightly different images registered by our eyes play a primary role in depth pereption. The brain merges this information and reates the pereption of a single 3D view. In the absene of the original sene or objet, the same pereption an be reated by employing systems eploiting this mehanism. The advent of three-dimensional television (3D TV) beame of relevane in early 1990s with the introdution of the first digital TV servies. The new type of media that epands the user eperiene beyond what is offered by now is being developed by the onvergene of new tehnologies from omputer graphis, omputer vision, multimedia, and related fields. Figure 1: Philips' lentiular lens 3-D display requires no speial glasses and an show 3-D senes simultaneously to several viewers. (Photo:Philips[1]) - 6 -

7 b. Three-Dimensional Television; a hain of proesses An integrated 3DTV system, involves different building bloks. The main bloks are: apturing of 3D moving senes, their representation, rendering, ompression and transport, and finally display. In the entire proessing hain there are numerous of hallenges []. For the aquisition of dynami senes, an array of ameras is needed in most ases. The amera setups range from dense onfiguration (Stanford Light Field Camera, [3]) to intermediate amera spaing, [4], and wide amera distribution (Virtualized RealityTM, [5]). For appliations involving multiple ameras, amera alibration follows the aquisition of the senes, in order to guarantee geometri onsisteny aross the different ameras. The purpose of the alibration is to establish the relationship between 3D world oordinates and their orresponding D image oordinates. One this relationship is established, 3D information an be inferred from D information and vie versa. From the aquired multiview data, one should onsider how to represent a 3D sene in a way that is suitable for latter proesses: rendering, ompression and transmission. Methods for 3D sene representation are often lassified as a ontinuum between two etremes. The one etreme is represented by lassial 3D omputer graphis methods. This approah an be alled geometry-based modeling, beause the geometry of the sene is desribed in detail. The other etreme is alled image-based modeling and does not use any 3D geometry at all. Purely image-based representations need many densely spaed ameras. In the rendering stage, whih is interrelated with the 3D representation, new views of senes are reonstruted. Depending on the funtionality required, there is a spetrum of renderings, from rendering with no geometry to rendering with epliit geometry. The tehnologies differ from eah other in the amount of geometry information of the sene. Having defined the data, effiient ompression and oding is the net blok in the 3D video proessing hain. There are many different data ompression tehniques orresponding to different data representations, [6]. For eample, there are different tehniques for 3D meshed, depth data et. The amount of multiview image data is usually huge, hene the data ompressing and the data streaming are hallenging tasks. The display is the last, but definitely not least, signifiant aspet in the hain of 3D vision. The display is the most visible aspet of the 3DTV and is probably the one by whih the general publi will judge its suess, [6]. Two main ategories of 3D displays are available, those whih imply personal speialized 3D headgear and those whih do not (autostereosopi displays) [7]

8 Displays in the first ategory, using personal headgear, are taking advantage of an obvious way to diret the views into the user s eyes. Eamples are head mounted display systems and systems using LCD shutter glasses. Although these solutions an provide a great immersive 3D eperiene, the publi relutane to wear devies is disouraging their mass-market aeptane. Autostereosopi displays with traking rely on real time traking of the viewer position and projeting the left and right parts of a stereosopi video stream into the orresponding user s eye. Current trakers and display optis are suited for a single-user 3D video eperiene only [8]. The ideal trakingless 3D displays shows so many images in different diretions in spae, that any viewer at any position sees different images with his left and right eye. Inherently, suh a display serves multiple users, and provides them not only with stereo imagery but also with the motion paralla ue. Current multi-view displays suh as the Philips 3D-LCD [9] show in the order of 10 images, within a narrow zone, e.g. 15, whih is repeated over a wider viewing angle to serve multiple viewers.. Problem Statement To enable the use of 3DTV in real-world appliations, the entire proessing hain as disussed before needs to be onsidered. There are numerous hallenges in this hain. In addition, there are strong interrelations between all the proesses involved. In this thesis the fous is set on the generation of the 3D ontent, and more speifially on the rendering part. The generation of 3D ontent is an ative researh area and lots of researh works an be found in the urrent literature. First the state of the art methods are ompared. Their advantages and disadvantages are disussed. The investigation of the methods shows that there is still spae for improvement and that none of the urrent methods an ompletely aomplish the requirements of a 3DTV appliation. We will introdue a new approah that mathes better a 3DTV appliation. The approah is tested on real data and its feasibility to reate 3D ontent is investigated. A ouple of adaptations of the approah are also disussed. This report has been strutured as follows. The seond hapter disusses related work. In this hapter the speial requirements and the equipment available are also disussed and the researh question is defined. The third hapter reviews the different analysis and rendering approahes that were used in the urrent projet, disussing the steps from one approah to the other and the novelties that eah of them introdues. In hapter four the eperimental setup is desribed. The evaluation of the disussed approahes, objetive and - 8 -

9 subjetive, is presented in hapter five. In hapter si the work onduted during this thesis projet is disussed. Some open issues and ideas for future researh are also stated

10 . Related work Content availability is a fator that influenes the suess of 3DTV. Content prodution is a diffiult task and it is even more diffiult to reate 3D effets in the ontent that are pleasing to the eye. As disussed before, a hain of interrelated stages have to be onsidered in order to enable the generation of 3D ontent. The Current literature with fous in the interrelated 3D representation and rendering stages is disussed in this hapter. As said before, the methods for 3D sene representation and rendering are often lassified as a ontinuum between two etremes. Geometry or model-based rendering and image-based rendering. In between the two etremes a number of methods eist whih make use of both approahes and ombine their harateristis. a. Model-based rendering (MBR) rendering with epliit geometry Geometry based rendering is used in appliations suh as games, Internet, TV and movies. The ahievable performane with these models might be eellent, typially if the senes are purely omputer generated. The available tehnology for both prodution and rendering has been highly optimized over the last few years, espeially in the ase of ommon 3-D mesh representations. In addition, state-of-the-art PC graphi ards are able to render highly omple senes with an impressive quality in terms of refresh rate, level of detail, spatial resolution, reprodution of motion, and auray of tetures []. With the use of geometry-based representations quite sparse amera onfigurations are feasible, but most eisting systems are restrited to foreground objets only [10,11,1]. In other words, despite the advanes in 3-D reonstrution algorithms, reliable omputation of full 3-D sene models remains diffiult. In many situations, only a ertain objet in the sene suh as the human body- is of interest. Here, prior knowledge of the objet model is used to improve the reonstrution quality. Voel-based representations, [13], an easily integrate information from multiple ameras but are limited in resolution. The work of Vedula et al., [13], based on the epliit reovery of 3D sene properties, first uses the so-alled voel oloring algorithm to reover a 3D voel model of the sene at eah time instant. The non-rigid motion of the sene between onseutive time instants is reovered, Figure. In the net stage, the voel models and sene flow beome inputs to a spatio-temporal view interpolation algorithm

11 Figure : A set of alibrated images at onseutive time instants. From these images, 3D voel models are omputed at eah time instant using the voel oloring algorithm. After omputing the 3D voel models, the dense non-rigid 3D motion or sene flow between these models is omputed. A prominent lass of geometry reonstrution algorithms is shape-from-silhouette approahes. Shape-fromsilhouette reonstruts models of a sene from multiple silhouette images or video streams. Starting from the silhouettes etrated from the amera pitures, a onservative shell enveloping the true geometry of the objet is omputed by reprojetng the silhouette ones into the 3-D sene and interseting them. This generated shell is alled the visual hull. While the visual hull algorithms are effiient and many systems allow for real-time reonstrution the geometry models they reonstrut are often not aurate enough for high-quality reonstrution of human ators. Sine senes involving human ators are among the most diffiult to reonstrut, researh studies foused on free viewpoint videos of human ators an be found in literature. A typial eample is the work of Carranza et al, [10], reently updated by Theobalt et al. [11], where a triangle mesh representation is employed beause it offers a losed and detailed surfae representation. Sine the model must be able to perform the same omple motion as its real-world ounterpart, it is omposed of multiple rigid-body parts that are linked by a kinemati hain. The joints between segments are parameterized to reflet the objet s kinemati degrees of freedom. Besides objet pose, the dimensions of the separate body parts also must be kept adaptable as to be able to math the model to the objet s individual stature. A virtual reality modeling language (VRML) geometry model of a human body is used [Figure 3(a)]. Indiatively, the model onsists of 16 rigid body segments, one eah for the upper and lower torso, nek and head; and pairs of the upper arms, lower arms, hands, upper legs, lower legs, and feet. In total, more than 1,000 triangles make up the human body model

12 (a) (b) Figure 3: (a) Surfae model and the underlying skeletal struture. Spheres indiate joints. (b) Typial amera and light arrangement during reording. The hallenge in applying model-based analysis for free-viewpoint video reonstrution is to find the way to adapt the geometry model automatially and robustly to the objet s appearane as it was reorded by the video ameras. Sine the geometry model is suitably parameterized to alter its shape and pose, the problem redues to determining the parameter values that ahieve the best math between the model and the video images. This task is regarded as an optimization problem. The objet s silhouettes, as seen from the different amera viewpoints, are used to math the model to the reorded video images. The foreground in all video images is segmented and binarized. At the same time, the 3-D objet model is rendered from all amera viewpoints using onventional graphi hardware, after whih the rendered images are thresholded to yield binary masks of the model s silhouettes. Then, the rendered model silhouettes are ompared to the orresponding image silhouettes: as omparison measure, or mathing sore, the number of silhouette piels is used that do not overlap when putting the rendered silhouette on top of the reorded silhouette. Conveniently, the logial elusive-or (XOR) operation between the rendered image and the reorded image yields those silhouette piels that are not overlapping. By summing over the non-overlapping piels for all images, the mathing sore is obtained. To adapt model parameter values suh that the mathing sore beomes minimal, a standard numerial nonlinear optimization algorithm runs on the CPU. For eah new set of model parameter values, the optimization routine evokes the mathing sore evaluation routine on the graphis ard. After onvergene, objet teture an be additionally eploited for pose refinement. One advantage of model-based analysis is the low-dimensional parameter spae when ompared to general reonstrution methods (Figure 5): the parameterized 3-D model may provide only a few dozen degrees of freedom that need to be determined, whih greatly redues the number of potential loal minima. Many highlevel onstraints are already impliitly inorporated into the model, suh as kinemati apabilities. Additional onstraints an be easily enfored by making sure that all parameter values stay within their anatomially plausible range during optimization

13 Figure 4: Analysis-by-synthesis: To math the geometry model to the multiview video footage, the foreground objet is segmented and binarized, and the 3-D model is rendered from all amera viewpoints. The boolean XOR operation is eeuted between the referene images and the orresponding model renderings. The number of non-overlapping piels serves as mathing sore. Via numerial optimization, model parameter values are varied until the mathing sore is minimal. Figure 5: From eight video ameras spaed all around the sene, model-based method an apture the omple motion of a daner. After model-based motion apture, a high-quality 3-D geometry model is available that losely, but not eatly, mathes the dynami objet in the sene. For photorealisti rendering results, the original video footage must be applied as teture to the model. Projetive teture mapping is a well-known tehnique to apply images as teture to triangle-mesh models. To ahieve optimal rendering quality, however, it is neessary to proess the video tetures offline prior to real-time rendering [10]: loal visibility must be onsidered orretly to avoid any rendering artifats due to the inevitable small differenes between model geometry and the true 3-D objet surfae. Also, the video images, whih are taken from different viewpoints, must be blended appropriately to ahieve the impression of one onsistent objet surfae teture. Beause model geometry is not eat, the referene image silhouettes do not orrespond eatly to rendered model silhouettes. When projeting the referene images onto the model, teture belonging to some frontal body segment potentially leaks onto other segments farther bak [Figure 6(a)]. To avoid suh artifats, eah

14 referene view s penumbral region must be eluded during teturing. To determine the penumbral region of a amera, verties of zero visibility are determined not only from the amera s atual position but also from a few slightly displaed virtual amera positions [Figure 6(b)]. For eah referene view, eah verte is heked whether it is visible from all amera positions, atual as well as virtual. A triangle is projetively tetured using a referene image only if all of its three verties are ompletely visible from that amera. (a) (b) Figure 6: Penumbral region determination: (a) Small differenes between objet silhouette and model outline an ause teture of frontal model segments to leak onto segments farther bak. (b) By projeting eah referene image onto the model also from slightly displaed amera positions, regions of dubious visibility are determined. These are eluded from teturing by the respetive referene image. Most surfae areas of the model are seen from more than one amera. If the model geometry orresponded eatly to that of the reorded objet, all amera views ould be weighted aording to their proimity to the desired viewing diretion and blended without loss of detail. However, model geometry has been adapted to the reorded person by optimizing only a omparatively small number of free parameters. The model is also omposed of rigid body elements whih is learly an approimation whose validity varies, e.g., with the person s apparel. In summary, the available model surfae an be epeted to loally deviate from true objet geometry. Aordingly, projetively teturing the model by simply blending multiple referene images auses blurred rendering results, and model teture varies disontinuously when the viewpoint is moving. Instead, by taking into aount triangle orientation with respet to amera diretion, high-quality rendering results an still be obtained for predominantly diffuse surfaes [10]. After uploading the 3-D model mesh and video ameras projetion matries to the graphis ard, the animated model is ready to be interatively rendered. During rendering, the multiview imagery, predetermined model pose parameter values, visibility information, and blending oeffiients must be ontinuously uploaded, while the view-dependent teture weights are omputed on the fly on the GPU

15 b. Image-based rendering (IBR) rendering with no geometry The other etreme in 3-D sene representation is alled image-based rendering and does not use any 3-D geometry at all. The main advantage is a potentially high quality of virtual view synthesis avoiding any 3-D sene reonstrution. However, this benefit has to be paid by dense sampling of the real world with a suffiiently large number of natural amera view images. In general, the synthesis quality inreases with the number of available views. Hene, typially a large number of ameras have to be set up to ahieve highperformane rendering, and therefore a tremendous amount of image data needs to be proessed. Contrarily, if the number of used ameras is too low, interpolation and olusion artifats will appear in the synthesized images, probably affeting the quality. Eamples of image-based representations are ray spae or light field [14, 15] and panorami onfigurations inluding onentri and ylindrial mosais [16]. The underlying idea in most of these methods is apturing the omplete flow of light in a region of the environment. Suh a flow is desribed by a plenopti funtion. The plenopti funtion was introdued by Adelson and Bergen, [17], in order to desribe the visual information available from any point spae. It is haraterized by seven dimensions, namely the viewing position (V, V y, V z ), the viewing diretion (θ, φ) or (,y) in Cartesian oordinates, the time t and the wavelength λ. (l (7) (V, V y, V z, θ, φ, λ, t)). Figure 7: The 7D plenopti funtion The image-based representation stage is in fat a sampling stage, samples are taken from the plenopti funtion for representation and storage. The problem with the 7D plenopti funtion is that it is so general that due to the tremendous amount of data required, sampling the full funtion into one representation is not feasible. Researh on image-based modeling is mostly about how to make reasonable assumptions to redue the sample data size while keeping the rendering quality. One major strategy to redue the data size is restraining the viewing spae of the viewers. There is a ommon set of assumptions that people made for restraining the viewing spae. Some of them are preferable, as they do not impat muh on the viewers eperienes. Some others are more restritive and used only when the storage size is a ritial onern. By ignoring wavelength and time dimensions, MMillan and Bishop [18] introdued plenopti modeling, whih is a 5D funtion: l (5) (V, V y, V z, θ, φ)

16 They reord a stati sene by positioning ameras in the 3D viewing spae, eah on a tripod apable of ontinuous panning. At eah position, a ylindrial projeted image was omposed from the aptured images during the panning. This forms a 5D image-based representation: 3D for the amera position, D for the ylindrial image. To render a novel view from the 5D representation, the loseby ylindrial projeted images are warped to the viewing position based on their epipolar relationship and some visibility tests. The most well-known image-based representations are the light field [15], and the Lumigraph [19] (4D). They both ignored the wavelength and time dimensions and assumed that radiane does not hange along a line in free spae. However parameterizing the spae of oriented lines is still a triky problem. The solutions they ame out happened to be the same: light rays are reorded by their intersetions with two planes. One of the planes is indeed with oordinate (u,v) and the other with oordinate (s,t), i.e.: l (4) (s,t,u,v) In Figure 8, an eample where the two planes, namely the amera plane and the foal plane, are parallel is shown. This is the most widely used setup. An eample light ray is shown and indeed as (u 0,v 0,s 0,t 0 ). The two planes are then disretized so that a finite number of light rays are reorded. If all the disretized points from the foal plane are onneted to one disretized point on the amera plane, we get an image (D array of light rays). Therefore, the 4D representation is also a D image array, as is shown in Figure 9. Figure 8: One parameterization of the light field. Figure 9: A sample light field image array: fruit plate. The differene between light field and Lumigraph is that light field assumes no knowledge about the sene geometry. As a result the number of sample images required in light field for apturing a normal sene is

17 huge. On the other hand, Lumigraph reonstruts a rough geometry for the sene with an otree algorithm to failitate the rendering (disussed later on) with a small amount of images. For this reason Lumigraph is sometimes lassified as an hybrid and not a pure image-based modeling tehnique. In the rendering stage in order to reate a new view of the objet, the view is splitted into its light rays, whih are then alulated by quadlinearly interpolating eisting nearby light rays in the image array. For eample, the light ray (u 0,v 0,s 0,t 0 ) in Figure 8 is interpolated from the 16 light rays onneting the solid disrete points on the two planes. The new view is then generated by reassembling the split rays together. Suh rendering an be done in real time and is independent of the sene ompleity. Lumigraph, has also the advantage that an be onstruted from a set of images taken from arbitrarily plaed viewpoints. A re-binning proess is therefore required. Geometri information is used to guide the hoies of the basis funtions. Beause of the use of geometri information, sampling density an be redued. Other than the assumptions made in light field, onentri mosais, [16], further restrits that both the ameras and the viewers are on a plane, whih redues the dimension of the plenopti funtion to three. In onentri mosais, the sene is aptured by mounting a amera at the end of a level beam, and shooting images at regular intervals as the beam rotates, as is shown in Figure 10. The light rays are then indeed by the amera position or the beam rotation angle a; and the piel loations (u, v): l (3) (α,u,v). Figure 10: Conentri mosai apturing The rendering of onentri mosais is slit-based. The novel view is split into vertial slits. For eah slit, the neighboring slits in the aptured images are loated and used for interpolation. The rendered view is then reassembled using these interpolated slits. Although vertial distortions eist in the rendered images, they an be alleviated by depth orretion. Conentri mosais do not require the diffiult modeling proess of reovering geometri and photometri sene models. Yet they provide a muh riher user eperiene by allowing the user to move freely in a irular region and observe signifiant paralla and lighting hanges. All these methods do not make any use of geometry, but they either have to ope with an enormous ompleity in terms of data aquisition or they eeute simplifiations restriting the level of interativity

18 . Hybrid tehniques rendering with impliit geometry In between the two above etremes, lies rendering with impliit geometry, a truly ative researh area. When 3D models of the objets and sene are unavailable, like in IBR tehniques, user interation is limited. If approimate geometry of objets in a sene an be reovered, then interative editing of real senes, whih is one of the objetives of 3DTV, is failitated. A lassial approah is the use of depth or disparity maps. Suh maps assign a depth value to eah sample of an image. Together with the original two-dimensional (-D) image the depth map builds a 3-D-like representation, sometimes alled.5-d, [0]. The point where impliit geometry beomes epliit is not very lear. In this report systems that use geometry information in the form of disparity maps are regarded as systems with impliit geometry, therefore rendering tehniques using depth information are inluded in this hapter. When the depth information is available for every point in one or more images, 3D warping tehniques an be used to render nearby viewpoints. An image an be rendered from any nearby point of view by projeting the piels of the original image to their proper 3D loations and re-projeting them onto the new piture. The most signifiant problem in 3D warping is how to deal with holes generated in the warped image. Holes are due to the differene of sampling resolution between the input and output images, and the disolusion where part of the sene is seen by the output image but not by the input images. To fill in holes, the most ommonly used method is to splat a piel in the input image to several piels in the output image. Closer to the geometry-based end of the spetrum, methods are reported that use view-dependent geometry and/or view dependent teture [1]. Instead of epliit 3-D mesh models also point-based representations or 3-D video fragments an be used [], (ETH). Washbush et al, [, 3], for eample, proposes a view-independent point-based representation of the depth information. The ETH Institute, whih is onduting this researh, uses several so-alled 3D video briks that are apturing high-quality depth maps from their respetive. Eah brik is movable and ontains one olor, two graysale ameras, and a projetor. The mathing algorithm used for depth etration is assisted by the projetors illuminating the sene with binary strutured light patterns. Teture and depth are aquired simultaneously. Eah brik aquires the sene geometry using depth from stereo algorithm. Depth maps are omputed for the images of the left and right graysale ameras by searhing for orresponding piels. Stereo mathing is formulated as a maimization problem over an energy whih defines a mathing riterion of two piel orrelation windows. An adaptive orrelation window is used overing multiple time steps only in the stati part of the orrelation windows. For the moving parts disontinuities in the image spae are going to be

19 etended into the temporal domain, making orrelation omputation more diffiult. Therefore the orrelation window is etended in the temporal dimension, only for stati parts, to over three or more images. Figure 11: The 3D video brik with ameras and projetor (left), simultaneously aquiring tetures (middle) and strutured light patterns (right) A two phase post proessing is applied to the disparity images. First the regions of wrong disparity are identified and then new disparities are etrapolated into these regions from their neighbours. To model the resulting three-dimensional sene, a view-independent, point-based data representation is used. All reonstruted views are merged into a ommon world referene frame. Three video briks were used in the eperiments of paper [3], thus three different views. The model is in priniple apable of providing a full view if the sene has been aquired from enough viewpoints. It onsists of a set of samples, where eah sample orresponds to a point on a surfae and desribes its properties suh as loation and olor. Every point is modeled by a three-dimensional Gaussian ellipsoid spanned by the vetors t 1, t, t 3, around its enter p. This orresponds to a probabilisti model desribing the positional unertainty of eah point by a trivariate normal distribution: with epetation value p and ovariane matri 1 p ( ) N ( ; p, V ) e 3 ( ) V T ( 1 3) ( 1 3) V t t t t t t 1 T 1 ( p ) V ( p ) omposed of 3 1 olumn vetors t 1. Assuming a Gaussian model for eah image piel unertainty, first the bak-projetion of the piel into three-spae is omputed whih is a D Gaussian parallel to the image plane spanned by two vetors t u and t v. Etrusion into the third domain by adding a vetor t z guarantees a full surfae overage under all possible views. This is illustrated in Figure

20 Figure 1: Constrution of a 3D Gaussian ellipsoid. Eah piel (u, v) is spanned by orthogonal vetors σ u (1,0) T and σ v (0,1) T in the image plane. Assuming a positional deviation σ, the piel width and height under unertainty are σ u = σ v = 1+σ, σ is estimated to be the average re-projetion error of the alibration routine. The depth of eah piel is inversely proportional to its disparity d as defined by the equation z f L L R d p L p R Where f L is the foal length of the retified amera, L and R are the enters of projetion, and p L and p R, the u-oordinates of the prinipal points. The depth unertainty σ z is obtained by differentiating the above equation and augmenting the gradient Δd of the disparity with its unertainty σ : fl L R z ( d ) ( d p L p R) After bak-projetion the point model still ontains outliers and falsely projeted samples. Some points originating from a speifi view may look wrong from etrapolated views due to reonstrution errors, espeially at depth disontinuities. In the 3D model, they may over orret points reonstruted from other views, disturbing the overall appearane of the 3D video. Thus, those points are removed by heking the whole model for photo onsisteny with all teture ameras. In the rendering stage they adopt an interesting approah desribed by Broadhurst et al., [4]. Broadhurst uses probabilisti volume ray asting to generate smooth images. Eah ray is interseted with the Gaussians of the sene model. At a speifi intersetion point with the sample i, the evaluation N(;p i ;V i ) of the Gaussian desribes the probability that a ray hits the orresponding surfae point. To ompute the final piel olor, two different approahes are desribed. The maimum likelihood method assoiates a olor with the ray using only the sample whih has the most probable intersetion. The seond approah employs the Bayes rule: it integrates all olors along eah ray weighted by the probabilities without onsidering olusions. Thus, the olor of a ray R is omputed as: - 0 -

21 R R R i in ( ; p i, V i) i N ( ; p i, V i) The maimum likelihood method generates risp images, but it also sharply renders noise in the geometry. The Bayesian approah produes very smooth images with less noise, but is inapable of handling olusions and rendering solid surfaes in an opaque way. The rendering method that the authors propose ombines both approahes in order to benefit from their respetive advantages. The idea is to aumulate the olors along eah ray like in the Bayesian setting, but to stop as soon as a maimum aumulated probability has been reahed. Reasonably, a Gaussian sample should be ompletely opaque if the ray passes its enter. The line integral through the enter of a three-dimensional Gaussian has a value of 1/π and for any ray R it holds that: R N ( ; p, V ) 1 Thus, they aumulate the solution of the integrals of the above equation by traversing along the ray from the amera into the sene and stop as soon as the denominator of the equation reahes 1/π. Assuming that solid surfaes are densely sampled, the probabilities within the surfae boundaries will be high enough so that the rays will stop within the front surfae. The results of omparison of this approah with the maimum likelihood and Bayesian rendering on noisy data are demonstrated in Figure 13. The large distortions in the maimum likelihood image, get smoothed out by the other two methods. However, the Bayesian renderer blends all the points inluding those from oluded surfaes, while urrent method renders opaque surfaes and maintains the blending. Figure 13: Comparison of maimum likelihood (left) and Bayesian rendering (enter) with Washbush et al. s approah (right). A representative result of the method is shown in Figure 14, where novel views of the aquired sene are rendered with the reonstruted 3D model. A sample video an be found in [5]

22 Figure 14: Re-renderings of the 3D video from novel viewpoints. The model is in priniple apable of providing a full view if the sene has been aquired from enough viewpoints. Compared to mesh-based methods, points provide advantages in terms of sene ompleity beause they redue the representation data and do not arry any topologial information, whih is often diffiult to aquire and maintain. As eah point in the model has its own assigned olor, they also do not have to deal with teturing issues. Moreover, a view-independent representation is very suitable for 3D video editing appliations sine tasks like objet seletion an be ahieved easily with standard point proessing methods. The point based data representation model is in general suitable for 3DTV appliations but still ontains many redundanies. Big lusters of outliers tend to grow in the disontinuity optimization stage if they dominate the orret depths in a olor segment. Furthermore as the authors alaim the resulting image quality ould also be improved by eploiting inter-brik orrelation and eliminating remaining artifats at silhouettes using matting approahes as done by Zitnik et al. [4]. Zitnik et al., [4], propose a quite sophistiated representation. First, guided by the fat that using a 3D impostor or proy for the sene geometry an greatly improve the quality of the interpolated views, they generate and add per-piel depth maps (multiple depth maps for multiple views). However, even multiple depth maps still ehibit artifats (at the rendering stage) when generating novel views. This is mainly beause of the erroneous assumption at the stereo omputation stage, that eah piel has a unique disparity. This is not the ase for piels along the boundary of objets that reeive ontributions from both the foreground and bakground olors. Zitnik addresses this problem using a novel two-layer representation inspired by Layered Depth Images [6]. The two layer representation is one of the most effiient rendering methods for 3D objets with omple geometries and it was proposed already in 1998 by Shade et al. A Layered Depth Image is an array of layered depth piels ordered from losest to furthest from a so-alled LDI amera. Eah layered depth piel ontains multiple depth piels at per-piel loation. The farther depth piels, whih are oluded from the viewpoint at the enter of LDI amera, will appear as the viewpoint - -

23 moves away from the enter of LDI amera. This image-based rendering tehnique is an abstrat of depth images, whih assigns a z-value in addition to the and y values normally assoiated with a piel. When rendering from an LDI, the requested view an move away from the original LDI view and epose surfaes that were not visible in the first layer. The previously oluded regions may still be rendered from data stored in some later layer of a layered depth piel. Figure 15: Layered Depth Image (LDI) and LDI amera There are two ways to generate an LDI from natural images: generate LDI from multiple depth images and generate LDI from real images diretly [6], illustrated as in Figure 15. If multiple olor and depth images from natural images are obtained, the LDI is onstruted by warping piels in other amera loations, suh as C1 and C. Otherwise, The LDI an be generated diretly from the input images by an adaptation of Seitz and Dyers voel oloring algorithm [7]. The voel oloring method assigns olors to voels (points) in a 3D volume so as to ahieve onsisteny with a set of basis images. In [7], a ordinal visibility onstraint is eploited in order to simplify visibility relationships. In partiular, it beomes possible to partition the sene into a series of voel layers that obey a monotoni visibility relationship: for every input image, voels only olude other voels that are in subsequent layers. Consequently, visibility relationships are resolved by evaluating voels one layer at a time. The voel oloring algorithm visits eah of the N 3 voels eatly one and projets it into every image. Therefore, the time ompleity of voel oloring is O(N 3 n), where n the number of images. Suh a ompleity prohibits the use of the algorithm for the modeling of a whole sene. Zitnik, [4], overomes this onstraint, by alulating the matting, or seond layer information, in speifi areas and not for the whole sene. Matting information is omputed within a neighbourhood of four piels from all depth disontinuities. A depth disontinuity is defined as any disparity jump greater than λ (=4) piels. Within these neighborhoods, foreground and bakground olors along with opaities (alpha values) are omputed using Bayesian matting [8]. The foreground information is ombined to form the boundary layer as shown in Figure 16. The main layer onsists of the bakground information along with the rest of the - 3 -

24 image information loated away from the depth disontinuities. Chuang et al. s algorithm do not estimate depths, only olors and opaities. Depths are estimated by using alpha-weighted averages of nearby depths in the boundary and main layers. To prevent raks from appearing during rendering, the boundary matte is dilated by one piel toward the inside of the boundary region. Figure 16: Two-layer representation: (a) disontinuities in the depth are found and a boundary strip is reated around these; (b) a matting algorithm is used to pull the noundary and main layers B i and M i (The boundary layer is drawn with variable transpareny to suggest partial opaity values) Figure 17 shows the results of applying the stereo reonstrution and two-layer matting proess to a omplete image frame. It is notieable that only a small amount of information needs to be transmitted to aount for the soft objet boundaries, and how the boundary opaities and boundary/main layer olors are leanly reovered. Figure 17: Sample results from matting stage: (a) main olor estimates; (b) main depth estimates; () boundary olor estimates; (d) boundary depth estimates; (e) boundary alpha (opaity) estimates. For ease of printing the boundary images are negated, so that transparent empty piels show up as white. In the rendering stage, given a novel view, the nearest two ameras in the data set are piked. For eah amera the main data and boundary data are projeted into the virtual view. The results are stored in separate buffers eah ontaining olor, opaity and depth. These are then blended to generate the final frame. The layered depth image approah, provides results with quite good visual quality. However, even for an approah like Zitnik s dense sene sampling is neessary. For eample in order to ahieve overage about 30 o, a onfiguration of 8 ameras was used. The disussed approah proesses eah amera independently from the others. Better quality ould be obtained by merging data from adjaent views when trying to estimate the semi-oluded bakground. This would also allow the multi-image matting to get even better estimates of foreground and bakground olor and opaities

25 d. Compatibility with 3DTV appliations This thesis addresses several problems and onstraints disussed here. Motivation to test a new approah emanates from the speial requirements that a 3DTV appliation has. The fat that the 3D video should inlude not only the foreground objets but also the bakground is onsidered as a premise. The fleibility and salability, important harateristis of an effiient approah, motivate the use of a perpiel algorithm. Per-piel operations allow fleibility in the sense that their ompleity depends on the resolution of the initial images. Therefore, omputational ost ould be dereased by using low quality images, or inreased when high resolution initial images are used. The salability is important, in the sense that more ameras an be added, and in this ase the algorithm preserves its harateristis but the quality of the final result is refined, or a wider view range is feasible. Parallel omputation, that is feasible for independent per-piel operations, ould also failitate a real time system. The approahes disussed in hapter three are also motivated from the limited reording failities (low number of ameras) available in the studio at the university. An approah that would be able to generate a 3D video aptured initially by four stereo ameras, of resolution, was a hallenging task onsidering the urrent literature studies demanding dense amera onfigurations. Motivated from 3DTV requirements and state of the art limitations is therefore the work presented in the net hapters of this thesis projet

26 3. Rendering The 3D video aquisition system used in this thesis onsists of four alibrated pairs of stereo ameras that are apturing high quality images (640480) from their respetive viewpoints. Therefore the input, of the algorithms disussed in this hapter, is eight images from different viewpoints, as shown in the following Figure. S C E N E Figure 18: Four stereo ameras are used, apturing in total 8 images. Starting point of the investigation of an algorithm that would fulfill the 3DTV requirements was a standard used in the ICT group. This approah does not follow the typial order of stages, as this was disussed in hapter one. The 3D representation and rendering stage are substituted by an analysis stage, analyzing the information from all the eight ameras, and a rendering stage, in whih the virtual views are being generated. Virtual are onsidered the views from positions (virtual amera positions) where ameras were not originally plaed (Figure 19). The two stages, 3D representation and rendering, are strongly interrelated. There is no intermediate representation, whih means that every time a new virtual view has to be generated the analysis stage has to run before. The method is using per-piel operations. Charateristis and features, that neighboring piels may have in ommon, are not taken into aount. This has the advantage that gives the method freedom, to regenerate only part of one view for eample without loss of auray. The drawbak is of ourse that it does not - 6 -

27 onsider properties of neighboring piels that ould be useful and save omputational load in an objet-based approah for eample. a. Basi approah In order to generate a new view, at a virtual position a ray traing sheme is adopted. Every piel of the virtual view is onsidered independently. Figure 19 is illustrating the method with a one piel eample. This piel omes from a depth position in the 3D sene. In order to figure out its depth position different depth values are tested. This is onduted as follows: a light ray is onstruted for the piel under onsideration. The ray is modeling all possible depths of the piel. Therefore it should be able to model all the depth positions possible in the original 3D sene. A step for the depth is seleted. For the whole range of depths, one depth-step at a time from the virtual view s piel is projeted to all the original images. The piels on whih is projeted are the orresponding piels for the given depth. 105m. 0m Figure 19: Here two different depth hypotheses (105m, 0m) are projeted to the original images

28 The piel in every original image mathing the given depth should have an RGB value. In ase the projetion of a depth does not orrespond to a piel within the borders of the original image then this original image is not onsidered effetive and is not partiipating in the regeneration of the virtual view. By olleting the RGB values, a set of N ams 3 values is formed, where N ams the number of effetive ameras. The mean value,, and the variane,, of the olor in this dataset are omputed for eah of the three olor omponents: k z Nam 1 k Nam z z (1) k z Nam 1 k Nam z z k z () where k r, g, b, and r stands for the red olor omponent, g for the green and b for the blue. Nam stands for the number of effetive ameras, and is the inde of the ameras. The r, g, b olor varianes are then summed up: RGB z z + z + z (3) r g b The RGB olor variane as it appears in equation 3 is alulated for every step of the ray, or in other words for every depth hypothesis. In ase the depth hypothesis is orret the variane should have a low value. This happens beause for the orret depth, the original ameras should be in agreement, sine they should be viewing the same sene point, and the olleted RGB values should onverge. A high variane implies disagreement among the ameras, and is not in general the ase of a orret depth. Therefore from all the depth hypotheses, the one with the lowest variane is seleted. z SEL min RGB ( z) (4) z For this depth the mean RGB value of all the effetive ameras is assigned to the new piel. The omputed variane for one piel in a virtual view as a funtion of possible depths is plotted in Figure 0. The piel under onsideration is a piel at the front part of the sene, on the table

29 Color Variane (a) Depth-variane plot Depth (b) Figure 0: (a): The depth hypothesis for a piel at the front part of the table is visualized. (b):depth-variane diagram. The above diagram is a visualization of the variane of the piel indiated in (a) for all the depths from 1m to 3m. In this diagram there is a lear mode (minimum), at 1.35 meters and this is the depth that will be seleted. However there are ases where the depth-variane diagram has not a lear minimum value. Possible auses of suh ases are e.g. homogeneous areas or ases of olusion. An eample of a homogeneous area is the white table, illustrated with an arrow in the net figure. The table is not overing only one speifi depth in the sene. The depth variane diagram is shown as well and it is obvious that in this ase, the original ameras are in an agreement, for a wide range of depths

30 Color Variane (a) Depth-variane plot Depth (b) Figure 1: (a): The depth hypothesis for a piel on the white table is visualized. (b): Depth-variane diagram. The above diagram is a visualization of the variane of the piel indiated in (a) for all the depths from 1m to 3m. The depth that is going to be seleted by the algorithm may not be the orret one. It would be the one with the lowest variane, but in this ase this is not a safe riterion. In Figure, the mean RGB values, as they were alulated for every depth are also plotted. For depth from 1. meters to 3 meters the olor is white, with some small hanges probably due to illumination hanges. Although the seleted depth ould be wrong, the olor of the piel in the regenerated view is going to be orret

31 Color Variane (a) Depth (b) Figure : (a): The depth hypothesis for a piel on the white table is visualized. (b) Depth-variane diagram for the piel indiated in for all the depths from 1m to 3m. (a). The dots sattered in the diagram indiate the mean RGB value for eah depth. Therefore homogeneous areas may are responsible for errors in the depth map of the sene but are not ausing any artifats in the final image. The olusion problem ours when part of the objets whih onstitute the sene, or part of the bakground is overed by other objets. In this appliation where multiple ameras are used, oluded areas an be present at some of the images aptured by the original ameras or even in all of them. This auses spread within the set of the olleted RGB values. A piel between the two boes at the right is seleted and its depth-variane results are visualized. The original ameras seem not to agree for the RGB values orresponding to piels in this area. The ause of the disagreement is that for some of the 8 original ameras the purple bo is oluding the bakground, for some

32 Color Variane of them the stripped bo is oluding the bakground and there are also those ameras whih are apturing the bakground. The depth hypothesis is therefore not valid, sine the orret depth may be not the one with the lowest variane. (a) Depth-variane plot Depth (b) Figure 3: (a): The depth hypothesis for a piel at the bakground is visualized. (b):depth-variane diagram. The above diagram is a visualization of the variane of the piel indiated in (a) for all the depths from 1m to 3m. b. Weighted approah In order to overome the problem of olusion, and make the disussed approah ompatible with the regeneration of oluded areas, the proimity of the original ameras to the virtual view is onsidered. In urrent literature amera proimity has been widely used. In the work of Min et al. for eample [9], given a viewpoint the nearest two images (amera i and i+1) are seleted and projeted into virtual view. Smoli et al. as well in [30], using a multiamera setup, are building 3D video objets. In order to projet the tetures from the original views onto the geometry, voel models, they are alulating a weight. For eah projeted teture a normal vetor n i is defined pointing into the diretion of the original amera. For - 3 -

33 generation of a virtual view into a ertain diretion v VIEW a weight is alulated for eah teture, whih depends on the angle between v VIEW and n i. The weighted interpolation ensures that at original amera positions the virtual view is eatly the original view. The loser the virtual viewpoint is to an original amera position the greater the influene of the orresponding teture on the virtual view. The approah disussed at the beginning of this hapter is therefore adapted in order to integrate the original amera virtual view position proimity. Every time a virtual view at a new position is alulated, a weight is assigned to every original amera: 1 w (5) 4 (( ) ( y y ) ( z z ) 10 vv vv vv Where ( vv, y vv, z vv ) is the position of the virtual view, and (, y, z ) the position of the original amera. The weight is different for every, of the =[1, 8], original ameras. When the distane approahes zero, the weight gets a great value, but it an never approah infinity, due to the 4 10 fator. We assumed that the minimum distane between a amera and a virtual view is 1m=10 -. So, the squared minimum distane 4 gives the fator10. In ase the projetion to one of the ameras is not effetive, it is projeted out of borders as already disussed this amera is assigned a zero weight. The weights of the ameras are partiipating in the alulation of olor variane. The olor variane is now alulated as follows: WEIGHT, k ( z ) 8 1 w k ( z ) 8 1 w (6) WEIGHT 8 w k ( z ) 1, k ( z ) 8 WEIGHT, k 1 w (7) where k r, g, b. The r, g, b weighted olor varianes are then summed up: ( ) ( ) + ( ) + ( ) (8) WEIGHT, RGB z w, r z w, g z w, b z

34 Color Variane The weighted olor variane is alulated for every depth hypothesis. The depth with the lowest variane is seleted: z WEIGHT, SEL min WEIGHT, RGB ( z) z (9) and its the mean RGB value as alulated in (6) is assigned to the piel. The depth-variane diagram, influened from the amera proimity is shown in the net figure, for a piel in the area with olusion, between the two boes. (a) Depth-variane plot erroneous foreground Depth (b)

35 Color Variane 00 Depth-variane plot orret bakground Depth () Figure 4: (a): The depth hypothesis for a piel at the bakground is visualized.(b,):depth-variane diagrams. The diagrams are visualizations of the variane of the piel for all the depths from 1m to 3m. (b) non-weighted variane, () weighted variane. In Figure 4, the depth variane diagram for the first approah disussed in hapter 3.a and the weighted variane diagram are illustrated. The piel under onsideration is between the two boes and it belongs to the bakground. When the first approah is used (Figure 4.b), the lowest variane is around 1.4 meters. This depth orresponds to the bo and is erroneously seleted. In the weighted approah the ameras at positions lose to the virtual view have a stronger impat on the variane. The lowest variane is now deteted for depths around. meters (depth that orresponds to the bakground).. Outlier removal approah The weighted approah ould solve the olusion problem but only partially. There are still artifats appearing in the regenerated images. In ase of olusion, or in ase of illumination hanges for eample, it an happen that the ameras are viewing different olors even for the orret depth. An insight on the values of the ameras for piels with erroneous olors, shows that the main ause of the errors is the eistene of outlier values. Taking as an eample a seond piel, neighboring to the one appearing in Figure 4, thus within the oluded area, it turns out that for this piel the weighted approah is not working

36 Color Variane Color Variane Depth-variane plot 9000 erroneous foreground Depth (a) 5000 Depth-variane plot erroneous foreground Depth (b) Figure 5: Depth-variane diagrams of a piel in the oluded area. The diagrams are visualizations of the variane of the piel for all the depths from 1m to 3m. (a) non-weighted variane, (b) weighted variane. Although the weights influene the variane of the ameras, the depth with the lowest v is erroneously around 1.6 meters (the ground truth lies around. meters). A plot of the RGB values of the eight ameras indiates the ause of the error:

37 Figure 6: The dots in the diagram stand for the olors of the eight ameras, for depth. meters. The olor of the dot represents the RGB olor of the amera. The dots are plotted in the RG spae. For the orret depth four of the ameras are viewing the stripped bo, one of them the purple bo and the rest three ameras the bakground. The variane is alulated for all the ameras, therefore if outliers are present for the orret depth, this is not going to be seleted by the algorithm. Probably there will be another depth value for whih the amera values would be less spread. Solution to this ause of errors is the removal of the outlier values. There are plenty of methods for the removal of the outliers. A binary lassifiation of the input data into inlier and outlier lass ould be ahieved with RANSAC, [31], or major vote method. Hard lustering, assigns eah data point (feature vetor) inside with a degree of membership equal to one or outside the luster, with a degree of membership zero. In our appliation the borderline between inliers and outliers is not risp. There is muh ontroversy on what onstitutes an outlier. The borderline depends on the speifi sene, and ould have a strong effet on the sharpness of the final image. In order to have the fleibility to handle suh situations, by adapting the parameters of the algorithm for eample, a non binary lustering algorithm without a priori assumptions on the number of lusters is used. The method used resembles the mean shift algorithm, an algorithm used broadly in image proessing appliations, for image segmentation for eample, [3], (Appendi D). The mean shift and the algorithm used here as well are unsupervised. The lusters and the outliers are identified using an iterative sheme. The loation of the luster entroids is not known a priori. Here, the mean value of the RGB values as alulated in (4), (the weights of the ameras are not used here), is the starting vetor for the number of iterations (t>0). t O.The inde t stands

38 The RGB values of eah of the eight ameras, are represented by a vetor V,.,( [1,8] stands for the number of the amera). V is therefore a 31 vetor, ontaining information for the three olor omponents of amera. The eight V vetors onstitute the dataset of the algorithm. The distane, t d,of the initial mean value d t t O, with respet to eah vetor V is alulated as: t ( z ) O ( z ) V ( z ) and is an estimate of the variane of the V values. The ameras for whih the distane is high are onsidered outliers, whereas those with low distane value are onsidered inliers. Therefore a salar weight is assigned to eah of them, omputed as the inverse of the sum (10) of the distanes plus a indiating the minimum aeptable size of a luster: t 1 w OUTR ( z) (11) t d ( z) The weights of the ameras are normalized, and a new vetor for the net iteration is defined as: 8 8 t 1 t O ( z ) w ( z) V ( z ) (1) OUTR, with 1 1 t w ( z ) 1 (13) The distane of the starting value O from the vetors V is alulated again, new weights are assigned to the ameras and the starting value O is reomputed with the new amera weights. OUTR The iterative proedure stops when a predefined number of iterations are eeuted. The sequene of points should finally onverge to one of the so-alled attrators of the iterative sheme. This is the final entroid of the luster. With the method desribed above, none of the ameras has a zero weight. Even the ameras that are onsidered outliers are assigned a very small weight whih may pratially elude them from the alulation of entroid of the luster (the final RGB mean value). At the end of the iterative proess a number indiating the number of the ameras that are pratially partiipating in the alulation of the mean value (effetive number of member-ameras) is omputed. 8 w OUTR ( z ) log w OUTR ( z ) 1 N ( z ) e (14) where w OUTR w OUTR, the weights after the iterations

39 There are different ways to integrate the outlier removal method in the sequene of proesses. One possible senario, is first to selet the depth, using the approah disussed at hapter 3.a, based on the olor variane (the amera proimity weights may be used as well). Then for the seleted depth, whih probably ontains outlier values, the outlier removal method an be performed. The mean value RGB value, the value that would be finally assigned to the piel, would be realulated without the outliers. This approah saves omputational ost, sine the outlier removal method whih is quite epensive is eeuted only one for every piel. However, is non-optimal. The initial RGB values and the olor varianes alulated for all the depths inlude outliers. A less ompromising senario, in terms of auray and orretness of the final result, is to use the outlier removal method for all the possible depths before the alulation of the variane. Therefore what we investigated is first to remove the outliers from the set of the RGB values of the original ameras and then proeed with the seletion of the depth. In more detail, given a piel and a depth, the outlier removal method outputs weights for all the ameras, w OUTR. These weights an be used to alulate the new mean and variane (non-weighted outlier removal approah). Only removing the outliers was tested and did not give suffiient results therefore the outlier removal weights have been linearly ombined with the amera proimity weights, removal), as alulated in (6): w (weighted outlier w ( z) w ( z) 0. 7 w (15) FINAL OUTR The 0.7 fator was assigned to the amera proimity weights based on eperimental observations and the fat that their influene should not overome the outlier removal weights. The mean RGB value is realulated based on the new ombined weights of the ameras. The variane of the eight RGB values from the enter of the luster (the new mean value) also. FINAL, k ( z ) 8 1 w 8 1 FINAL w k ( z ) FINAL (16) 8 w FINAL k ( z ) 1 FINAL, k ( z ) 8 FINAL k 1 w FINAL (17)

40 Color Variane where k r, g, b. Both the variane and the number of effetive ameras, as alulated in 14, are used as riterions in the seletion of the depth. The number of effetive ameras is used to prevent seletion of depths for whih very few ameras are onsidered inliers. This an happen in ase the values are randomly distributed in the RGB spae. Then the outlier removal method ould hoose one or two observations as inliers, having also a very low variane. This is general not the orret ase so the seletion of lusters with few members is stopped, in the following way. Starting from the lowest depth, the first loal minimum variane is seleted and it is only replaed if a lower variane with a higher number of ameras is present for a higher depth. The reason that the sanning proedure starts from the lower depths to the higher ones is that we want to selet the first visible objet (foreground objet) and not objets behind it. As an eample, one piel in the oluded area is onsidered. As it was already disussed in hapter 3.a and 3.b this piel is the ase that annot be orretly generated with the basi or weighted approah. Even when the weights are onsidered, the seleted depth is not orret (in the area of 1.6 m) Depth-variane plot 9000 erroneous foreground Depth (a)

41 Color Variane Color Variane 5000 Depth-variane plot erroneous foreground Depth (b) 100 Depth-variane plot 90 5 ams ams orret bakground 3 ams ams am Depth () Figure 7: Depth-variane diagrams of a piel in the oluded area. The diagrams are visualizations of the variane of the piel for all the depths from 1m to 3m. (a) non-weighted variane, (b) weighted variane, () variane after the removal of outliers. In the above diagram the blue line is the olor variane diagram, after the outlier removal. The olorful dots indiate the number of the effetive number of ameras (N). In this eample the ombination of a low variane and a high number of amera members happens for a depth around.4 meters. In this ase the outlier removal method works for the olusion problem

42 There are still ases for whih this approah would not work. When the majority of the ameras are viewing the wrong olor, and the minority the orret, the outlier removal approah will onsider inliers the wrong values. The real inliers will be eluded from the alulation of the final olor of the piel. In suh a ase that the orret values are eluded the result is worse than the one of the basi or the weighted approah

43 4. Eperimental setup The presented eperimental setup uses a pair-wise multi-view amera setup of 8 views, whereas the ameras are mounted without any geometri restritions. Video frames are reorded at a resolution at 15 fps. Figure 8: Eperimental setup. Before the reording of the sene, the ameras were alibrated.the amera alibration step is an integral part of the proess of 3D video aquisition. It is determining the internal amera geometri and optial harateristis (intrinsi parameters) and the 3D position and orientation of the amera frame relative to a word oordinate system (etrinsi parameters). This step is neessary to quarantee geometri onsisteny aross the different terminals. A alibration plate ontaining 47 dots was aptured in four different positions and an minimum energy alibration algorithm, as proposed by Redert et al. at [33], was used

44 Figure 9: Calibration plate plaed in four different positions for the alibration algorithm. After the alibration step, eah of the ameras aptured one image from a different viewpoint. The reorded sene is stationary. The algorithm has been tested in stationary senes; however it an be also used for video sequenes. In this ase eah frame is reonstruted independently from the previous ones therefore the temporal harateristis of a video are not taken into aount. Different senes were reorded, with different bakgrounds, simple and more omple ones, people or just objets. (a) (b) () (d)

45 (e) (f) (g) Figure 30: Captured senes with different seneries. (h) From the above senes, two were seleted to serve as the senes in whih the algorithms will be tested. A sene with a olorful, omple bakground and four boes and a sene with a simple bakground and a human, were hosen. From now on the first sene is referred as four boes sene (Figure 30.d), and the seond one as offee sene (Figure 30.g). In both senes there are objets oluding the bakground or other objets so the handling of olusions an be also assessed. Professional lighting equipment was not used. The lighting of the room where the reordings took plae is the normal eiling lighting. Changes in illumination are present in the sene

46 5. Evaluation Objetive evaluation of image and video quality is important for image proessing systems. The quality assessment is by itself a wide researh area, having as a goal to design algorithms for objetive evaluation of quality in a way that is onsistent with subjetive human evaluation. By onsistent it is meant that the algorithm s assessment of quality should be in lose agreement with human judgments, regardless of the type of distortion orrupting the image, the ontent of the image, or strength of distortion. In the image oding and omputer vision literature, the most frequently used measures are deviations between the original and oded images, with varieties of the mean square error (MSE) or signal-to-noise ratio (SNR) being the most ommon measures [34-36]. The reason for their widespread popularity is their mathematial tratability. However they do not neessarily orrespond to all aspets of the observer s visual pereption of the errors, [34, 35], nor do they orretly reflet strutural oding artifats. For multimedia appliations, there has been an inrease in the use of quality measures based on human pereption [37-4] Sine a human observer is the end user in multimedia appliations, an image quality measure that is based on a human vision model seems to be more appropriate for prediting user aeptane and for system optimization. This lass of distortion measure in general gives a numerial value that will quantify the dissatisfation of the viewer in observing the reprodued image in plae of the original. The alternative is the use of subjetive tests in whih subjets view a series of reprodued images and rate them based on the visibility of the artifats. Subjetive tests are tedious, time onsuming and epensive, and the results depend on various fators suh as the observer s bakground, motivation, et., and atually only the display quality is being assessed. An objetive evaluation although not being the optimal one is usually part of the testing of an image proessing tehnique. In this thesis, PSNR (peak signal-to-noise ratio) is used. The derivative signal to noise ratio, a riterion that onsiders the sensitivity of the human eye in the edge information is also omputed. A subjetive evaluation, based on the author s visual interpretation is finally disussed. Different approahes eist for omputing PSNR of a olor image. One simple approah is to alulate the metris for the three R,G,B omponents and then average them. But, the human eye is most sensitive to intensity information. Therefore the image is first onverted to a olor spae that separates the intensity hannel, here YCbCr. The Y (luma), in YCbCr represents a weighted average of R, G, and B. G is given the most weight, again beause the human eye pereives it most easily. With this onsideration, the PSNR is omputed only on the luma hannel

47 The PSNR, is alulated as follows: PSNR 10 log MSE (18) where MSE i j, [ I o ( i, j) I ( i, M r j)] (19) Where I o [i,j] and I r [i,j] are the intensity values at the (i,j) th piels of the original and reonstruted images respetively and M the number of piels. In order to ompare the reonstruted views with the original images, the parameters of the virtual amera were set to the same values as one of the original ameras. For every virtual view the original view that oinides with it is eluded. Therefore in the evaluation part eight views are generated, using eah time seven ameras. This influenes a lot the PSNR values presented below. Espeially in the weighted approah, eluding the losest in the virtual view amera deteriorates the final result. The PSNR measure is omputed for ropped images. The borders, not being aptured from all the ameras, have been eluded from the evaluation as they are not representative. For images with dimensions these are the PSNR results: 4 boes sequene view 1 view view 3 view 4 view 5 view 6 view 7 view 8 basi approah weighted outlier removal Table I: PSNR for ropped images, 4 boes sene (in db)

48 view 1 view view 3 view 4 view 5 view 6 view 7 view 8 basi approah weighted outlier removal Coffee sequene Table II: PSNR for non-ropped images, 4 boes sene (in db) view 1 view view 3 view 4 view 5 view 6 view 7 view 8 basi approah weighted outlier removal Table III: PSNR for ropped images, offee sene (in db) view 1 view view 3 view 4 view 5 view 6 view 7 view 8 basi approah weighted outlier removal Table IV: PSNR for non-ropped images, offee sene (in db) The results of the ropped images have a higher PSNR as it was epeted. For both sequenes the weighted approah has the best performane. The outlier removal method does not enhane the quality of the final images in general. For a ouple views the outlier removal method has a higher PSNR than the weighted approah and for other views lower than the basi approah. The performane of the methods depends on the senery and as it was eplained in hapter 3. there are ases that the removal of the outlier an deteriorate the final result

49 Another remark is that when the whole image is evaluated the outlier removal method has better metris than the basi and the weighted approah. The method handles better the generation of the borders of the images, but this is not indiative of its general behaviour. Yalazan et al.,[43] are proposing the derivative signal to noise ratio as an objetive fidelity riterion whih onsiders the sensitivity of the human eye to the edge information. Human eye uses edge information while pereiving an image. The authors laim that the new riterion should be sensitive to regions of fast hanges (edges) in an image in order to agree with subjetive deisions. Derivative SNR 10 log i, j I ( i, j) ( I ( i, j) I ( i, j)) o 10 i, j Where I o ( i, j) and I r ( i, j) are approimate gradients at the (i,j) th piel of the original and reonstruted images respetively, whih are alulated as: o r (0) I ( i, j) I ( i, j) I ( i 1, j) I ( i, j) I ( i, j 1) (1) The last row and last olumn are not used in the alulation of derivative SNR sine the above equation annot be applied to those piels. The equation omputes the horizontal and vertial hanges at eah piel, hene the edge information is emphasized. In other words I(i,j) etrats the high frequeny omponents of the image. When a reonstrution proess is not muh suessful in reproduing the edges of the original image, the noise term in the denominator of the equation gets large resulting in a poor value of the derivative SNR. The derivative SNR riterion whih emphasizes the ontribution from the high frequeny region is epeted to be more orrelated to subjetive deisions beause of its similarity to the pereptional mehanism of the human eye. 4 boes sequene basi approah (10-5 ) weighted (10-5 ) outlier removal (10-5 ) view 1 view view 3 view 4 view 5 view 6 view 7 view Table V: Derivative SNR for ropped images, 4 boes sequene (in db)

50 Coffee sequene basi approah (10-5 ) weighted (10-5 ) outlier removal (10-5 ) view 1 view view 3 view 4 view 5 view 6 view 7 view Table VI: Derivative SNR for ropped images, offee sequene (in db) The values of the derivative SNR highlight the fat that the outlier removal method is the least visually aeptable. As it is going to be disussed in the subjetive evaluation, the outlier removal method gives nonsmoothed images and this irritates the visual pereption. In order to evaluate subjetively the methods, the results on Appendi A are used. The virtual views regenerated from the basi approah are visually aeptable taking into aount that the method remains quite simple and fast. To have a better insight, the sene under onsideration an be divided into different parts. The borders of the images, where there are artifats, should be eluded from the evaluation phase. In the first sene, the bakground is highly tetured. The bakground in the regenerated images keeps the high teture. The white table is a homogeneous area at the original views eept few piels due to illumination hanges; in general the reonstruted table has a smooth view. The problem with this approah is loated in the area around the boes and espeially between the two boes at the right (the purple and the stripped one). Part of bakground that should be visible is erroneously reonstruted. Figure 31: A virtual view generated by the basi approah. When the amera proimity weights are integrated in the method the generated virtual views are better. The improvement is visible espeially for the last virtual views, where the olusion problem is present

51 Figure 3: A virtual view generated by the weighted approah. The quality of the images generated by the outlier removal method seems to deteriorate in omparison with the weighted approah. A grain effet is visible espeially in homogeneous areas like the table. The amera values are not averaged, some of them are not partiipating in the alulation of the olor of the piel. This has as a onsequene that the surfaes are less smooth. The olusions are handled well, and this was epeted sine the proimity weights are used as well. Figure 33: A virtual view generated by the outlier removal approah The offee sequene, has a smooth simple bakground, and does not inlude a highly oluded area. The results for this sequene are visually better. The basi approah gives a bit blurred images, but the other two methods seem to work effiiently

52 6. Disussion and Future work The goal of the researh of this thesis was to study the feasibility of a simple algorithm to reate 3D ontent for 3DTV appliations. The requirements of a 3DTV appliation for a whole sene video with editing apabilities and high quality, and the onstraints imposed by the number of the available ameras formulate the researh area. First the state of the art methods were investigated. There are two main ategories, as it was disussed in detail in hapter two. The model-based approahes, using omple geometry models of the sene. Syntheti videos, using blue-sreen as a bakground for eample, an be generated with suh tehniques but their ompleity makes their use prohibitive for the generation of a video of the whole sene. The other ategory, image-based approahes, is able to reonstrut 3D videos for whole senes and that is that makes it more promising for 3DTV appliations. The dense sampling of the sene that is required ompliates its use. Current methods annot work effiiently without imposing a lot of limitations. That was the motivation of this thesis. We investigated a new approah that uses per-piel operations to allow for parallel omputations, salability and fleibility. For eah piel to be rendered a depth hypothesis was generated. It was solved by projeting the piel in the amera planes and omparing the olor values. The algorithm worked properly, with some eeptions, e.g. areas with olusion. To overome the olusion problem amera proimity weights were assigned to the ameras. The distane between the position of the virtual and original amera was alulated. Original ameras loser to the amera view were assigned a greater weight. The improvement in the visual quality of the generated images was signifiant. However artifats were still present. Investigation of the RGB values of the eight ameras for one piel, indiated that there were different reasons for whih the amera agreement hypothesis for the orret depth was not true. An olusion for eample is a potential ause for outlier values in the RGB dataset. To solve the outlier problem we investigated two approahes, a non-weighted outlier removal and a weighted outlier removal method. The generated images are not visually better in omparison with the ones generated with the basi approah. However for some speial areas the outlier removal method works properly. So, integration of more sophistiated riterions in the depth seletion stage may lead to better results

53 The methods disussed have the advantage that an generate a 3D video of the whole sene while they keep the omputational ompleity of the system low and have as input only eight original images. They an diretly benefit by using more ameras or ameras with higher resolution in order to enhane the final result. The fat that there is no intermediate representation, adds on the freedom of the adopted methods but on the other hand any inter-view relations are not taken into aount. Every time a new virtual view is generated the analysis stage has to run. To sum up, after analyzing in detail the behavior of piels in the reonstrution stage we gained a very good insight of their behavior. This information was inorporated in the approahes tested. The results are very promising. Some improvements an be made. The senery for eample an be segmented in areas aording to their speial harateristis (e.g. oluded areas, objets at the front). Then eah segment ould be regenerated by a different approah aording to its speial harateristis. The depth maps of the generated images, (Appendi C) ould be used as an indiator of whih areas need speial attention. In Appendi B, a lustering eperiment highlights the fat that the speial harateristis within an area of the image are spatial onsistent. b. Future Researh Layered Depth Images are used widely for rendering 3D objets. Our approah ould be etended to more than one layer. The per-piel analysis failitates the generation of layers from the original images. Preliminary eperiments were onduted in whih the layers were seleted as modes of the olor variane. The depth with minimum olor variane was onsidered as the first layer, the depth with the seond minimum variane the seond one. The spatial and temporal onnetivity, both present in sequene of images, were not taken into aount. In a future researh their use ould enhane for eample the non-smooth result of the outlier removal method

54 7. Appendi A. Results Four boes sequene Figure 34: Si of the virtual views for the 4 boes sequene generated with the basi approah. The positions if the virtual ameras are shown in figure

55 Figure 35: Si of the virtual views for the 4 boes sequene generated with the weighted approah

56 Figure 36: Si of the virtual views for the 4 boes sequene generated with the outlier removal approah

57 offee sequene Figure 37: Si of the virtual views for the offee sequene generated with the basi approah

58 Figure 38: Si of the virtual views for the offee sequene generated with the weighted approah

59 Figure 39: Si of the virtual views for the offee sequene generated with the outlier removal approah. Figure 40: The red stars are indiating the positions of the si virtual views shown in the previous figures, in relevane to the original ameras onfiguration

60 B. Clustering eperiment Here, a piel is onsidered, and the outlier removal algorithm is iterated eight times, having as a starting point, V, one of the vetors ontaining the RGB information for one of the ameras eah time. The dataset of the algorithm are the eight V vetors. For the different iteration points the algorithm outputs different mean values. Then the eight different mean values are ompared, and values with distane less than a threshold are ombined. In this way the possibility of different lusters or single outlier values within the dataset of the eight V vetors is investigated. The purpose of this eperiment is to test in areas with different harateristis what amera lusters are generated for different depths. The spatial onsisteny of these results is also investigated. In the following diagram, one piel in the front part of the four boes sequene is onsidered, and the lusters found for every depth are visualized. The blue line is the variane of all the eight ameras as a funtion of depth alulated with the basi approah as disussed in setion 3.a. The represents one luster, its y value indiates the variane of the luster and finally the olor stands for the number of the ameras partiipating in the luster. This figure orresponds to one piel in the front part of the image, on the table. It an be notied easily that for the orret depth, around 1.4 meters, all the eight ameras are in agreement. Therefore there are no outliers. 3 1 Figure 41: Position of the three piels disussed in this setion

61 ais Color Variane Depth-variane plot 8 ams 7 ams ams 5 ams 4 ams ams ams Depth 1 am Figure 4: Depth olor variane plot. The symbol stands for a luster of ameras. Its olor indiates the number of the members of the luster, its height the variane of the luster. For a piel in the front part of the image (Figure 0), all the eight ameras are in agreement for the orret depth. To hek the onsisteny of the lustering between neighboring piels, the lustering algorithm now is performed in more piels, sequential in the ais. Only the biggest luster of eah depth is plotted in the net diagram ams 50 7 ams 40 6 ams 5 ams 30 4 ams 0 3 ams 10 ams Depth 1 am Figure 43: The number of amera members of a luster is indiated by the olor of the symbol. 40 piels suessive in the ais are onsidered. A piel in the oluded area is now onsidered. The reated lusters omprise of 1-3 ameras highlighting the fat that the ameras are distrated by olusion. The same holds for neighboring piels

62 y ais Color Variane Depth-variane plot 8 ams ams 6 ams ams 4 ams ams ams Depth 1 am Figure 44: Depth olor variane plot. The symbol stands for a luster of ameras. Its olor indiates the number of the members of the luster, its height the variane of the luster. For a piel in the olusion area (Figure 7), the ameras are not in agreement and they are forming separated small lusters ams 40 7 ams ams 5 ams 0 4 ams ams ams Depth Figure 45: The number of amera members of a luster is indiated by the olor of the symbol. 40 piels suessive in the y ais are onsidered. 1 am A piel in the bo positioned in the middle of the table is onsidered. The ameras are in agreement but for a wide range of depths (Figure 46). The lustering is performed for neighboring piels, in both ais and y (Figures 47 and 48). It an be notied that within this neighborhood of piels, the number of amera members is onsistent for similar depths

63 y ais Color Variane 1000 Depth-variane plot 8 ams ams ams 5 ams ams ams 000 ams Depth 1 am Figure 46: Depth olor variane plot. The symbol stand for a luster of ameras. Its olor indiates the number of the members of the luster, its height the variane of the luster. For a piel on the bo, the ameras are in agreement for a wide area of depths ams 40 7 ams ams 5 5 ams 0 4 ams ams ams Depth 1 am Figure 47: The number of amera members of a luster is indiated by the olor of the symbol. 40 piels suessive in the y ais are onsidered

64 ais 45 8 ams 40 7 ams ams 5 5 ams 0 4 ams ams ams Depth Figure 48: The number of amera members of a luster is indiated by the olor of the symbol. 40 piels suessive in the ais are onsidered. 1 am C. Depth maps Our approah when reonstruting a virtual view is impliitly reonstruting its depth map. Three depth maps are shown here reonstruted by the basi, the weighted and the outlier removal approah respetively. Figure 49: virtual view and its depth map reonstruted with the basi approah

65 Figure 50: virtual view and its depth map reonstruted with the weighted approah. Figure 51: virtual view and its depth map reonstruted with the outlier removal approah. D. Mean Shift algorithm Mean shift whih was proposed in 1975 by Fukunaga and Hostetler [44], is a nonparametri, iterative proedure that shifts eah data to loal maimum of density funtion. An intuitive desription of the mean shift algorithm is shown in Figure 36. Given a set of data points, the objetive is to find the densest region