Texture Un mapping. Computer Vision Project May, Figure 1. The output: The object with texture.

Transcription

1 Texture Un mappin Shinjiro Sueda Dinesh K. Pai Computer Vision Project May, 2003 Abstract In computer raphics, texture mappin refers to the technique where an imae is pasted onto a 3-dimensional surface. This technique can sinificantly increase the realism of a scene without increasin the complexity [Heckbert 96]. All modern raphics hardware can perform this operation almost for free. However, obtainin such imaes from real world can be a hard problem. We present a simple and mechanical way in which we can extract arbitrary textures from an arbitrary object. Fiure 1. The output: The object with texture. Introduction Texture mappin has been used extensively in interactive raphics applications ever since the hardware manufacturers started incorporatin the texture mappin loic into their video cards. It is popular, because it is efficient, and it is effective. A well-textured, adequately-tessellated scene looks much better than a non-textured, well-tessellated scene. This is because texture mappin is a per-pixel operation, while tessellatin is a per-vertex operation. Our oal is to extract the texture map imae from a series of photoraphs. This can be thouht of as an inverse problem of computer raphics. In computer raphics, the final scene is constructed from the model s eometry information and a texture map imae. Our oal is to somehow extract the texture map imae from the final scene and the eometry. We assume that we know the eometry of the object, since we can hand scan the object that we are takin the picture of usin FastScan [FastScan].

2 Fiure 2. Polhemus FastScan in action. Firstly, we ive an overview of the texture mappin technique. Then, we present some of the difficulties we came across durin implementation. Finally, we demonstrate the results of our method on a toy football. Backround Texture mappin was invented by Ed Catmull [Catmull 74] in the 70 s, but it was not until the early 90 s, when the popularity of video ames exploded, that prorammers started usin this technique extensively. In simple terms, a texture map is an imae that is pasted onto an object to add colour information to each pixel of the object. In a standard raphics pipeline, the input is a list of vertices, and the output is the pixels on the screen. Each vertex passed in as the input has various fields associated with it. Standard fields include vertex position and vertex normals. The raphics pipeline can be thouht of as a black box function whose job is to process the fields of the input vertices and convert them into appropriately coloured and positioned pixels on the screen. If an object is to be texture mapped, an additional set of vertex field must be passed into the pipeline. This set of extra information is called the texture coordinates, and it encodes the coordinates in the texture map imae that must be looked up when colourin the vertex. The main difficulty in texture mappin arises from the fact that the eometry is 3D, while the texture imae is 2D. The texture coordinates ive the mappin between 3D eometry information to 2D texture imae information. By passin in 2D texture coordinates for each vertex, we can indirectly wrap the 2D texture imae onto the 3D object. Texture mappin can be used for many applications [Haeberli 93]. However, we will only discuss the most popular usae of textures applyin diffuse patterns to objects. Solution Convertin a photoraph taken from an ordinary camera into a texture map is not a trivial task, since the orientation of the object that we want to model is not explicit in the

3 photoraph. By usin the Vicon [Vicon] motion capture system, we know the exact positions of both the camera and the object and thus we can enerate the texture map from the photoraph. The main idea is to render the object in OpenGL, compare the resultin imae to the photoraph, and infer the texture map from the differences between the two imaes. Fiure 3. Vicon motion-trackin system. Vicon motion trackin system allows us to et the precise position and orientation information of the object and the camera. Vicon is a marker-based trackin system, meanin that we must place markers on the camera and the object before we can track them. By emittin and measurin the reflected infrared sinal from the retro-reflective markers, Vicon trianulates and reports the positions of the markers. By attachin markers to the camera, we can track the camera, but there needs to be some preprocessin in order to know where exactly the camera s pin-hole location is with respect to the frame defined by the markers on the camera. The first difficulty that must be addressed is the problem of camera calibration. When the two imaes are bein compared, they must line up exactly, or else comparin them does not make sense. Camera calibration is a well known problem, and there are countless software packaes available in academia. We chose to use the Matlab packae made available by Caltech [Bouuet 03]. This packae ives us both the intrinsic and the extrinsic parameters of the camera, by takin as input a series of pictures of a rid of a known size.

4 Fiure 4. Series of rid pictures used by the calibration packae. The camera model in OpenGL is a simple pin-hole projection model. In eneral, the intrinsic parameters provided by camera manufacturers are for a thick-lens camera model. Therefore we cannot use those values in the OpenGL model. The camera calibration packae ives us the correct pin-hole model parameters that we are lookin for. The intrinsic parameters required by OpenGL are the aspect ratio and the field of view. The aspect ratio is simply the width of the photoraph divided by the heiht (1024 / 768). The field of view is not iven explicitly by the packae, and therefore a simple calculation is required. Instead of the field of view, the packae ives the focal lenth in pixels. Usin simple eometry, the field of view, θ, is: y θ tan = 2 2 z y θ = 2arctan, 2z where y is the heiht of the imae (768 pixels) and z is the focal lenth. The packae showed that the focal lenth in vertical pixels is not the same as the focal lenth in horizontal pixels, since the CCDs in cameras are never exactly square. We simply took the averae of the two values in our calculation. Fiure 5. Calculatin the FOV, theta.

5 The other relevant intrinsic parameter not directly related to OpenGL renderin is the lens distortion. The calibration packae included an undistort feature, which takes in an imae, and outputs an imae with the lens distortions cancelled out. However, we did not use this feature, since the outputted imae was restricted to be rey-scale. We felt that as lon as we keep the object centred on the photoraph and away from the edes, the effects of lens distortion are minimal and can be inored. The three extrinsic camera parameters required by OpenGL are the camera location, camera look at direction, and the camera up direction. The calibration packae ives us two sets of extrinsic parameters, Tc and Rc. Tc ives the location of the camera with respect to the calibration rid, and Rc is the rotation matrix that takes a point in the rid frame to a point in the camera frame. A point P in the rid frame and a point c P in the camera frame are related to each other throuh the followin riid motion equations (The leadin superscript indicates which frame the entity is expressed in): c P = ( Rc)( P) + Tc T c P = ( Rc )( P Tc) (Note: The inverse of Rc is Rc T, since Rc is orthoonal.) Then the camera location, look at, and up vectors in the rid coordinate frame are: T loc = Rc ( Tc) 0 T lookat = Rc 0 Tc 1 0 T up = Rc 1 0 Fiure 6. Camera coordinate frame reported by the calibration packae. So now, we have the extrinsic parameters in the rid coordinate frame. How can we use this information to et the parameters in world coordinates? As noted earlier, iven the Vicon marker locations, we do not know the three extrinsic parameters required in the coordinate frame defined by those markers. In order to et the extrinsic parameters in the world frame, we first need those parameters in camera s marker coordinates. This can be obtained by performin the camera calibration step while trackin the markers

6 placed on both the camera and the rid with Vicon. The series of transformations required for the three parameters are as follows: c c w P= E E P w c 1 c w w ( E) E P P =, where the matrix a b E denotes the riid transformation matrix that transforms a vector in frame b into frame a, and the decorators c,, and w specify camera, rid, and world respectively. The riid transformation matrices w c E and w E are constructed from the marker locations iven by Vicon. This ives us c P, which denotes the extrinsic parameters in the camera s marker frame. Gettin the three extrinsic parameters in world coordinates is trivial now that we have c P. When the camera moves to a new orientation, Vicon ives us the new transformation, w c E. Given this, the extrinsic parameters in world frame is then: w w c P= E P c Since we are assumin that we know the eometry of the object, it is very easy to render the object in OpenGL now that we have both the intrinsic and the extrinsic parameters of the camera in world coordinate frame. (Hopefully, the rendered imae and the photoraph will match up exactly!) After the renderin, each pixel in the rendered imae and the photoraph is compared aainst each other, and the texture imae is updated appropriately. Specifically, for each pixel (x, y) in the rendered imae, we set the pixel of the texture imae (a.k.a texel) correspondin to that pixel to be the colour value of the photoraph at pixel (x, y). One obvious question is, What do you mean by texel correspondin to that pixel? As stated in the introduction, the raphics pipeline takes as input the object s vertex information, which includes the texture coordinates, and transforms it into pixels on the screen. Texel correspondin to the pixel simply means the texture coordinates (s, t) of each pixel (x, y) on the rendered imae. We have stated that the raphics pipeline is a black box that transforms vertex information into pixels on the screen. This is not quite true, as raphics hardware manufacturers have started implementin prorammable vertex and pixel pipeline into their products [Fernando 03]. This new feature opens up the black box so that the prorammers can control all the processin that oes on inside the pipeline. Usin this feature, ettin the texture coordinates of a iven pixel is very easy. Instead of encodin colour information into the final pixel, we simply encode the texture coordinates of the pixel as its output. Then the rendered imae will be encoded with the s coordinate in the red channel, and the t coordinate in the reen channel. The process of updatin the texture is as follows: 1. Read the colour information at each pixel (x, y) in the rendered imae. This colour encodes the texture coordinates at that pixel. 2. Read the colour information at correspondin pixel (x, y) in the photoraph. 3. Write the colour from the photoraph into the output texture at coordinates (s, t), where s is the red component and t is the reen component of the pixel from the rendered imae.

7 The fact that we can take as many photoraphs of the object as necessary is one of the main advantaes of our method. Each photoraph outputs a separate texture map. To combine the output textures into a sinle texture, we simply take the averae of the textures, if there are any overlaps between two textures. Results We decided to use a toy football as our model, because of its eometric simplicity. Since it is convex, only 6 photoraphs are required. Fiure 7. Camera and object Setup. Note the placement of markers on both the camera and the toy football. Fiure 8. The photoraph from the camera is shown on the left, and the correspondin renderin of the object is shown on the riht. Note the exact correspondence of the object in the two imaes.

8 Fiure 9. Fillin the texels. For each pixel in the rendered imae, et the correspondin texture coordinates and update the colour of the texture at those coordinates. In this example, the current pixel in the rendered imae is (550, 300). The RGB colour of this pixel is (135, 204, 0), which corresponds to (0.53, 0.8, 0). The RGB colour of the (550, 300) th pixel in the photoraph is (71, 88, 82). Therefore, we set the (0.53, 0.8) th pixel of the output texture to be (71, 88, 82).

9 Fiure 10. Repeat the procedure six times to obtain all the required textures. Now we need to mere these 6 textures into a sinle texture. Fiure 11. The final texture obtained by combinin the 6 textures and takin the averae per pixel.

10 Fiure 12. Four views of the object rendered with the combined texture. Conclusions We have presented a simple and mechanical way in which texture maps can be extracted from a series of photoraphs. By knowin the exact location and orientation of both the camera and the object, we are able to render the object in OpenGL so that the rendered scene matches up with the photoraph. One issue to be considered in the future is the effect of lihtin in the photoraphs. Since we know the eometry of the object, we can calculate the normal of the object very easily. For ideal diffuse surfaces, cancelin the lihtin effects should then be quite easy. Implementation Notes We used OpenGL with Nvidia s GeForce FX raphics accelerator to render the scene. The reason for usin a GeForce FX was because of its capability to render to a floatin point pixel buffer. Normally, each colour channel in the RGB format of the pixel uses only 8 bits, which isn t quite enouh for our purposes of encodin texture coordinates. However, with GeForce FX and floatin point buffer, we were able to use 32 bits per

11 channel, providin us with enouh precision to encode the texture coordinates for each pixel. References [Bouuet 03] Jean-Yves Bouuet, Camera Calibration Toolbox for Matlab, [Catmull 74] Ed Catmull, A Subdivision Alorithm for Computer Display of Curved Surfaces, PhD thesis, Dept. of CS, U. of Utah, Dec [FastScan] Polhemus, [Fernado 03] Randima Fernando and Mark J. Kilard, The C Tutorial: The Definitive Guide to Prorammable Real-Time Graphics, Addison-Wesley, [Haeberli 93] Paul Haeberli and Mark Seal, Texture Mappin as a Fundamental Drawin Primitive, Euroraphics, June [Heckbert 96] Paul Heckbert, Survey of Texture Mappin IEEE Computer Graphics and Applications, Nov [Vicon] Oxford Metric Systems,