Image Prediction Based on Kalman Filtering for Interactive Simulation Environments

Transcription

1 Image Prediction Based on Kalman Filtering for Interactive Simulation Environments M. SCHNITZLER, A. KUMMERT Department of Electrical and Information Engineering, Communication Theory University of Wuppertal Rainer-Gruenter-Str., 49 Wuppertal GERMANY Abstract: - In this paper a new method is presented to increase the frame rate of CGI (computer generated images) applications by means of image prediction. This method is based on Kalman filtering for the prediction of the new viewpoint position. The method processes two consecutive input images and the dedicated z-buffer content, which are computed by the graphics hardware, for the prediction of the next (future) image frame. The input data is transformed into an output image by 3D image warping. An enhanced z-buffer algorithm decides which pixels are visible. Key-Words: - Kalman filter, image prediction, interactive simulation Introduction The field of computer-based image generation for visual interactive simulation environments (virtual reality) is of great interest in different application areas like architecture, town planing, entertainment, or training by driving simulators. The algorithms which are used today require special graphics hardware for the computation of all pixels of an image. They do not take into account, that successive images in a sequence are related to each other. bviously, the computational load of the graphics hardware is proportional to the frame rate. A high frame rate is desirable to avoid flickering. Frame rate doubling leads to nearly twice the hardware expense and thus to enormous costs. Hence, image interpolation and prediction methods are of interest for achieving higher frame rates, without raising significantly costs. The major drawback of image interpolation methods is the additional latency they produce which can be avoided by applying image prediction techniques. n the other hand, image interpolation methods are performing better with respect to exposure areas compared to image prediction. The main reason for the better results are due to the additional knowledge (informatio about exposure and occlusion areas. I I Fig.: Time flow of interpolation and prediction. = original sequence, I = interpolated sequence, P = predicted sequence Viewpoint Prediction The knowledge about future viewpoint positions is very important for a good prediction quality. The viewpoint prediction is done by a so-called Kalman filter. The latter is efficiently realised due to its recursive description formula. The motion model takes into account the velocity and the acceleration of the viewpoint position.. Kalman filter Kalman filtering allows the prediction of the viewpoint position under system uncertainties. It is tolerant against measurement errors in the P P

2 corrector step. The measurements and the system states are modelled stochastically. The system state equations of the Kalman-filter can be formulated as w ( n + ) = Aw( + (, () o ( = Cw( + -(. () The first equation describes the motion model of the viewpoint position. w( is the current state vector of the system. The new state vector w ( n +) can be estimated via the system matrix A. The uncertainties of the system are modelled by (. The second equation determines the filter output by means of the matrix C and the system state vector w (. -( describes the measurement uncertainties in a stochastic way. The estimation ŵ of the state vector w is computed in the so-called prediction step via and w ˆ ( n n ) = Awˆ ( n n ), (3) T P ( n n ) = AP( n n ) A + Q, (4) o ( = Cwˆ ( n, (5) where the argument ( n n ) symbolizes an estimate for the time step n taking into account measurement data collected up to time step n. In equation (3) and (4) the new state vector and the system covariance matrix P are estimated, respectively, whereas the system uncertainties are characterised by the covariance matrix Q. The correction step is done by the following three equations K = P( R T T n n ) C [ CP( n n ) C + ], (6) w ˆ ( n = wˆ ( n + K[ o( Cwˆ ( n n )], (7) P ( n = P( n n ) + KCP( n n ). (8) The Kalman gain K is defined by equation (6). The system uncertainties and the measurement uncertainties influence via covariance matrix P and matrix R, respectively, the Kalman gain. The matrix R represents the covariance of the measurement noise.. Motion model The motion model for viewpoint estimation is based on velocity and acceleration of the viewpoint position. In the continuous-time case differential equations are used to describe the problem. In the discrete-time model difference equations have to be solved. The relationship between the position s ( and the velocity v ( can be described by t s( = v(ϑ) dϑ (9) in the continuous-time case, and for the velocity v ( and the acceleration a ( we have t v( = a(ϑ) dϑ. (0) The position at time t + T as function of the position at time t is defined as s ( t + T ) = s( + Tv(, () if the velocity is constant during time interval T. An improved modelling takes into account acceleration a ( by s ( t + T ) = s( + Tv( + T a(. () The last term always contains the uncertainty of the model. If the acceleration a ( is changing in the time interval T, the estimation would be degraded. 3 3-D Image Warping The output image I is calculated by means of 3D image warping. The parameters for the warping equations can be determined from the known viewpoint positions c 0 and c of the available images I 0 and I, the estimated viewpoint position c, and the used camera model. In our case a pine hole camera model is used. The pine hole camera can be described by the projection matrix P as follows width 0 height P = 0. (3) 0 0 f

3 The parameter f represents the distance between the projection centre and the projection plane. If only the field of view fov is given, the projection matrix is defined as width 0 height P = 0. (4) width fov 0 0 cot The warping equation itself can be formulated as c z u = P P z u + P ( c ). (5) u and u are the input pixel coordinate vector and the output coordinate vector, respectively, of the input and output images x u = y, x u = y. (6) z and z are the input and output image z-buffer values at the position u and u, respectively. Please notice that at this step only images I and I are considered. The third image I 0 will be taken into account later on in section 4. If we define w w w3 P P = w w w3 (7) w3 w3 w33 and w4 P ( c c) = w4, (8) w34 we obtain after some algebraic transformations the new pixel coordinates x, ) as ( y wx + w y + w3 + w4z = w3x + w3 y + w34z x, (9) The last equations enable to calculate the new pixel position and fill them with the RGB and z-buffer values. The z-buffer is important for visibility during warping and the following combination step. The new z-buffer content is given by w 34 z = w3x + w3y +. () z z 4 Combination A single image does not contain enough information to describe the whole 3D scenario. That is the reason why 3D image warper produce exposure artefacts. We need at least two consecutive images I 0 and I to avoid or reduce this effects. In other words, transformations (5) to () are performed twice. At first on basis of input image I 0 producing a first output image I /0. At second, input image I is used to compute a second output image I /. Finally, the actual output image I is computed on basis of I /0 and I /0 in the combine step where the z-buffer content of I / and I /0 plays a key role. The image information associated with the smaller z-value is used. If both input images have the same occlusion areas, a different approach has to be used. This areas are filled with information from the neighbour pixels by using a weighted sum. Fig. : Exposure artefacts generated by the warping stage are shown as black regions. wx + w y + w3 + w4z = w3x + w3 y + w34z y. (0)

4 5 Results PSNR[dB] = 0log NM N M i= j= 55 ( x [ i, j] x [ i, j] ) pic ref E ( x [ i, j] ) M ( x [ i, ) N M SMSE = M pic ref j] NM i= j= 3 3 Table : Definition of objective (PSNR) and subjective (SMSE) assessment criteria. The quality of the generated images is the most important aspect. Hence, images had been tested subjectively and objectively. Subjective means that a group of people has to look at the new images and has to judge the quality. The objective assessment uses mathematically formulated error or quality measures. bjective results have the advantage that the method can be easily compared with other methods. However, a good correlation between the Fig. 3: An example for an RGB-image and the corresponding z-buffer content. objective results and the subjective perception is not always available. The objective tests are based on the PSNR (Peak Signal to Noise Ratio) and the SMSE (Subjective Mean Square Error) which are defined in table. For a correct interpretation it is important to take into account, that an algorithm performs better if the corresponding PSNR value is high and the SMSE value is low. 3D image warping was tested with an image sequence having a resolution of pixels. Fig. 3 shows a sample of the image sequence and the corresponding z-buffer content. For objective tests the original testsequence was sub-sampled from 60 fps to 30 fps to generate reference images for the evaluation of the prediction algorithm. 6 Conclusion In this paper a new image prediction method for CGI systems has been presented. The use of Kalman filter for the prediction of the upcoming viewpoint is leading to good results. The image prediction using 3D image warping nearly reaches the same good results as a comparable interpolation method [6] (in spite of the fact that prediction algorithms have less information available than interpolation techniques). n the other hand, prediction methods do not add additional latency to the overall performance of the CGI system. The presented method is a profound basis for further research and development in the field of image prediction.

5 References: [] J. et al Gomes, Warping and Morphing of Graphical bjects, Morgan Kaufmann Publishers, 999. [] R. E. Kalman, A new approach to linear filtering and prediction theory, Journal of Basic Engineering, Vol. 83D, No., pp [3] W. Mark, Post-Rendering 3D Image Warping: Visibility, Reconstruction and Performance for Depth-Image Warping, Dissertation, University of North Carolina at Chapel Hill, 999. [4] L. McMillan, An image-based approach to three-dimensional computer graphics, Dissertation, University of North Carolina at Chapel Hill, 997. [5] G. Wohlberg, Digital image warping, IEEE Computer Society Press, 99. [6] W. Zeise, Zwischenbildinterpolation zur Erhöhung der Bildgenerierrate in visuellen interaktiven Simulationsumgebungen, Dissertation, University of Wuppertal, 999.