FPGA IMPLEMENTATION OF ADAPTIVE TEMPORAL KALMAN FILTER FOR REAL TIME VIDEO FILTERING March 15, 1999

Transcription

1 FPGA IMPLEMENTATION OF ADAPTIVE TEMPORAL KALMAN FILTER FOR REAL TIME VIDEO FILTERING March 15, 1999 Robert D. Turney +, Ali M. Reza, and Justin G. R. Dela + CORE Solutions Group, Xilinx San Jose, CA , USA Department of Electrical Engineering and Computer Science, UWM Milaukee, Wisconsin , USA ABSTRACT Filtering noise in real-time image sequences is required in some applications like medical imaging. The optimum approach in this case is in the form of adaptie 3-D spatialtemporal filter, hich is generally ery complex and prohibitie for real-time implementation. Independent processing of the image sequences, in spatial and temporal domains can resole some of these implementation difficulties. Some of the existing spatial filters can easily be modified for real-time implementation. Adaptie temporal filters, hoeer, are more inoled. In this paper, an adaptie temporal filter is proposed that lend itself to hardare implementation for real-time temporal processing of image sequences. The proposed algorithm is based on adaptie Kalman filtering hich is relatiely simple and effectie in its performance. Adaptation in this case is ith respect to motion in the image sequence as ell as ariation of noise statistics. An efficient hardare implementation of this algorithm, based on FPGA technology, is proposed. 1. INTRODUCTION The problem that is addressed in this study is the degradation of image sequences due to noise that is usually incurred during image acquisition process. Design of the optimum noise filter can only be based on the assumption that the image sequence and noise are stationary and their statistics are knon. Statistics of image sequences and noise can be estimated if these signals are really stationary. In practice, hoeer, this assumption is not alid and only adaptie algorithm can be utilized. General adaptie 3-D spatial-temporal filters are ery complex and prohibitie for real-time implementation[1]. A more practical approach ould be to process each frame independently by using an adaptie spatial filter and to filter each pixel in time domain by using an adaptie temporal algorithm. Some of the existing adaptie and/or robust (nonlinear) spatial filters can easily be modified for real-time implementation. Adaptie temporal filters, hoeer, are more inoled and require more hardare resources. In this case, application of a simple lo-pass filter results into a peculiar lagging or ghosting effect that is due to the filtering of high frequency components (motion) in the temporal domain. Adaptation in time therefore not only requires estimation of the noise statistics but also a means to control the leel of filtering hen there is a motion associated to the corresponding pixel in the image sequence. The general problem of adaptie filters is discussed in[] and in particular some releant algorithms for adaptie spatial-temporal noise filtering for ideo images are reieed in [1]. Conentional approach in designing optimum filters is to use minimization of the mean-squareerror (MSE) hich results in Wiener filter for stationary signals and recursie Kalman filters for signals ith timearying statistics[3]. In this case, due to the real-time processing requirement and the fact that there is no linear phase restriction on temporal filters, recursie filters are the only feasible solutions. It is therefore of interest to deelop an adaptie Kalman filter that can adjust its parameters based on the ariations in the noise statistics and detection of motions in the image sequence. In the next section, a relatiely simple but effectie adaptie Kalman filtering algorithm for temporal filtering of image sequences is proposed hich addresses some of the desired implementation requirements.. ADAPTIVE TEMPORAL FILTER One important factor for adaptie filtering of ideo images is to find an algorithm that has acceptable performance and can lend itself to a real-time implementation. Wiener filter approach and general discrete recursie Kalman filtering both hae same steady-state solution if the signal and noise are stationary. In the case of time arying statistics, Kalman filter ould be the suitable approach. In practice, hoeer, not only the signal and noise are non-stationary but also their statistics are unknon. Adaptie filters are based on dynamically adjusting the parameters of the supposedly optimum filter based on the estimates of the unknon parameters. Adaptie Kalman filter can be based on an on-line estimation of motion as ell as the signal and noise statistics aailable data. Let x ( represent a pixel grayscale on frame k. The ideal noise-free pixel alue is represented by s ( hich is assumed to be a first-order AR model. This is a more realistic and simple model that is usually used to represent the temporal behaior of pixels in ideo signals[1]. Under

2 this assumption, the process and measurement equations are: 1. The process model is s ( k + 1) = as( + (, in hich a is a constant that depends on the signal statistics and ( is the process noise (assumed to be a hite independent zero mean Gaussian random process ith ariance of ).. The measurement signal is x ( = s( + ( in hich ( is the independent additie zero mean Gaussian hite noise ith ariance of. Here it is implicitly assumed that the noise and signal are stationary random processes that are fully determined by their second-order statistics. Although this assumption is not true in practice, but for local statistics, it can safely be used ith some approximation. The recursie Kalman filter is deeloped based on the folloing definitions: 1. The filter output is represented by y ( hich is the estimate of the signal, at time k.. Variance of the estimation error is theoretically defined by ( = E [ y( s( ], hich is initially unknon. { } 3. Kalman filter gain is represented by Κ (. The oerall Kalman filter algorithm is then gien as follos: Algorithm A ( Let y ( 1) = 0, and 1) =, and start by setting k = 0 LOOP: for k do the folloing operations: a ( + Κ( = a ( + + y( = Κ(? + a? 1 Κ(? y( k 1 [ ] ) [ 1 Κ( ] ( ( = a + Increment k, k R k + 1, and go back to LOOP END In this algorithm, there are seeral parameters, hich are unknon in practice. These parameters are,,, and a. The algorithm can be made adaptie by properly estimating these parameters, using local information. Estimation of these parameters can be based on optimization of a criterion function like minimum meansquare error (MMSE). No matter hich estimation technique is used, the quality or reliability of the estimates ill alays depend on the length of data used to estimate them. Based on the aforementioned assumptions, the folloing instantaneous estimates can be used to achiee fast and simple implementation: 1. Parameter a, defined by E{ k 1) } E{ x ( }, can be estimated by using a ˆ = 1( x ( + x ( ). For stability reasons it is suggested to use some a priori information about the signal and keep this parameter constant.. Simple estimates of = E [ ay ˆ ( ], as { } { y( ay ˆ ( 1 } ell as = E [ k )], are calculated, in turn, by ˆ = [ ay ˆ ( k 1 ) ] ˆ [ y( ay ˆ ( ] = Κ ˆ =. and The adantage of the adaptie algorithm is that the Kalman gain starts ith relatiely large alues, and gradually decreases hen the temporal signal is stationary. This is a desired property, hich produces more noise filtering as time adances. Hoeer, hen there is a motion, estimates of the noise and process ariances, ˆ and ˆ, increase hich results in less filtering of the signal. This ill reduce the noise filtering so that it can better follo the motion ith minimal lagging effect. In this adaptation, there is still no explicit ay of estimating the motion. One approach to motion estimation is to compare to consecutie temporal samples and use their magnitude difference to infer existence or nonexistence of motion. This estimation can be conducted by using a statistical test based on the assumption of hite Gaussian noise[4]. For example, hen there is a sudden change in the signal, the difference beteen ay ( and x (, ith a gien confidence leel, goes beyond its statistical ariation. In this case, a simple test can be used to estimate sudden changes (motion) in the signal. Assuming that represents the ariance of the aforementioned differences, then based on Gaussian noise distribution, it can be said that a motion is present if γ ay(? Γ. If the test is positie, then the = gain calculation in the Kalman filter can be reinitiated by assuming =. The ne gain alue significantly ( reduces the lagging effect hile improes the noise filtering. Other modifications can also be utilized. For example, e can set the Kalman gain equal to 1 and reinitiate the filtering right after the motion. Or set both ( and equal to and initiate the Kalman gain to Κ( ) = ( a + 1)( a + ) k right after the motion. The oerall algorithm is represented is presented as follos:

3 Algorithm B k = Let y ( 1) = 0, =, and =, and start the loop by setting 0 LOOP: for k do the folloing operations: + Κ = ; + + y ( = Κ? + 1 Κ y( k 1 ( ) ) ; or [ 1) ] y ( = y( + Κ k ; if else D =? Γ, = ; = ; = Κ ; R ( 1 Κ) + ; end-if Increment k, k R k + 1, and go back to LOOP END Selection of the threshold Γ is ery important. In this case, it can be shon that γ has χ distribution ith one degree of freedom[4]. For the purpose of statistical test of hypothesis, different alues of Γ, for different confidence leel, are tabulated in Table I. Proper use of these alues ill result in the same confidence leel as stated in [4]. In other ords, if the noise is hite and independent from signal, using 95% confidence leel ill produce only 5% error, in aerage, hen it is applied in motion detection. In Figure 1, the Kalman filter ith motion detection is compared ith the standard nonadaptie optimum Kalman filter by simulating a sudden change in the signal to represent motion. In this case the SNR is set to 0dB. In this simulation, the confidence leel is kept at the highest leel; namely 99.9% ( Γ = 3. 9 ). Table I: Threshold alues for Motion detection Confidence Leel Γ Γ 99.90% % % % % In any gien application, the user should proide a reasonable alue for based on his or her a priori information. As it is eident, higher confidence leel results in a better performance. It should be emphasized that this approach is sensitie to impulsie noise and assigns an impulsie noise to a motion. Using local spatial filtering can minimize the problem of impulsie noise. In other ords, instead of using original frames, e can use spatially filtered ersion of that frame sequence just for the purpose of motion detection. The problem of impulsie noise can also be resoled by using some form of nonlinear filter Noisy signal Kalman filter ith motion detection Non-adaptie Figure 1- The result of Kalman filtering, using motion detection, in comparison ith non-adaptie filtering hen SNR is 0dB and Γ= FPGA IMPLEMENTATION The implementation of the adaptie Kalman filter is performed ith standard memory components and a Xilinx FPGA. Memory components are used for frame and parameter buffers hile the FPGA is used for pixel and parameter calculations. The system architecture shon in Figure illustrates three input buffers holding y (,,, and a Xilinx implementation to calculate updated alues for these buffers in addition to generating the proper output y (. The system can easily operate at a 66 Mhz clock enabling 104x Frames/s operation. System clock rates of 80 Mhz and 100 Mhz can also be achieed for more aggressie system bandidth requirements. The pixel calculation data path for y( is straight forard once the K parameter is calculated. The parallel pipelined structure used for the sample processing algorithm inoles pre-subtraction, to input ariable multiplier and post-addition as shon in Figure 3. After the initial latency a sample y( is output eery system clock. Hardare resources to perform this data path operation is approximately 458 Logic Cells. The parameter calculation is more inoled and also requires a parallel pipelined structure for sample processing since each pixel in the frame also has parameters and

4 . The algorithm requires pre-addition and diision for the K parameter calculation. To perform the comparison for the inequality e can normalize for and define D and Γ. y(k-1) X( XILINX FPGA Video Buffer Figure - System leel implementation for Kalman filtering. X( y(k-1) K - * + y( Figure 3- Pixel Calculation implementation Kalman filtering. Using a s complement function on x ( from the pixel calculation, e derie the necessary signal for the x1 mux parameter selection. The calculation of ne parameters inoles an adder, subtracter, ariable multiplier and loadable constant coefficient multiplier as shon in Figure 4. After initial latency a ne updated parameter is gien eery system clock. The pixel and parameter calculation blocks are latency synchronized such that K and are properly aligned. The output and parameters are aligned such that one memory controller can handle reads and rites to input buffers. Hardare for resources for the parameter calculation is approximately 1664 Logic Cells A/B K + * * x1 x1 -y(k-1) > Γ ` Figure 4- Parameter Calculation implementation for Kalman filtering. The total hardare resources for the FPGA is 1 Logic Cells alloing for a Xilinx XC4036XLA or XCV50 to be used for implementation of the Kalman filtering operation. Extensie use of Xilinx Smart-IP aailable in the COREGEN tool alloed for optimal performance and shortened design cycle. 4. CONCLUSION General problem of ideo image filtering has been discussed and some simulation results hae been presented. VLSI implementation of the deeloped algorithm, using Xilinx FPGA has been presented. It should be understood that if there is an impulsie noise in the image sequence, this Kalman filter algorithm should be used in conjunction ith pre-spatially non-linear filtered frames. 5. REFERENCES [1] J. C. Braileam, R.P. Kleihorst, S. Efstratiadis, A. K. Katsaggelos, and R.L. Lagendijk, Noise reduction filters for dynamic image sequences: A reie, Proceedings of The IEEE, ol. 83, no. 9, September [] S. Haykin, Adaptie Filter Theory, 3 rd Ed., Prentice Hall, Ne Jersey, [3] R. G. Bron and P. Y. C. Hang, Introduction to Random Signals and Applied Kalman Filtering, 3 rd Ed., John Wiley and Sons, Ne York, 1997.

5 [4] R. E. Kirk, Statistics, An Introduction, 3 rd Ed., Holt, Rinehart and Winston, Inc., Texas, 1990.