Motion based 3D Target Tracking with Interacting Multiple Linear Dynamic Models

Transcription

1 Motion based 3D Target Tracking with Interacting Multiple Linear Dynamic Models Zhen Jia and Arjuna Balasuriya School o EEE, Nanyang Technological University, Singapore jiazhen@pmail.ntu.edu.sg, earjuna@ntu.edu.sg Subhash Challa Faculty o Engineering, The University o Technology, Sydney, Australia schalla@eng.uts.edu.au Abstract In this paper, an algorithm is proposed or vision-based object identiication and tracking by autonomous vehicles. In order to estimate the speed o the tracking object, this algorithm uses inormation captured by on-board sensors such as camera and inertial sensors. To ormulate the tracking algorithm it is necessary to use a proper model which describes the dynamics o the tracking object. However due to complex nature o the moving object, it is necessary to have dierent dynamic models. Here, several simple and basic linear dynamic models are combined to approximate unpredictable, complex dynamics o the moving target. With these basic linear dynamic models, a detailed description o the three dimensional (3D) target tracking scheme using an interacting multiple model (IMM)alongwithanExtendedKalmanFilteringispresented. Theinal state o the target is estimated as a weighted combination o the outputs rom each dierent dynamic model. Perormance o the proposed interacting multiple dynamic model tracking algorithm is demonstrated through experimental results. 1 Introduction Vision-based target tracking is an active research area and has many applications. This paper ocuses on applications o target tracking by autonomous vehicle navigation. Intelligent transportation systems play a vital role in the society. Target tracking schemes aid the intelligent navigation system by providing inormation about the dynamic behaviour o the moving objects in the visinity o the vehicle[3]. The proposed vision system uses optical Flow as the main source o inormation to identiy the target and its motion. Optical low vectors relect the relative motion o a target in the image stream. Past research [7] show that tracking can be achieved by using optical low vectors with other visual cues. In this paper, optical low vectors are used with the motion data to estimate the 3D velocity o the target and used with other visual cues to identiy the region o interest (ROI) in the image. Target tracking is a challenging problem due to the unpredictable dynamic nature o the tracking object. It is diicult to model these non-linear dynamics with a single set o equations. Tracking o such targets thus alls into the area o adaptive state BMVC 24 doi:1.5244/c.18.29

2 estimation (multiple models)[6]. According to Bar-Shalom [2] the Interacting Multiple Model (IMM) perorms signiicantly better than other multiple model methods. The IMM algorithm computes the state o the object under each possible model using several ilters. Each ilter uses a dierent combination o the previous conditions estimated (mixed initial condition). The mixed initial condition o IMM can reduce the estimation error by adjusting the initial condition or each estimates. IMM is also aster in computation than other multiple model algorithms, or example IMM only requires r ilters to operate in parallel while the generalized pseudo-bayesian(gpb) multiple models algorithm requires r or r 2 to operate in parallel. Furthermore, the IMM algorithm is able to keep the estimation error less than the raw measurement error and provides a signiicant improvement (noise reduction). Thus, in this paper IMM algorithm is deployed to use several simple and basis linear dynamic models [4] or dynamic target tracking. In order to improve tracking perormance, the Kalman Filter is combined with weighted models or state estimation. Figure 1: Proposed Tracking System The proposed tracking system is described schematically in Fig.1. Here, clustered motion vector ields and color inormation in the image are used together to identiy the region o interest (Section 2). Next, optical low vectors are used with visual depth inormation and are combined with the kinematic model o the moving camera system to estimate the 3D position and velocity o the target(section 3). In the inal step, the target s dynamic eatures represented by simple linear dynamic models by the IMM are used in the Extended Kalman Filter or tracking(section 4). 2 Target Identiication 2.1 Target 3D Visual Feature Extraction In this paper, Horn s gradient-based iterative algorithm is modiied or optical low estimation due to its ast computational speed and accuracy[1][5]. Since the cameras are moving, the background introduces optical low vectors together with the motion o the target. Assuming that the moving target is a rigid object with uniorm motions, the region o the target can be separated rom the background using the motion vector ield clusters through template matching[7]. Here, K-means algorithm is used to segment the optical low vector ields into several clusters

3 to identiy dierent motion ields in the image. With clustered optical low vector ields a template matching technique is used to extract the ROI rom the background motions. The ROI s coordinates are considered as 2D coordinates o the target in the image. Here, the initial template is obtained manually rom the clustered optical low motion ields. The extraction o the ROI is carried out using target s motion ields. This makes the tracking system robust to track unknown targets independent o its other optical properties such as shape, size etc.. During the tracking process, the template is updated in real-time with the estimated ROI. The visual depth inormation is obtained through stereo image pairs. Assuming the camera axis are parallel and separated by b 1 and the ocal length o the camera s to be, visualdepthz canbeestimatedbasedondisparity[5]givenbyz = b 1 x. l x r Here, coordinate dierence o the same point in let and right images (x l x r)is known as disparity d, which is estimated by the sum o absolute RGB dierence (SAD) algorithm [8]. 2.2 Target 3D Dynamic Inormation Estimation Ater identiying 3D visual eatures o the target, this section looks at the usion algorithm or estimating the 3D velocities and position o the target. (a) (b) Figure 2: Experiment Setup with Target World Coordinate Velocity Estimation Procedure First, assume the cameras mounted on an autonomous vehicle moving in the environment as shown in Fig.2(a). The target velocity can be obtained rom the detailed image processing process depicted in Fig.2(b). First the motion o the target in the image stream is obtained as the optical low vectors V O.F..ThenV O.F. is transormed to estimate the target image velocity V IM.NextV IM is transormed into the camera rame to get the target velocity relative to the cameras V IM C. The target velocity relative to the cameras in the world coordinate V IM W can be estimated rom V IM C.At thesametimethevelocityothemovingcameraintheworldcoordinatev C W can be estimated rom the kinematics model o the autonomous vehicle. Finally the target s

4 velocity in world coordinate V W T can be written as: V W T = V W IM + V W C (1) This equation shows how the image visual eatures can be used with the cameras motion parameters or the target velocity estimation. In this paper, a simple two wheeled kinematic model o an autonomous vehicle is used to estimate the camera s velocity. Two tachometer sensors measure the wheels velocity. Since the vehicle only has the velocity in the X and Z directions as shown in Fig.2(a), the camera axis only has the Y axis rotation. Thus based on the process in Fig.2(b), the target velocity in the world coordinate V T W can be estimated as V W 1 T x ( b 1 d x + S VT W d 2 x V O.F.x b 1 d ) cos( k u 1 k u 2 t) y = VT W 1 ( b 1 d x + S x V O.F.x b 1 d ) sin( k u 1 k u 2 t) z d 2 +b 1 d sin( k u 1 k u (k u1+k u2)cos(k u1 k u2 2 d 2 t) t)+ 2 1 ( b 1 d y + S d 2 y V O.F.y b 1 d 1 ) b 1 d sin( k u 1 k u (k u1+k u2)sin(k u1 k u2 2 d 2 t) t)+ 2 Here (x,y) is the target 2D image coordinate, which is obtained rom the previous ROI identiication. S represent the eective sizes o the pixel in the image which transorm the optical low vectors V O.F. to the image coordinate to get V IM. (u 1,u 2 )arethe tachometer output voltages, k is the linear parameter between the tachometer output voltages and the angular speeds o the wheels, which gives v1=k u 1 and v2=k u 2. b is the distance between two wheels. t is the time since the camera begins to move. The target in the world coordinate position P W can be also obtained rom (3). (2) P W = cos( k u 1 k u 2 sin( k u 1 k u 2 t) Sx x Z sin( k u 1 k u 2 t) Z + t (k u 1+k u 2 )cos( k u 1 k u 2 2 S y y Z t) Sx x Z +cos( k u 1 k u 2 t) Z + t (k u 1+k u 2 )sin( k u 1 k u 2 2 t) t) (3) Equations 2 and 3 show the data usion o target optical low vectors (V O.F.x,V O.F.y ), visual disparity d, image coordinate (x,y) and motion sensor data (u 1,u 2 )toestimate the target 3D velocity and position in the world coordinate. 3 IMM Target Tracking System At this stage an Extended Kalman Filter is used or the tracking o target position and velocity. Here, a rigid body with M points o motion is considered. Then the motion state equation can be written as Eq.(4). P V k+1 = Φ r P V k + n v I k, Z = C(h)k, h= {V O.F.x ;V O.F.y ;u 1 ;u 2 ;d;x;y} (4) where Φ r denotes the target dynamic models, P =(P x,p y,p z ) denotes the 3D position o the tracked point, V =(V x,v y,v z ) its velocity. The resulting position and velocity

5 measurement vector is Z. Here C k is the target position and velocity measurement unction rom Eq.(2) and Eq.(3). It is worth noting here that this measurement vector h comprise target visual eatures and vehicle motion sensor data as its dierent components, which provide a data usion scheme or the target tracking. Thecoordinateothetrackedpointinthe2Dimagep can be obtained by (5) P x = Π( P P k )=Π(Π 1 (p k 1,(P z ) k )) (5) y k Where Π 1 denotes the inverse transormation o a 2D image point p and its depth P z onto the 3D world coordinate space. 3.1 Fusion o Linear Dynamic Models or Kalman Filters Several simple and basic dynamic models [4] are available or Extended Kalman Filtering. 1. Driting Points Model. Driting point model is represented by Φ i = Id (6) That is the new position is same as the old position plus some Gaussian noise. This dynamic model is commonly used or objects with known dynamic behaviour. 2. Constant Velocity Model. p gives the position and v the velocity o a point moving with constant velocity. The constant velocity model is given in Eq.(7). x = p v Id (t)id, Φ i = Id 3. Constant Acceleration Model. a is the acceleration o a point moving with constant acceleration. Then the constant acceleration model is given in Eq.(8). x = p v a, Φ i = Id (t)id Id (t)id Id 4. Periodic Motion Model. It is assumed that a point is moving on a line with a periodic movement. Then stack the position and velocity into a vector u =(p,v) and with the time interval t, the motion model can be obtained with a orward Euler method as shown in Eq.(9). u i = u i 1 + t du dt = u i 1 + tsu i 1 = 1 t t 1 (7) (8) u i 1 (9) Now with the above 4 linear dynamics models, one cycle o the IMM algorithm is described [2]. 3.2 IMM Approach First the prior probability that Φ j is correct (the system is in Model j) isshownin Eq.(1). P {Φ j Z } = µ j () (1)

6 where Z is the prior inormation and r j=1 µ j() = Calculation o the mixing probabilities (i, j = 1,..., r). The probability that mode Φ i was in eect at k 1giventhatΦ j is in eect at k conditioned on Z k 1 is Eq.(11). µ i j (k 1 k 1) = 1 c j p ij µ i (k 1), c j = r i=1 p ijµ i (k 1) (11) Here it is assumed that the model jump process is a Markov process (Markov chain) with known model transition probabilities p ij p ij = P {Φ(k)=Φ j Φ(k 1) = Φ i } (12) 2. Mixing (j =1,...,r). Starting with ˆx i (k 1 k 1) one computes the mixed initial condition or the ilter matched to Φ j (k) as ˆx j (k 1 k 1) = r ˆx i (k 1 k 1)µ i j (k 1 k 1) (13) i=1 The covariance corresponding to the above is P j (k 1 k 1) = r µ i j (k 1 k 1){P i (k 1 k 1) + [ˆx i (k 1 k 1) i=1 ˆx i (k 1 k 1)][ˆx i (k 1 k 1) ˆx j (k 1 k 1)] } (14) 3. Model-matched iltering (j = 1,..., r). The estimate Eq.(13) and covariance Eq.(14) are used as input to the ilter matched to Φ j (k), which uses z(k) toyield ˆx j (k k) andp j (k k). Then the likelihood unctions corresponding to the r ilters Λ j (k)=p[z(k) Φ j (k),z k 1 ] (15) are computed using the mixed initial condition Eq.(13) and the associated covariance Eq.(14) as: Λ j (k)=p[z(k) Φ j (k), ˆx j (k 1 k 1),P j (k 1 k 1)] (16) 4. Model probability update (j =1,...,r). This is done as ollows: µ j (k)= 1 c Λ j(k) c j, c= r j=1 Λ j(k) c j (17) where c j is the expression rom Eq.(11). 5. Estimate and Covariance combination. Combination o the model-conditioned estimates and covariances is done according to the mixture equations P (k k)= ˆx(k k)= r ˆx j (k k)µ j (k) (18) j=1 r µ j (k){p j (k k)+[ˆx j (k k) ˆx(k k)][ˆx j (k k) ˆx(k k)] } (19) j=1

7 3.3 IMM For Kalman Filter Estimation According to the IMM, several simple linear dynamic models Φ introduced in the previous sections are used as the dierent state transition matrix (Eq.(4)) or the Kalman Filtering. In the IMM algorithm, the likelihood o each mode-conditioned estimates Λ j (k) in Eq.(16) are generated with the Kalman Filter measurement residue residure j (k) residue(k)=z(k) C(k) x(k k 1) (2) where z is the measurement value, C is the measurement matrix and x(k k 1) is the Kalman Filter predict rom k 1tok. Then Eq.(16) can be written as Λ j (k)=n[residue j (k);,s j (k)] (21) where S j (k) is the residue variance which can be estimated rom Eq.(22) S j (k)=c(k) P j (k k 1) C(k) T + R(k) (22) here R(k) is the covariance o the measurement noise. Using the above recursive IMM algorithm, the interacting multiple linear dynamic model or Kalman Filter can be derived. 4 Experimental Results A moving platorm is designed with two sensors(tachometers and CCD Cameras) to test the tracking algorithm. Tachometer measures the wheel speed o the platorm, which is then used to estimate the speed o the camera with the aid o a kinematic model. Two color CCD cameras (ocus length = 5.4mm)are connected with the Hitachi IP55 image processing cards or the image capturing and processing. The image size is 256x22 with a capturing rate 15/s (a) (b) (c) (d) (e) Figure 3: Moving Target with Moving Background As shown in Fig.3 a blue target is moving in the image sequence. Here the cameras are also moving, as a result background is changing rom rame to rame. Figures 3(d) and 3(e) show the estimated optical low ields. The 1% dense optical low ield relects the target motion together with the background motion. Fig.3(e) is the image with the enlarged optical low vectors.

8 Next the K-means clustering is implemented to segment the low vector ields into several regions(fig.4). The K-means algorithm is sensitive to slight movements in the (a) (b) (c) (d) (e) Figure 4: Motion Field Segmentation Result image sequences. Thereore, it is necessary to eliminate the smaller motion vectors by deining a suitable threshold. Figures 4(a) and 4(b) are the segmented images ater thresholding. Figures 4(c) and 4(d) show the K-means clustering result. Here, dierent colors represent dierent motion cluster regions. It is clear rom the clusters that motion regions have the same color, thus it can be easily identiied by a template matching technique. In this experiment the matching template is updated in real time through the target depth inormation. Fig.4(e) shows the template matching value. The smallest value indicates the region o interest. 6 Camera Platorm Moving Trajectory in the 2D Plane Moving Platrom Postion in X Z Plain 15 Target Velocity Measurement Target Estimated Velocity Target World Coordiante Velocity 5 Target World Coordinate Velocity Target Estimated Velocity in World Coordinate (Y) Y Velocity In World Coordinate (X) Velocity In Word Coordinate (Y and Z) Target Estimated Velocity in World Coordinate (Z) X (a) (b) (c) Figure 5: Target Velocity Estimation Result Figure 5(a) shows the moving trajectory o the moving platorm in the 2D plain,

9 which is estimated based on the two wheels tachometer data and the kinematic model o the autonomous vehicle. This trajectory shows that the cameras are moving toward the targets. Figure 5(b) shows the target velocity in the X direction. The estimated velocity is quite close to the real target velocity. Figure 5(c) shows the target Y and Z direction velocity. In this experiment, the target has little movement in these direction, thus the values are almost zero. (a) (b) (c) (d) (e) Figure 6: Target 3D Tracking Result with IMM Figure 6 shows the 3D target tracking results. Here the rectangle in the images is obtained through the inverse pinhole camera projection o the estimated target world coordinate position rom IMM based Extended Kalman Filtering (Eq.(5)). With the updating template, the size o the rectangle is scaled due to the 3D depth changes. From these images, the rectangle can accurately it the target region which shows that the tracking system perorms well. As shown in Fig.7 the interacting multiple linear dynamics algorithm perorms well orestimatingthevelocityandpositionothetargetbasedonanextendedkalman Filter. The errors rom the IMM estimation are relatively smaller. 5 Conclusion In this paper, a novel target-tracking scheme is proposed. Optical motion ields are used as the major source o inormation or tracking a moving object by autonomous vehicles. According to Forsyth[4] several simple and basic linear dynamic models are combined using an IMM algorithm to provide an accurate approximate model or the dynamics o the tracking target. Through experiments, the proposed tracking scheme demonstrated satisactory results or dynamic target tracking. In conclusion it can be stated that the proposed target tracking scheme is suitable or target identiication and tracking by autonomous vehicles. Reerences [1] J.L.Barron, D.J.Fleet, S.S.Beauchemin, T.A.Burkitt, Perormance o optical low techniques Proceedings o IEEE Computer Society Conerence on Computer Vision and Pattern Recognition, pp , [2] Y.Bar-Shalom, X.-R. Li, Estimation and Tracking: Principles, Techniques, and Sotware Artech House Boston MA, USA, 1993.

10 4 Position Estimation Result 4 Position Estimation Error rom Dierent Models Position Error Measurement IMM Driting Point Constant Velocity Constant Acceleration Periodic Motion 6 8 Driting Point Constant Velocity Constant Acceleration Periodic Motion IMM (a) (b) 1 8 Velocity Estimation Result 1 Velocity Estimation Error rom Dierent Models Driting Point Constant Velocity Constant Acceleration Periodic Motion IMM Velocity 2 Error 2 Measurement Driting Point Constant Velocity Constant Acceleration Periodic Motion IMM (c) (d) Figure 7: Estimation Result rom Dierent Models [3] G.N.DeSouza, A.C.Kak, Vision or Mobile Robot Navigation: A Survey IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol.24, No.2, pp , 22. [4] D.A.Forsyth, J.Ponce, Computer Vision: A Modern Approach (1st edition) Prentice Hall, pp ; pp , 22. [5] B.K.P.Horn, Robot Vision MIT Press, pp.28-29; pp.41-43, [6] J.Evans, R.Evans, Image-enhanced multiple model tracking Automatica, [7] Zhen Jia, A.Balasuriya, S.Challa, Vision data usion or target tracking Proceedings o the Sixth International Conerence o Inormation Fusion, Volume:1, Pages: , 23 [8] K.Muhlmann, D.Maier, R.Hesser, R.Manner, Calculating dense disparity maps rom color stereo images, an eicient implementation Proceedings o the IEEE Workshop on Stereo and Multi-Baseline Vision, pp.3-36, 9-1 Dec.21.