Video Data and Sonar Data: Real World Data Fusion Example

Transcription

1 14th International Conferene on Information Fusion Chiago, Illinois, USA, July 5-8, 2011 Video Data and Sonar Data: Real World Data Fusion Example David W. Krout Applied Physis Lab Greg Okopal Applied Physis Lab Evan Hanusa Eletrial Engineering Abstrat Reently a data set was olleted that inludes video data and imaging sonar data in the underwater environment. This data set provides a unique example of a data fusion problem that is not ommonly found in the underwater environment. This paper presents the fusion of data from a video amera and an imaging sonar, whih is then proessed by a target traking algorithm. Keywords: Traking, Video, Imaging Sonar. 1 Introdution This paper presents objet traking results using a likelihood fusion algorithm and a Joint Probabilisti Data Assoiation (JPDA) traker on a unique data set that inludes video and imaging sonar data. It is ommon to see video and sonar data ombined in a robotis appliation. For example, the Simultaneous Loalization and Map Building problem [1] has been well studied for Sonar sensors. However, it is rare to find this kind of data set in the underwater environment. This paper will disuss how a fusion/traking algorithms were applied to the data set. Setion 2 will summarize the sea trial where the data was olleted. Setion 3 and 4 will disuss the data proessing algorithms used to reate detetions from the different sensors. Setion 5 will review the mathematial basis for the Joint Probabilisti Data Assoiation algorithm. Setion 6 will show the traking results from the data set. Setion 7 will summarize the paper and disuss areas of ontinued work. 2 Sea Trial Desription The primary goal of the sea trail was to test the Detetion, Classifiation, and Loalization (DCL) ability of the BlueView ( imaging sonar on surfae and underwater objets. The test bed was mounted on an Applied Physis Laboratory/ (APL/UW) test vessel in Lake Washington (Seattle, WA) to ollet data on various surfae and underwater obstales. The olleted data was quite extensive and inluded several Figure 1: BlueView P450. segments of data olleted over 3 days. This paper will fous and a small subset. The test bed onsisted of a BlueView imaging sonar, a P450-45: 450 khz, 45 degree field of view, 20 degree vertial beam width, 2 degree azimuthal resolution, 4 inh range resolution sonar ( The P450, see Figure 1, was mounted to a braket that ould be lowered into the water off the stern of a Researh Vessel (R/V). A video amera (Sony HDR-HC7) was mounted on the top side of the braket. The braket allowed for the sensors to be pointed in various bearings on the fly, and both sensors were mounted so they were always pointing in the same diretion. The video data was olleted with the initial purpose of doumentation of the sea trial itself. However, it was soon disovered that the video data ombined with the imaging sonar data provided a unique data set for data fusion and was the motivation for this paper. One of the experiments during the sea trial was to see how well the sonar performed on a hannel buoy. A hannel buoy is used to mark a hannel for ship navigation. This setion ISIF 614

2 of the data was hosen beause the hannel buoy was piked up very well by the P450 and it was easy to see in the video data (the green buoy stands out quite niely). This paper desribes the first attempt to fuse and trak with this data set and in the future more diffiult targets/objets will be examined as well. The snippet of imaging sonar data that was proessed onsisted of approximately 100 pings (Pings ), whih was about 40 seonds long due to a ping rate of 3 Hz. The HD video was syned in time to math the imaging sonar ping rate. 3 Video Proessing The goal of the video proessing is to generate ontats from the HD video soure. To aomplish this, the algorithm transforms the video to a pereptually-motivated olor spae, learns a bakground model for eah pixel in eah dimension of the new olor spae, and then determines whether eah newly-observed pixel is a part of the bakground or a foreground objet. Foreground pixels are then joined via a two-pass onneted omponents algorithm, and ontats are generated for eah unique objet whih inlude desriptive data suh as the position and extent in the video s x-y plane. An example of the result of this proessing, from the frame of video orresponding to sonar ping 525, is shown in Figure 2. The red retangle represents the position and spatial extent of the reported ontat. Figure 3: S-plane of the onverted image for ping 525. Note that the green buoy is distint in this representation. where M is the running average, x and y are the oordinates in the video plane, n is the time index, I is the image (video) data, and α is the mixing parameter that ontrols how quikly new information will be inorporated into the bakground model. Additionally, a running average of the inter-frame differene is alulated as D(x, y; n) = (1 α)d(x, y; n 1) +α I(x, y; n) I(x, y; n 1) (2) We illustrate this bakground modeling in Figure 4 for a single hypothetial pixel. Figure 2: Video frame orresponding to ping 525. The ontat is indiated by a red retangle. For eah new frame of video, the frame is transformed to a new olor spae, in this ase HSV (hue, saturation, and value), whih is a pereptually-motivated ylindrial representation of a red-green-blue (RGB) olor model [2]. For this experiment, we foused on the S-plane beause the green hannel buoy was distint in this plane. This an be seen in Figure 3. In order to extrat foreground objets, the algorithm must learn a model of the bakground; we use the averaging bakground method [3]. For eah pixel and in eah olor dimension, the algorithm omputes a running average via M(x, y; n) = (1 α)m(x, y; n 1) + αi(x, y; n) (1) Figure 4: Learned bakground model for a hypothetial pixel represented by the green line. The running average and average inter-frame differene margins are represented by blue and red lines, respetively. For a unitary threshold (γ = 1), pixels whose values fall outside of the red lines would be onsidered foreground pixels. In order to detet foreground objets, the algorithm tests the pixels of a new frame for those that lie some fixed multiple of the average inter-frame differene from the running average: I(x, y; n) > M(x, y; n) + γd(x, y; n) (3) I(x, y; n) < M(x, y; n) γd(x, y; n) (4) 615

3 where γ is a user-adjustable threshold. If either Eq. (3) or Eq. (4) evaluates as true, then the pixel is onsidered to be a foreground pixel. In Figure 5, we show the foreground pixels for ping 525. Figure 5: Foreground pixels (shown in white) for ping 525. Figure 6: A sample frame for ping 525 from the BlueView P450. This is a sreen apture from the BlueView viewing software, and the range is zoomed into less that 45 meters to make the buoy easier to see. In the final stage of the video proessing algorithm, the foreground pixels are joined into objets via a two-pass onneted omponents labeling algorithm [4]. A ontat is reported for eah unique objet, and the ontat inludes both the position of the objet in the x-y plane of the video frame and the width and height of the objet as measured in pixels. 4 Imaging Sonar Proessing Similar to the video data, the goal of the imaging sonar proessing is to generate a list of ontats. A single ping from the imaging sonar an be seen in Figure 6, where the highlights from the hannel buoy an be learly seen. It should also be lear that there are no false alarms in this frame (no highlights other than the buoy). The hannel buoy highlights are well above any bakground noise. This low false alarm rate is harateristi of the pings for the data snippet seleted. Eah frame was first thresholded, and then proessed using a simple template mathing algorithm. Figure 7 shows the frame after thresholding and template mathing. Sine the objet in this dataset was very simple (i.e., a blob), this tehnique worked well. The objet was deteted in almost all the frames under the assumption of only one detetion per ping. This assumption will need to be removed in future studies, but for this snippet of the data the objet was easy to detet. The mean loation, width, height, and total area, were alulated for eah blob detetion and saved in a ontat list. A list of ontats was then sent to the fusion/traking algorithm along with a similar ontats list from the video data. 5 Joint Probabilisti Data Assoiation Filter JPDA [5] is an extension of PDA to allow for multiple targets. This desription of the JPDA algorithm assumes an underlying Kalman Filter. This an be modified to inlude Figure 7: An image of a frame for ping 525 from the Blue- View P450 after thresholding and template mathing. This is the full range bearing image. The atual range is from 0 to 156 meters and the bearing is from degrees to degrees. the extended Kalman Filter (EKF) or the unsented Kalman Filter(UKF), but an unmodified Kalman filter is shown below for ease of explanation. The state at time k is predited: x i (k k 1) = F x(k 1), P (k k 1) = F P (k 1)F T + Q, (5) where F is the motion model, P is the ovariane matrix, and Q is the proess noise. The residuals, ỹ ij, are alulated between gated ontats y j and the predited target loation: 616

4 ỹ ij = y j (k) Hx i (k k 1), where H is the measurement matrix. The distane, d ij, is alulated for all gated ontats, j, using the residual ovariane S i from the Kalman Filter. The distanes are sored using the Gaussian pdf assumption: d 2 ij = ỹ T ijs 1 i ỹ ij, g ij = e d2 ij /2 (2π) M/2 S i, (6) where g ij is the distane sore, and M is the spatial dimension (2 in this example). The probability of the data, Y k, given an assoiation, A, is alulated by: P (Y k A) = CN fa j:g ij>0 (g ij )P N d D (1 P D) Nm, (7) where C is the density of false ontats, N fa is the number of false ontats, is the normalizing onstant, P D is the probability of detetion, N d is the number of traks that have a ontat assoiated, and N m is the number of traks without an assoiated ontat. The assoiation weight, β ij is the sum of the probabilities of all the assoiation, M ij, that assoiate ontat j with trak i: β ij = P (Y k A)M ij (A), j > 0, A β i0 = 1 β ij. (8) j>0 These values for β ij are used in the same way as standard PDA: ỹ i = β ij ỹ ij. (9) j 6 Traking/Fusion Results Fusion of the video and sonar measurements is done through a likelihood surfae preproessing step, as desribed in [6]. The likelihood surfae preproessing allows the measurements, whih have different error harateristis, to be ombined into a single measurement and then this an be traked using any standard ontat-based traker. The position likelihood surfae is defined as the likelihood that there is a ontat at loation given video ontat z v and sonar ontat z s, By Bayes rule, L( z v, z s ) = P ( z v, z s ). (10) L( z v, z s ) = P (z v, z s ) P (z v, z s ). (11) Assuming that the error on the video and sonar measurements are independent, L( z v, z s ) = P (z v )P (z s ) P (z v, z s ). (12) For eah ontat, the measurement error is modeled as a Gaussian, P (z ) = N (; µ z, Σ z ), (13) where µ z = [x meas ; y meas ] and Σ z = [w meas /2 0; 0 h meas /2], where w meas and h meas are the estimates of the objets width and height from the sonar proessing. In the the ase of the video measurement data, the y dimension is not the same as the y dimension in the sonar measurements, so only the x measurement is used: µ z = [x meas ] and Σ z = [w meas ]. The estimate of the ontat s loation, ĉ from the fused measurements is found by: ĉ = arg max N(; µ v, Σ v )N(; µ s, Σ s ) P (z v, z s ). (14) Assuming that is uniform, and observing that the arg max is only a funtion of, ĉ = arg max N(; µ v, Σ v )N(; µ s, Σ s ) (15) JPDA requires a measurement error for all the ontats that are supplied, so a 2-D Gaussian is fit around the maximum, and this is used as measurement error for the JPDA algorithm. The likelihood surfae preproessing step is robust, and an also be used when only one measurement is available, or if there are multiple ontats for a single sensor. Figure 8 shows the results of fusing the video and sonar data, as well as traking the fused data. The traker is run on the fused data, and finds one onfirmed trak, indiated by the blue line. The target is moving from the top right of the figure to the bottom left. The large initial jump of the trak is due to the initialization of the Kalman filter parameters. At the end of the trak, the variane of the fused measurement error beomes larger, and the trak s position relies mostly on the Kalman filter s predited loation. 7 Conlusion This paper presents traking and fusion results from a unique data set. Results are reported on a small setion of the sea trial data set where the target (hannel buoy) is easy to pik out in the video data and the sonar data. As expeted, the data proessing algorithms worked quite well and the fusion/traking algorithm is able to trak the target. In future work, more diffiult traking senarios will be explored and the limits of the algorithms presented will be tested. For example, there are setions of the data set that have multiple targets as well as more lutter whih will present a more hallenging senario. The diversity of targets (various sizes of ships, for example) also present more hallenges. For example, the objet detetion for a blob is muh different that the outline of a ship. The algorithms for the data proessing will need to take into aount the variety of target shapes, whih was not explored in this paper. Figure 9 is 617

5 y (m) Traking ombined video and sonar data ontats from sonar ombined ontats onfirmed traks x (m) Figure 8: Traking results for fused data. The red dots are the ontats from the sonar, the red x s are the ontats that result from fusing the sonar and video data, and the blue line is the onfirmed trak from the JPDA traking algorithm. we will generalize the approah by studying eah plane in the HSV olor model, instead of fousing solely on the S- plane. From a fusion standpoint, it is envisioned that the fusion of the two data soures an be aomplished at different plaes in the proessing hain. In this paper, the detetions were fused in post proessing of the separate data streams, whih ould be further explored. For example, the y measurement in the video data ould be a feature that is fused to a ontat from the imaging sonar. It is also possible that both data streams ould be ombined in ommon imaging referene frame and then detetions ould be proessed. Although this paper presents a baseline approah to the traking/fusion problem of this unique data set, it should be lear that there are many avenues to explore going forward. Referenes [1] Paul M. Newman John J Leonard Juan D. Tardos, Jose Neira. Robust Mapping and Loalization in Indoor Environment Using Sonar Data. International Journal of Robotis Researh, [2] M. K. Agoston. Computer Graphis and Geometri Modelling: Implementation and Algorithms. Springer, [3] G. Bradski and A. Kaehler. Learning OpenCV. O Reilly, [4] L.G. Shapiro and G.C.Stokman. Computer Vision. Prentie Hall, [5] Y. Bar-Shalom and M. Sheffe. Sonar Traking of Multiple Targets using Joint Probabilisti Data Assoiation. IEEE Journal of Oeani Engineering, 8(3): , [6] D. Krout and E Hanusa. Likelihood Surfae Preproessing with the JPDA algorithm: Metron Data Set. In Information Fusion, th International Conferene on, July Figure 9: A sample frame from the BlueView P450, showing multiple ship hulls. a sreen shot of the sonar imaging data with multiple objets, whih in this ase are the hulls of multiple rereational power boats. The video proessing approah used in this paper is quite naive; one a model of the bakground is learned, a single pixel that does not fit the model will generate a ontat. In the future, the video proessing algorithm will be made more sophistiated by inorporating knowledge of the expeted targets to filter out spurious ontats. Additionally, 618