Suitability Analysis Based on Multi-Feature Fusion Visual Saliency Model in Vision Navigation

Transcription

1 Suitability Analysis Based on Multi-Feature Fusion Visual Saliency Model in Vision Navigation Zhen-lu Jin, Quan Pan, Chun-hui Zhao, Yong Liu School of Automation Northwestern Polytechnical University Xi an, , China Abstract Matching-area suitability analysis in vision navigation system for unmanned aerial vehicle (UAV) is a very worthy but full of challenges research area. In this paper, a multi-feature fusion based visual saliency model () was established by introducing invariant features of speeded-up robust features (SURF) directly into the visual saliency model, based on which the extraction method of suitable matching-areas was proposed. With the integration of cross-scale SURF feature maps in the way we defined, the conspicuity map of SURF channel is obtained. By adding SURF channel into the traditional visual saliency model and fusing multi-feature of SURF, color, intensity and orientation, the model is proposed. Based on the, salient locations in sensed map could be obtained and chosen as suitable matching-areas. Simulation results show that the error of image registration with extracted matching-areas based on meet the demands of vision navigation system. The proposed method may provide new ideas for autonomous navigation of UAV in the future. Keywords Visual Saliency Model; Multi-feature Fusion; SURF; Suitability Analysis; Vision Navigation I. INTRODUCTION Suitability analysis is a key issue in UAV scene matching vision navigation. The quality of the chosen areas influences the reliability and effectiveness of vision navigation observably. Hence, the selection of areas with robust suitability is a primary step for vision navigation. In general, the selection of suitable matching areas is an effective way to assure the matching precision for scene matching vision navigation system. Based on a series of basic hypotheses, several definitions and theoretical approximation methods regarding matching probability were studied in [1]. This paper discussed the area suitability for the very first time and established the foundation of this research field. So far, lots of papers have aimed at this issue [2-7], and there are mainly two kinds of suitability analysis methods: image signal correlation calculation method and comprehensive features evaluation method. However, the supposed signal correlation model usually cannot fit well with the actual situation, and parameters of the model are often difficult to be achieved. So, the image signal correlation based method is not always reliable. The comprehensive features evaluation method determines suitable matching areas by constructing statistic models of relationship This work was supported by Major Program of National Natural Science Foundation of China ( ), by National Natural Science Foundation of China ( and ), by Ai Sheng Innovation Foundation of Northwestern Polytechnical University (2012K001-). between image features and matching assessment indexes. This kind of method needs a mass of samples, and a suitable threshold in the model is hard to be given for most kinds of landscape features and imaging conditions. Moreover, the principal factors that influence matching suitability ( including landscape features, imaging differences, performance requirements for scene matching and so on ) are interacting and interdependent with each other. There will be conflicts and contradictions among results of suitability analysis. Therefore, the existing two kinds of methods for area suitability analysis both have certain inevitable drawbacks. Visual saliency [8-10] is a kind of psychological adjustment mechanism found in human perception system for visual scene analysis. With the help of this mechanism, we could extract relevant and useful information from outside circumstance and reject useless information. In general, visual saliency is the guarantee of efficiency and reliability in human visual perception process. In the research of vision navigation, lots of researchers are conducting studies about visual saliency [11-16]. Based on human visual attention system and radar, Wang et al. proposed a robust and real-time method for vision navigation by information fusion in [11]. Kottas et al. presented an efficient and consistent aided navigation system using line detector and visual saliency model in [12]. Siagian et al. designed a novel autonomous location and navigation system for mobile robot in [1] and [1]. The main idea is to choose rough location based on Gist model and then to refine the position with vision saliency model and Monte-Carlo method. Ouerhani et al. applied a modified visual saliency model into robot navigation field for landmark autonomously selection and recognition in [15]. Frintrop et al. presented a simultaneous location and mapping scheme for mobile robot based on visual landmark detecting, tracking and matching with active gaze control technology in [16]. Based on visual saliency, salient locations could be abstracted rapidly, which are often significant with certain feature. Using image matching method based on this feature, the corresponding salient area should show good matching performance. Therefore, between the visual salient locations and the suitable matching-areas for scene matching vision navigation, there are certain consistency. By introducing visual saliency model, a new area matching suitability analysis method should be given, and this novel method would be able to provide certain guiding significance for this issue.

2 The visual saliency model presented in [10] only needs to consider the saliency of visual scene itself and requires few priori knowledge. This model is a visual saliency calculation model of biologically credible, bottom-up, fast computation, and is also by far one of the most influential computation models. In order to facilitate the description, the computation model proposed in [10] would be called Visual Saliency Model (VSM) in this paper. With extracted color, intensity and orientation features, the VSM method integrates these features and abstracts salient locations. However, the above three features are not enough to extract the suitable matching-areas in vision navigation for UAV. Therefore, this paper will introduce Speeded - Up Robust Features (SURF) [17] into the traditional VSM calculation model and establish a Multi- Feature Fusion Visual Saliency Model (). Although there are a few papers trying to combine VSM and SURF [18-19], the VSM mechanism and SURF are processed individually and the VSM is used to reduce the number of abstracted SURF points. In our method, SURF is considered as a parallel channel with color, intensity and orientation in VSM, and the density of SURF points in all scale spaces is computed to achieve the conspicuity map of SURF channel. In a word, our method aims at coupling tightly the VSM and SURF for the very first time. Based on the proposed computation model of, salient locations in sensed map will be extracted and chosen as suitable matching-areas. Moreover, two universal and effective image registration methods based on Normalized Cross Correlation (NCC) and SURF [20] would be utilized to match the chosen suitable matching-areas in scene matching experiments, and the effectiveness of the presented method could be validated. II. VISUAL SALIENCY As an important psychological adjustment mechanism of the human visual perception system, visual attention could reduce the complexity of understanding the visual scene by pre-selecting visual information before further processing. Visual saliency computation model is originated from the study of attention mechanism in human visual system. From the formation mechanism of visual attention, it is usually divided into two kinds: one is bottom-up and featuredriven, and the other is top-down and task-driven. The topdown visual attention depends heavily on specific task, requiring correct human cognition or certain prior knowledge. The bottom-up visual attention relies only on the visual scene itself and determines saliency by pure visual input. Due to independence from task, bottom-up visual attention possesses a fast processing speed. One of the most widely used bottom-up computation models is proposed by Itti et al. in [10]. A. Visual Saliency Model Itti et al put forward a bottom-up Visual Saliency Model (VSM) in [10], which is mainly composed of two parts: calculation of saliency map and shifts of visual attention focus. The basic idea could be summed up as follows: First, multiple features are extracted from the input image, including color, intensity and orientation. Second, with Gaussian pyramids, center-surround difference operator and normalization calculation, three feature conspicuity maps are formed. Then, the saliency map is obtained by integration of the feature conspicuity maps. Lastly, attention focus is shifted in the winner-take-all neural network to get the salient locations, and the return inhibition mechanism is used to suppress the current attended location and shift attention focus to next salient area. Let r, g, b represent respectively the red, green and blue value of input image, the intensity can be defined as I = ( r+ g+ b)/. The broadly-tuned colors are calculated respectively by: R = r g+ b /2 G = g r+ b /2 B = b r+ g /2 Y = r+ g /2 r g /2 b With dyadic Gaussian pyramids method, a 9-scales pyramid will be produced in each channel of I, R, G, B and Y, represented by I( σ ), R( σ ), G( σ ), B( σ ) and Y ( σ ) respectively, where σ [0,,8]. As for the orientation channel, Gabor filters are used to convolve with the pyramid in intensity channel, and the obtained pyramid is represented by o o o o O( σθ, ), where θ {0,5,90,15 } is orientation specified in Gabor filters. Using center-surround difference operator (represented by ), the feature maps in intensity, red/green, blue/yellow and orientation channels could be achieved as follows: (1) I(,) cs = Ic () Is () (2) RG(,) cs = Rc () Gc () Gs () Rs () () BY(,) cs = Bc () Yc () Ys () Bs () () Ocs (,, θ) = Oc (, θ) Os (, θ) (5) Where c = {2,, } is the scale of center map, s = c+ δ is the scale of surround map, and δ {, } is the scale difference. It is important to note that, different feature channels concern different aspects of the image, and the magnitudes of these channels usually differ apparently from each other. Moreover, a strongly salient location appears in one channel may be easily masked by noise or less salient in other channels. Therefore, to combine the feature maps effectively, the normalization process is essential. After normalization (represented by N) and cross-scale integration (represented by ), the conspicuity maps in intensity, color and orientation channel are obtained: c+ c= 2 s= c+ ( (, )) I = N I c s (6)

3 c+ c= 2 s= c+ ( (, )) ( (, )) C = N RG c s + N BY c s (7) Input Image c+ O = N N( O( c, s, θ )) { o o o o c= 2 s= c+ θ 0,5,90,15 } (8) The final saliency map is the linear combination of above conspicuity maps, i.e. ( ) 1 S = N I + N C + N O (9) B. VSM based Area Matching Suitability Analysis By utilizing visual saliency model, the attended regions could be extracted easily and rapidly. The abstracted salient regions are usually significant with certain features, such as color, intensity, orientation, etc. Theoretically, image matching in these salient locations should show higher matching precision. Therefore, by introducing visual saliency calculation, a new analysis method for matching suitability is proposed. The major processes of VSM based matching-area suitability analysis are as follows: 1) Extract the color, intensity and orientation features from sensed image. 2) Produce the gaussian pyramids in each feature channels. ) Based on the center-surround difference operator and normalization, compute the feature maps of different channels. ) Integrate the feature maps in all scales to get the feature consipicuity maps of each channel. 5) Combine all consipicuity maps linearly to obtain the saliency map. 6) Using winner-take-all and return inhibition mechanisms to choose salient locations as the suitable matching-areas for scene matching. III. BASED SUITABALITY ANALYSIS Considering the continuous change of platform attitude, there will be large rotation angle, severe scale zoom and significant illumination difference between sensed image and reference image. Under these circumstances, the traditional scene matching methods based on pixel feature extraction usually require to calculate the deviation angle in heading of the UAV so as to forecast the rotation angle between sensed image and reference image and then conduct scene matching. This kind of scene matching method is ordinarily complex and the estimated rotation angle is usually not precise. However, a new kind of scene matching method by introducing invariant features, such as Speeded-Up Robust Feature (SURF) [1], could obtain high precision and robustness during image registration. For suitability analysis in vision navigation of UAV, by introducing SURF feature channel, a Multi Feature Fusion Visual Saliency Model () is proposed in this paper. Due to the robustness of SURF feature and its essentiality in image registration, the proposed model is robust to Color Linear Filtering Intensity Center-Surround Difference and Normalization Orientation Feature Feature Feature 12 images 6 images 2 images Cross-scale Image Integration and Normalization Linear Combination Saliency Winner-Take-All Salient Locations Return Inhibatation Fig. 1 Flow Chart of Model SURF Feature 6 images Cross-scale SURF Feature Density Calculation large rotation angle, severe scale zoom and significant illumination change. Based on this model, the process of extracting suitable matching-areas is stable and effective, ensuring the reliability and validity of vision navigation system. The flow chart of model is shown in Fig. 1. A. Multi Feature Fusion based Visual Saliency Model In order to add SURF channel to VSM visual attention model, the SURF pyramid is firstly computed using images of 9-scales. The SURF feature at pixel position of (i, j) in scale σ could be expressed as Surf σ ( i, j) ( i j) 1, SURF feature exists at pixel, SURF feature does not exist at pixel, = 0, Where, σ [0,,8]. ( i j) (10) And then, SURF feature maps in different scales are adjusted to be the same size as that of feature maps in color, intensity and orientation channel defined in the VSM visual attention model. By integration of cross-scale feature maps, it is easy to get SURF conspicuity map S. This step is actually to compute the density of SURF points cross all scales at each pixel as defined in (11). Surf i j 8 (, ) Surf ( i, j, σ ) = (11) σ = 0 Finally, after normalization and linearly weighted combination of color, intensity, orientation and SURF conspicuity maps, the saliency map in model could be achieved by

4 ( ) 1 S = N I + N C + N O + N S (12) B. Based Area Matching Suitability Analysis The extraction processes of suitable matching-areas based on model are as follows: 1) Extract the color, intensity orientation and SURF features from sensed image. 2) Produce the gaussian pyramids in each feature channels. ) Based on the center-surround difference operator and normalization, compute the feature maps of color, intensity and orientation channels. ) Integrate the feature maps in all scales to get the feature conspicuity maps, where the SURF conspicuity map is obtained by computing the cross-scale density of SURF points. 5) Combine all consipicuity maps linearly to obtain the final saliency map. 6) Using winner-take-all and return inhibition mechanisms to choose salient locations as suitable matching-areas. IV. SIMULATION AND ANALYSIS All the algorithms are coded by Matlab 2011a in Windows 7 6 bit operating system and are operated on a Pentium IV.0 GHz CPU with 2 GB memory capacity. Experimental data: All the reference images are captured from Google earth, in which the landscape features includes roads, buildings, farmland, mountains and so on. The sensed images are achieved from the reference images by rotating, zooming, adding noise and changing color, intensity contrast and so on. This procedure is trying to simulate the differences between sensed images and reference images. Since the correction process of sensed images in real-world system could not completely eliminate the imaging differences with reference images, there will exist diversities in color, intensity, noise, scale, and rotation angle. (b) (c) (d) Fig.2 Reference Images (column 1) and Sensed Images (column 2) (The scale ratios between sensed images and corresponding reference images are all 1.1. With clockwise being positive, the rotation angles between sensed images and reference images are -, -5, and -5 respectively for (a) - (d)) A. based Salient Areas Extraction The following experiment is aimed at extracting salient locations based on the proposed model. Four pairs of reference images and sensed images are showed in Fig. 2. The feature conspicuity maps of sensed image presented in Fig.2 (b) are shown in Fig., and the corresponding salient locations extraction results are presented in Fig.. To illustrate the advancement of our saliency model, the salient locations extraction results based on the original VSM model are also listed in Fig.. (a) Color (b) Intensity (c) Orientation (d) SURF Fig. Feature s of Sensed Image Presented in Fig. 2 (b) (a)

5 (a) so they could be used as suitable matching-areas for scene matching in UAV vision navigation. Currently, the scene matching algorithms could be divided into two classes: the algorithms based on areas and the algorithms based on features. In this paper, we choose the representative NCC matching method and SURF matching method to validate the performance of suitability analysis methods. The scene matching error curves are shown in Fig. 5, and the specific scene matching error data are detailed in Table I - IV. Regarding the scene matching error, it is calculated as the Euclidean distance between the estimated locations of sensed image in the reference image using scene matching algorithms and using manually image matching method. (b) (a) (c) (b) (d) Fig. Salient Areas Extraction Results of Sensed Images (Each row is corresponding to image pairs presented in Fig. 2 (a) - (d). The first column is VSM based areas extraction results using STB toolbox in [21], and the second column is salient areas extraction results by our method based on. Where, yellow irregular box shows the salient location) As shown in Fig., certain consistency and complementarity exists between the SURF conspicuity map and the other feature conspicuity maps in color, intensity and orientation channels. Moreover, as shown in Fig., the introduction of SURF channel ensures the possibility of extracting several other salient locations based on model, and these new locations are significant in SURF feature. In order to compare the performance of the two models for suitability analysis, the following part is going to conduct some scene matching experiments. B. based Matching-Area Suitability Analysis The extracted salient areas based on visual saliency models are significant with intensity, color, orientation and/or SURF, (c) (d) Fig. 5 Scene Matching Error Curves by NCC and SURF (Each row is corresponding to image pairs presented in Fig. 2 (a) - (d). The first column is VSM based areas extraction results using STB toolbox in [21], and the second column is areas extraction results by our method based on. Where, black represents the NCC matching method, while red represents the SURF matching method)

6 TABLE I. Scene Matching Error of Fig. 2(a) TABLE II. Scene Matching Error of Fig. 2(b) TABLE III Scene Matching Error of Fig. 2(c) TABLE IV. Scene Matching Error of Fig. 2(d) According to Fig., Fig. 5 and Table I - IV, following analyses are easy to be obtained: 1) As for the images shown in Fig. 2 (a), the VSM model and model both found 10 salient areas. However, the 10th salient location found by the former model is eliminated in the results of the latter method, and the SURF registration error of this area reaches as high as pixels. Moreover, a new salient area numbered as 8 is added, the NCC and SURF registration errors of this area are both below 5 pixels. The registration performance is very excellent. 2) As for the images shown in Fig. 2 (b), the VSM model and model found 1 and 15 salient areas respectively. Compared with the former model, three new salient locations are discovered and numbered as 8, 1, 15 in the results of the latter method, the NCC and SURF registration errors of the three areas are both below 5 pixels. In the results of the latter method, the salient location numbered as 6 is almost at the same position as the salient location numbered as 7 in the results of the former method, but the SURF registration error is much smaller (reduced from pixels to 1.52 pixels). What s more, in the results of the former method, the salient location numbered as 1 is not ever a salient one in the results of the latter method, the NCC registration error of this area is as large as pixels and the SURF registration error is also very big being pixels. ) As for the images shown in Fig. 2 (c), the VSM model and model found 11 and 17 salient areas respectively. Compared with the former model, six new salient locations are discovered and numbered as 8, 9, 10, 12, 16, 17 in the results of the latter method, the maximum NCC registration error of these six areas is.8522 pixels and the maximum SURF registration error is pixels. The registration precision is pretty high. ) As for the images shown in Fig. 2 (d), the VSM model and model found 10 and 12 salient areas respectively. Compared with the former model, two new salient locations are discovered and numbered as 11 and 12 in the results of the latter method, the NCC and SURF registration errors of the three areas are both below 5 pixels. The registration performance is very excellent. 5) Overall, in scene matching experiments of extracted salient locations based on the proposed model, the maximum NCC registration error is about 10 pixels and the minimum NCC registration error is about 1 pixel, while the maximum SURF registration error is about 8 pixels and the minimum SURF registration error is also about 1 pixel.

7 In summary, it can be found that no matter which kind of image registration methods is adopted, the extracted suitable matching-areas based on visual saliency models could achieve very high precision in scene matching and meet the requirements of UAV autonomous vision navigation system. In addition, registration errors of matching-areas abstracted by the proposed model are lower than that of the traditional VSM model. Therefore, it can be concluded that the established model is very effective for suitability analysis in vision navigation system. V. CONCLUSIONS This paper presents a novel suitability analysis method based on model in vision navigation for UAV. An improved version of the traditional visual saliency model that considering SURF feature has been proposed and used to extract spatial salient locations. These locations are then chosen as suitable matching-areas in vision navigation for UAV. Due to the introduction of invariant feature of SURF, the proposed approach is able to cope with severe navigation environments and guarantees the precision of vision navigation system. Simulation results have validated the ability of this method to select stable matching-areas. This paper could provide new ideas and theoretical guidance for engineering applications of UAV systems in the future. REFERENCES [1] Johnson M W. Analytical development and test results of acquisition probability for terrain correction devices used in navigation systems, AIAA 10th Aerospace Sciences Meeting, [2] G. Zhang, and L. Shen, Rule-Based Expert System for Selecting Scene Matching Area, Intelligent Control and Automation, Springer Berlin, [] JIAO Wei, LIU Guang-bin, ZHANG Jin-sheng, ZHANG Bo, and QIAO YU-kun, Immune PSO Algorithm-Based Geomagnetic Characteristic Area Selection, Journal of Astronautics, vol. 1, Jun. 2010, pp [] WANG Zhe, WANG Shi-cheng, Zhang Jin-sheng, Qiao Yukun, and CHEN Li-hua, A Matching Suitability Evaluation Method Based on Analytic Hierarchy Process in Geomagnetism Matching Guidance, Journal of Astronautics, vol. 0, Oct. 2009, pp [5] Chen, Yingying; Qian, Xinqiang; Yuan, Ming; Gao, Enting, Predicting the suitability for scene matching using SVM, International Conference on Audio, Language and Image Processing, 2008, pp [6] Wang Peng, Geomagnetic Aided Navigation Suitability Evaluation Based on Principal Component Analysis, International Conference on Industrial Control and Electronics Engineering (ICICEE), 2012, pp [7] Zhang G, Shen L. Rule-Based Expert System for Selecting Scene Matching Area, Intelligent Control and Automation, Springer Berlin [8] Treisman, A. M. & Gelade, G. A feature-integration theory of attention, Cogn. Psychol. vol. 12, pp , [9] C. Koch, S. Ullman, Shifts in selec1ive visual attention: towards the underlying neural circuitry, Human Neurobiology, vol., pp , [10] L. Itti, C. Koch, E. Niebur, A model of saliency-based visual attention for rapidscene analysis, IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 20, pp , [11] Tao Wang, Jingmin Xin, Nanning Zheng. A Method Integrating Human Visual Attention and Consciousness of Radar and Vision Fusion for Autonomous Vehicle Navigation, th International Conference on Space Mission Challenges for Information Technology, 2011, pp [12] Dimitrios G. Kottas, Stergios I. Roumeliotis. Efficient and Consistent Vision-aided Inertial Navigation using Line Observations, Multiple Autonomous Robotic Systems Labroratory, University of Minnesota, [1] C. Siagian, L. Itti. Biologically inspired mobile robot vision localization, IEEE Trans. on Robotics, 25(): , [1] Chin-Kai Chang, Siagian, C., Itti, L., Mobile robot vision navigation and localization using Gist and Saliency, IEEE/RSJ International Conference on Intelligent Robots and Systems, 2010, pp [15] Nabil Ouerhani, Heinz HÄugli, Gabriel Gruener, and Alain Codourey. A Visual Attention-based Approach for Automatic Landmark Selection and Recognition, Attention and Performance in Computational Vision (Lecture Notes in Computer Science), vol. 68, pp , [16] Frintrop, S., Jensfelt, P., Attentional Landmarks and Active Gaze Control for Visual SLAM, IEEE Trans. on Robotics, 2(5): , [17] Bay H, Ess A, Tuytelaars T, et al, Speeded-up robust features (SURF), Computer Vision and Image Understanding, 7(): 6-59, [18] Fernando López-García, Xosé Ramón Fdez-Vidal, Xosé Manuel Pardo and Raquel Dosil, Scene Recognition through Visual Attention and Image Features: A Comparison between SIFT and SURF Approaches, Ch.12 in Object Recognition, edited by Tam Phuong Cao, 2011, pp [19] Sergieh, H.M.; Egyed-Zsigmond, E.; Doller, M.; Coquil, D.; Pinon, J.- M.; Kosch, H., Improving SURF Image Matching Using Supervised Learning, in Proc. Int. Conf. Signal Image Technology and Internet Based Systems (SITIS), Nov. 2012, pp [20] Yufan Wang, Qiuze Yu, Wenxian Yu, An improved Normalized Cross Correlation algorithm for SAR image registration, Geoscience and Remote Sensing Symposium, 2012, pp [21] Dirk Walther and Christof Koch, Modeling attention to salient protoobjects, Neural Networks, 19, pp , 2006.