Methodology of context-dependent recognition of ground objects aboard unmanned aerial vehicle

Transcription

1 Methodology of context-dependent recognition of ground objects aboard unmanned aerial vehicle Mukhina Maryna, Barkulova Iryna, Prymak Artem Abstract Two approaches were combined into one algorithm in order to reach better accuracy of ground object detection and tracking. Blob analysis provides the separation of detected object by morphological criterion (mostly minimum area threshold is used) and is used to determine the region of interest for further selection of feature points. Points are detected and described by SURF method. Due to combination the amount of processing information is decrease that leads to increasing of the accuracy and efficiency of algorithm. A search and flight correction algorithm for the most extended landmark by which UAV can be followed was proposed. According to the visual analysis of the effectiveness of this algorithm it is possible to get correct accurate results of location. The algorithm of the normalized cross correlation (NCC) calculation and its application to the template matching was also represented. Index Terms normalized cross correlation, speed up robust feature, binary large object I. INTRODUCTION Nowadays Unmanned Aerial Vehicle (UAV) is an advanced technology with a great potential to change radically the methods of wars and to get new civil applications. It is an inseparable part of civil and military programs. The importance of UAVs was fully demonstrated in recent years. Regardless of their purpose, the number and applications of UAVs will increase significantly in future. Today UAVs have an increasing role in many social missions such as mobile aerial monitoring operations, border patrol, wildlife survey, military training, search and rescue procedures, real-time monitoring, harmful and hazardous industrial facilities. Also they solve tasks of combat and military applications. The main goal of the monitoring systems is detection and recognition of the object, so object classification is an essential process for UAV monitoring system. Recognition problems for airborne video observation can be solved be means of the context-dependent classification. Applying classification, it is needed to get two aims: find the difference between different classes of objects and support the fusion process of bounding-boxes, which are spatially connected. We are focused on two classes: moving objects and stationary objects. Object detection is determined as the process of distinguishing interesting objects from the background. Fast and efficient object detection becomes important topic in the advanced computer vision systems of unmanned aerial vehicles (UAV). For fast and unpredicted environment observed from aboard UAV it is urgent to detect the contrast ground objects like targets, aerodromes and runways without beforehand learning. As for other applications of ground object detection there are surveillance systems with intelligent computer vision, especially for landing maneuvering. They can be combined with other tasks such as object position estimation where firstly it is necessary to detect the ground object, and only then to estimate its position in the region of interest (ROI). The main problems arising in ground object detection are variable dimensions of the output which is caused due to the large number of objects that can be present in any given frame of video sequence. Any machine learning algorithm requires a fixed dimension of input and output for the model to be trained. Another important problem for object detection systems is the requirement of real-time (30 fps) application with keeping the prescribed level of accuracy. The complex model requires correspondingly more time for inference. The compromise between accuracy and performance needs to be chosen. II. RELATED WORKS Application of context-dependent classification for recognition tasks is proposed in [1]. In the context-free classification, the starting point was the Bayesian classifier. Morphological features such as object form, area and eccentricity were considered through context-dependent classification and approximated by linear dependences with acceptable error variance. Realization of research was done in MATLAB 2014a on the set of serial images of known forms. The object of BlobAnalysis was created. As result, dependences which can be used for object recognition have been obtained, and further they can be used together with interesting point detectors, like SURF, for increasing the reliability and accuracy of object recognition. 850

2 On the bases of probabilistic models, such as Bayesian classifier and Markov chain model, the detection by only two features related to Binary Large Objects (BLOB) analyses was executed in [2]. The algorithm for fast calculation of the normalized cross correlation (NCC) and its application to the problem of template matching is represented in [4]. Given a templatet, whose position is to be determined in an image f, the basic idea of the algorithm is to represent the template, for which the normalized cross correlation is calculated, as a sum of rectangular basis functions. Then the correlation is calculated for each basis function instead of the whole template. The result of the correlation of the template t and the image f is obtained as the weighted sum of the correlation functions of the basic functions. III. PROBLEM STATEMENT The problem is formulated as following. A sequence of video frames with contrast ground object is given. It is necessary to detect the target with the only known description of brightness and area. SURF points are interested of only region of interest where the information from BLOB analysis is taken for the supposed object. SURF method [3] searches the feature points using Hessian matrix. Hessian matrix determinant reaches extreme at points of maximum change in the gradient of brightness. Hessian matrix for two-dimensional function and its determinants are defined as follows: 2 2 f f 2 x xy H f x, y, 2 2 f f 2 xy y f f f det H 2 2. x y xy It identifies spots, corners and border lines well. For each point the direction of maximum brightness and scale, taken from the scale factor of Hessian matrix are calculated. The gradient at a point is calculated using Haar filters. It should be noted, despite of the fact that SURF is used to find objects in the image, the method itself does not work with objects. SURF does not detect the object from the background. It considers the picture as a whole and looking for features of the image. Correlation function [5] for the SURF method can be executed by multiplying of two matrices of descriptors of reference and current images: T surf D reference Dcurrent T D (1) D ( N ) D(1) D( n) D (1) D ( N ) D (1) D ( n) Normalized cross correlation (NCC) can be found by multiplying of two matrices of descriptors of compared images: T E D D where D i, in both images. NCC i j D j matrices of found feature points i and j In order to solve the problem with identifying the object it is proposed to use SURF together with BLOB analysis. The blob method follows the approaches of standard image processing - segmentation. In pixel coordinates a blob is defined as a group of eight-connected rectangular pixels having intensities above threshold intensity. Blob objects can be analyzed and then their morphological characteristics are calculated like area, coordinates of centroid, perimeter, orientation, etc. And the idea of proposed algorithm is to combine both approaches. Blob analysis is used to select contrast objects with predefined characteristics and to determine ROI in which the further SURF points will be detected. A main criterion is a blob object, which can be easily calculated as: A 1, ( i, j) R where ( i, j ) are indexes of pixel in ROI R which are equal to 1, that is, related to foreground objects. Since from frame to frame there may be transformation of object form due to both UAV motion and object motion, then counting the changes with the help of homography matrix is proposed. It can be used to reshape ROI in order not to lose the object because of intensive UAV maneuvering. Correlation function for BLOB objects W and normalized cross correlation ( E NCC blob blob ) for it are also calculated. For detecting linear reference points the Hough Line Transform [6] is a transform used to detect straight lines. To apply the transform, first an edge detection pre-processing is desirable. In machine learning there is a relationship between a set of input features ( x1, x2, x 3...) and produced outputs ( y1, y2, y 3...). This relationship is built on a target function. And it is a function that is produced by a machine learning algorithm seeks to replicate. The actual estimate of this function than is produced by a machine learning algorithm is called a decision function. This decision function is selected from a set of possible functions which map the input features to the produced outputs. The aim of a learning algorithm is to classify feature vectors that are not present within the training set. This is called generalization. It is common that many machine learning algorithms will be able to correctly learn the training set, but may produce random predictions on any test data not in the training set. This phenomenon is known as overfitting and can lead to poor generalization performance. Commonly used machine learning algorithms are represented in [7]. 851

3 For Hough Transforms, we will express lines in the Polar system. Hence, a line equation can be written as: cos r y x. sin sin Arranging the terms: r xcos ysin. In general for each point x y, we can define the 0, 0 family of lines that goes through that point as: r x cos y sin It means that each pair ( r0, ) represents each line that passes by ( x 0, y 0 ). The Hough Line Transform keeps track of the intersection between curves of every point in the image. If the number of intersections is above some threshold, then it declares it as a line with the parameters, r of the intersection point. You may observe that the number of lines detected vary while you change the threshold. The higher threshold is, more lines will be detected. Correlation function for Hough transform objects W Hough and normalized cross correlation ( E ( W )) have to be calculated. E NCC ( W Hough ) = s NCC reference Hough a a + b b s current reference, current - standard deviations of reference and current images, [ a,b ] - vector of line of reference image, [ a,b ] - vector of line of current image. Algorithm of linear features combination was developed taking into account the possible significant geometric distortions. 3-dimensional target model is used as a template image, the current image, obtained from the sensor, is 2-dimensional. Then it is assumed, both, template and current images, were processed earlier and have represented the combination of contour lines. The algorithm includes the procedures of rough and exact search. Since the procedure of first type is the original part of the algorithm, then they will be investigated. 3-dimensional template model includes all the lines, visible for the sensor at the target sighting at the all potential angles. Every linear feature is described by means of coordinates of one of the target points of the line, its length and orientation. For correlation processing it is necessary to know some other parameters, connected with each segment, these parameters characterize uncertainty of line position. It is represented the algorithm of rough search (Fig. 1).Steps 1 and 4: There is control of the search of azimuth, angle of position and scale definition. Step 2: Since the set of target sighting is a priori known, the target coordinates of obtained current image could be calculated, but it is necessary take into account the directions of sensors at the moment of image reception. Using prior target coordinates as the beginning of transform, the contour features of current image, by means of intervals and orientation calculation, are transformed in Hough space. 852

4 Contour of search of azimuth and elevation Contour of search of scale value A prior estimation of position and errors statistics of navigation filter 1. Search by azimuth and elevation 2. Current model transform 1. Calculation of target coordinates. 2. Transform in Hough space using the target as original, definition of criteria correlation functions Wsurf, WHough, Wblob and their normalized correlation coefficients 3. Correction Current and reference images 3. Reference model transform 1. 3-dimensional model transform in 2-dimensional model 2. Transform in Hough space using the target as original, definition of criteria correlation functions Wsurf, WHough, Wblob and their 4. Search of scale value 1. Scale selection in search region 2. Reference model modification in accordance with chosen hypothesis of the scale 5. Calculation of correlation value for current values of azimuth and elevation and scale value 6. Selection of the best position To algorithm of exact search Fig.1. The algorithm of rough search of azimuth, elevation and scale definition. 853

5 Step 3. 2-dimensional template model is transformed in Hough space in the same way as the current image transform described in step 2. Step 4. Since the value of the scale was assumed as given, the template or current model is corrected to the correspondent scale. IV. RESULTS For studying the proposed algorithm, video sequence from aboard UAV was used. The flight above several recognizable objects like runway, ground marks and vehicles was done. There were clear background like green field and foreground objects. It was processed in MATLAB 2014a environment. [8]. The results of Blob detection can be visualized by the following (Fig. 1). Fig. 2. Workspace of Matlab for variable stats which contains three fields by default: Area, Centroid, BoundingBox. For the previously processed frame by blob selection the features points were found and detected which of them are inside ROI (it is taken two times much as BoundingBox from variable stats). The visualization of SURF detection is presented in Fig. 3. Fig. 3. Arrow-like object is identified with the help of blob and then SURF points are found Example of SURF points descriptor is presented in Fig. 4. Fig. 1. Blob segmentation and its colored visualization Example of object characteristics is presented in Fig. 2. Fig. 4. Workspace of Matlab for variable Ipts1 (interesting points) which contains such fields as: Centroid points (x, y), Scale (Gaussian sigma), Laplacian (0 or 1), Orientation (in radians), Descriptor (8x8 reshaped in 64x1) Example of the best matching SURF points in ROI determined from blob analysis is presented in Fig. 5. Matching was done using the method [8] proposed by author - using normalized correlation coefficient (NCC) by multiplication descriptor matrices of two images and finding best matches by checking the threshold of NCC > Fig. 5. Matching of the 1 st and 12 th frames 854

6 Volume xx, Issue xx, Month 20xx, ISSN: A search algorithm for the most extended landmark by which unmanned aerial vehicle can be followed by and implemented flight correction was proposed. With the help of Hough line transform it is possible identify lines in both images and videos. Roads and horizon line were detected by Hough line transform in the experimented images and video. The threshold value was varied during the work with images to obtain the best result of landmark detecting (Table 1). Visualization of the object detection with the greatest length, as linear landmark, which allows estimating unmanned aerial vehicle location, was obtained. The visual analysis of the effectiveness of this algorithm for inertial navigation system correction showed the algorithmic software is appropriate for use on unmanned aerial vehicle board and due to applying computer vision systems, gives as correct results of location determining as possible. TABLE I. CHANGES OF THRESHOLD PARAMETER Frame Threshold Quantity of Quantity of detected lines unique lines Fig 6. Received determination of the main road (horizon line is detected as landmark) V. CONCLUSIONS The results showed that the use of this technique, the orientation evaluation UAV allows us to solve a number of high-precision tasks such as accurate positioning and following along a landmark, the problem of targeting, automatic landing and others. For the proposed algorithm of combination Blob and SURF methods the object detection is done by combining the estimation of feature points in ROI of blob object with specified characteristics like area, perimeter, orientation, etc. Accuracy in prediction of before detected object location is provided by context-dependent classification. Results of experimental research show the high reliability of such approach and sufficient accuracy for estimation of geometric transforms. References [1] Mukhina M. P., Barkulova I. V. Algorithm of variative feature detection and prediction in context-dependent recognition //Electronics & Control Systems Т [2] Mukhina M. P., Barkulova I. V. Structure of aided classification of ground objects by video observation //Electronics & Control Systems. 2017]. Т [3] H. Bay, T. Tuytelaars, and L. Van Gool. SURF: Speeded up robust features. In Proceedings of the 9th European Conference on Computer Vision, [4] Briechle K., Hanebeck U. D. Template matching using fast normalized cross correlation //Optical Pattern Recognition XII. International Society for Optics and Photonics, Т С [5] Baklickij V. K., Bochkarev A. M., Musyakov M. P. Metody filtracii signalov v korrelyacionno-ekstremalnyh sistemah navigacii //M.: Radio i svyaz T. 216 [6] Mukhina M. P., Tkachenko O. Y., Barkulova I. V. Accuracy research method of the modified algorithm for detecting linear landmarks //Electronics & Control Systems Т [7] Hughes M. A framework for automated landmark recognition in community contributed image corpora : дис. Dublin City University, [8] Mukhina M. P., Barkulova I. V. Algorithm of ground object detection with modified SURF method by using morphological features IEEE 5th international conference on methods and systems of navigation and motion control (MSNMC), October 16-18, 2018, Kyiv. Maryna Mukhina is currently working as professor at Department of Aviation Computer-Integrated Complexes of National aviation university, Kiev, Ukraine. She received MScDegree in Electrical Engineering, Candidate of Sci (Engineering) Degree (2005) and Doctor of Sci (Engineering) Degree (2016) at National aviation university. Research interests include correlation-extreme aidedna vigation systems, data fusion algorithms, image processing. Author of 40 scientific papers, textbooks, articles and proceedings. Iryna Barkulova is post-graduate at Department of Aviation Computer-Integrated Complexes of National aviation university, Kiev, Ukraine. She received Master Degree of Computer-integrated technologies at National aviation university. Research interests include correlation-extreme aided navigation systems, data fusion algorithms, image processing. Author of 15 scientific articles. Artem Prymak is post-graduate student at Department of Aviation Computer-Integrated Complexes of National aviation university, Kiev, Ukraine. He received Master Degree of Computer-integrated technologies at National aviation university. Research interests machine learning, data fusion algorithms, adaprtive control systems, big data processing algorithms, computer vision. Author of 5 scientific articles. All Rights Reserved 2017 IJARCET 855