Terminal Phase Vision-Based Target Recognition and 3D Pose Estimation for a Tail-Sitter, Vertical Takeoff and Landing Unmanned Air Vehicle

Transcription

1 Terminal Phase Vision-Based Target Recognition and 3D Pose Estimation for a Tail-Sitter, Vertical Takeoff and Landing Unmanned Air Vehicle Allen C. Tsai, Peter W. Gibbens, and R. Hugh Stone School of Aerospace, Mechanical and Mechatronic Engineering, University of Sydney, N.S.W., 006, Australia {allen.tsai, pwg, hstone}@aeromech.usyd.edu.au Abstract. This paper presents an approach to accurately identify landing targets and obtain 3D pose estimates for vertical takeoff and landing unmanned air vehicles via computer vision methods. The objective of this paper is to detect and recognize a pre-known landing target and from that landing target obtain the 3D attitude information of the flight vehicle with respect to the landing target using a single image. The Hu s invariant moments theorem is used for target identification and parallel lines of the target shape are investigated to obtain the flight vehicle orientation. Testing of the proposed methods is carried out on flight images obtained from a camera onboard a tail-sitter, vertical takeoff and landing unmanned air vehicle. Keywords: Tail-sitter vertical takeoff and landing unmanned air vehicle, computer vision, moment invariants, vision-based pose/attitude estimation, target identification/detection, parallel lines, vanishing points, perspective transformation and vision-based autonomous landing. 1 Introduction Using cameras as sensors to perform autonomous navigation, guidance and control for either land or flight vehicles via computer vision has been the latest developments in the area of field robotics. This paper focuses on how a tail-sitter Vertical Takeoff and Landing (V.T.O.L.) Unmanned Air Vehicles (U.A.V.) can use visual cues to aid in navigation or guidance, focusing on the 3D attitude estimation of the vehicle especially during the terminal phase: landing; where an accurate tilt angle estimate of the vehicle and position offset from the landing target is required to achieve safe vertical landing. Work in this particular field focusing on state estimation to aid in landing for V.T.O.L. U.A.V. has been done by a number of research groups around the world. Work of particular interest is the U.S.C. A.V.A.T.A.R. project[1]; landing in unstructured 3D environment using vision information has been reported but only the heading angle of the helicopter could be obtained from those vision information. Work done by University of California, Berkeley[] focused on ego-motion estimation where at least two or more images are L.-W. Chang, W.-N. Lie, and R. Chiang (Eds.): PSIVT 006, LNCS 4319, pp , 006. Springer-Verlag Berlin Heidelberg 006

2 Terminal Phase Vision-Based Target Recognition and 3D Pose Estimation 673 required to determine the 3D pose of a V.T.O.L. U.A.V. Yang and Tsai[3] have looked at position and attitude determination of a helicopter undergoing 3D motion using a single image, but did not present a strategy for target identification. Amidi and Miller [4] used a visual odometer but the visual data could not be used as stand alone information to provide state estimation in 3D space to achieve landing. This paper will depict how from a single image, target identification is achieved as a V.T.O.L. U.A.V. undergoes 3D motion during hover, just before landing, as well as being able to ascertain 3D pose from the appearance of the landing target which is subject to perspective transformation. The 3D attitudes of the vehicle are determined by investigating parallel lines of the landing target. Target identification is achieved through the calculations of Hu s invariant moments of the target. Paper Outline: Section will discuss in detail the image processing techniques undertaken, set up of the video camera on the T-Wing and the design of the landing pad; section 3 looks into the mathematical Hu s invariant moments theorem applied in order to detect and recognize the landing target, section 4 presents the mathematical techniques used to carry out 3D pose estimation, and section 5 presents and discusses the results for the strategies proposed tested on flight images taken from a U.A.V.; and lastly for Section 6, conclusion and directions for future work are drawn. Vision Algorithm The idea of the vision algorithm is to accurately maintain focus on an object of interest, which is the marking on the landing pad (here on after will be referred to as the landing target), by eliminating objects that are not of interest. The elimination of unwanted objects is achieved by a series of transformation from color to a binary image, filtering and image segmentations. In this section the image acquisition hardware set-up and the set up of the landing pad is firstly introduced and following that is the details of the vision algorithm..1 Image Acquisition Hardware Set-Up The capital block letter T is used as the marking on the landing pad to make up the landing target. The idea behind using a T is that it is of only one axis of symmetry whereas the more conventional helicopter landing pads, either a circle or the capital block letter of H, have more than one axis of symmetry. The one axis symmetry will allow uniqueness and robustness when estimating the full 3D attitudes and/or positions of the flight vehicle relative to Ground Coordinate System which is attached to the landing target. The camera used was a C.C.T.V. camera, the imaging sensor was of a 1/3 Panasonic Color CCD type. The resolution is of 737 horizontal by 575 vertical pixels with a field of view of 95º by 59º and records at 5 Hz. Recording of the images of the on-board camera was achieved by transmission using a.4ghz wireless four channel transmitter to a laptop computer where a receiver is connected. The camera was calibrated using online calibration toolbox [5]. The test bed used for the flights was

3 674 A.C. Tsai, P.W. Gibbens, and R.H. Stone the T-Wing [6] tail-sitter V.T.O.L. U.A.V. which is currently under research and development at the University of Sydney. The set up of the camera on the flight vehicle is shown on the following simulation drawing and photos: Fig. 1. Simulation drawing and photos of the camera set-up on the T-Wing. Image Processing Algorithm The low level image processing for the task of target identification and pose estimation requires a transformation from the color image to a gray-scale version to be carried out first. This basically is the elimination of the hue and saturation information of the RGB image while maintaining the luminance information. Once the transformation from color to gray-scale is achieved, image restoration, i.e. the elimination of noise, is achieved through a nonlinear spatial filter, median filter, of mask size [5 5] is applied twice[7] over the image. Median filters are known to have low pass characteristics to remove the white noise while still maintaining the edge sharpness, which is critical in extracting good edges as accuracy of attitude estimation is dependent on the outcome of the line detection. After the white noise is removed, a transformation from gray-scale to binary image is required by thresholding at 30% from the maximum intensity value, in an effort to identify the object of interest: the landing target, which is marked out in white. Despite that, the fact of sun s presence and other high reflectance objects that lie around the field where the experiment was carried out; it is not always possible to eliminate other white regions through this transformation. Image segmentation and connected component labeling was carried out on the images in an attempt to get rid of the objects left over from gray-scale to binary transformation that are of no interest. These objects are left out by determining that their area is either too big (reflectance from the ground due to sunlight) or too small (objects such as other markings). The critical area values for omitting objects are calculated based on the likely altitude and attitudes of the flight vehicle once in hover, which is normally in between two to five meters and ±10º. Given that the marking T is of known area and geometry the objects to be kept in interest should have an area within the range 1500 to 15,000 pixels after accounting for perspective transformation. This normally leaves two or more objects still present in the image, which was the case for about 70% of the frames captured. The Hu s Invariant Moments theorem is then applied as a target identification process in an attempt to pick out the correct landing target. Once the landing target is determined the next stage is to determine the edges of the block letter T in order to identify the parallel lines which are to be used later on for attitude estimation. The edge detection is carried out via the Canny edge detector [8];

4 Terminal Phase Vision-Based Target Recognition and 3D Pose Estimation 675 the line detection is carried out using the Hough transform [8]. The following figure shows the stages of image processing: Fig.. Image processing stages (from left to right and down): 1 st the grayscale transformation, median filtering, binary transformation, rejection of small objects, rejections of large objects, and finally target identification with line detection 3 Target Identification The landing target identification procedure is accomplished by using the geometric properties of the target. This involves investigating the moment of inertia of the target shape. Hu s invariant moments[9] are known to be invariant under translation, rotation and scaling in the D plane. This feature is very well suited to tasks associated with identifying landing targets. The following equation represents the moments of a D discrete function: m pq = i j p q i j I ( i, j) where (p + q) represent the order of the moments and I(i,j) is the intensity of an image. The indexes i, j corresponds to the image plane coordinate axes x and y indexes respectively. The central moments are the moments of inertia defined about the centre of gravity and are given by the following equation: μ pq = i j p q ( i x ) ( j y ) I ( i, j ) where the indexes have the same meaning as in equation (1), x and y are the centroids of the target shape. (1) ()

5 676 A.C. Tsai, P.W. Gibbens, and R.H. Stone The normalized central moment is defined as follows: η μ μ pq γ 00 pq = (3) Where γ = (p + q) /, for p + q =, 3,... The first four orders of invariant moments can be determined by the normalized central moments, they are as follows: φ + 1 = η 0 η 0 ( η 0 η 0 ) 11 φ = + 4η φ = ( η 3η ) + (3η η ) φ 4 = ( η 30 + η 1 ) + ( η 1 + η 03 ) In this paper all these four orders of invariant moments were tracked to carry out identification of the landing target allowing for perspective distortion. An object is considered to be the target if the sum of errors for all four orders is the minimum of all the objects. As the tail-sitter V.T.O.L. U.A.V. approach the landing phase, the flight vehicle will always undergo 3D motion, the invariant moment method described above is only known to be invariant under D scaling, translation and rotation. Therefore it is critical to investigate all facets of the invariant moments. As proven by Sivaramakrishna and Shashidharf [10], it is possible to identify objects of interest from even fairly similar shapes even if they undergo perspective transformation by tracking the higher order moments as well as the lower order ones. This method is applied later on to determine the correct target, i.e. target identification. (4) 4 State Estimation Due to the inherent instability with V.T.O.L. U.A.V. near the ground during landing, it is necessary to be able to accurately determine the 3D positions and 3D attitudes of the flight vehicle relative to the landing target in order to carry out high performance landing. The information given by the parallel lines of the target is one of vision-based navigation techniques used to obtain high integrity estimation of the 3D attitudes of the flight vehicle. Because of the landing pad marking: T, there are two sets of nominally orthogonal parallel lines that are existent, the two sets of orthogonal parallel lines each containing only two parallel lines are named A and B. The parallel lines in set A and set B correspond to the horizontal and vertical arm of the T respectively. 4.1 Coordinate Axes Transformation Before establishing the relationship of parallel lines to determine the vehicle attitude, the transformations between several coordinate systems need to be ascertained. The first coordinate system to be defined is the image plane denoted as the I.C.S. (Image Coordinate System), and then the camera coordinate system (C.C.S.) which

6 Terminal Phase Vision-Based Target Recognition and 3D Pose Estimation 677 represents the camera mounted on the flight vehicle. Lastly the flight vehicle body axes denoted as the vehicle coordinate system (V.C.S.). The line directions of the three flight vehicle body axes in the C.C.S. are pre-known and denoted as d normal, d longitudinal and d lateral. To describe the relative orientation between the landing target and the flight platform, the global coordinate system (G.C.S.) is also required. The axes of the G.C.S. are denoted as X, Y and Z. The X axis is parallel with the horizontal bar of the block letter T and pointing to the T s right; the Y axis is parallel with the vertical bar of the T and a positive down direction. These two axes are situated at the centre point where the vertical and horizontal bars meet rather than the conventional centre of gravity of the shape. The Z axis is therefore pointing up according to the right hand rule. The transformation from the C.C.S. to the V.C.S. amounts to an axis re-definition according to: ~ X ~ V. C. S. = X C. C. S. (5) 4. Flight Vehicle Orientation Estimation Via Parallel Lines Information Moving onto the application of parallel lines of the target shape to deduce 3D attitudes of the flight vehicle; it is well known that a set of 3D parallel lines intersect at a vanishing point on an image plane due to perspective transformation. The vanishing point is a property that indicates the 3D line direction of the set of parallel lines [11]. Considering a 3D line L represented by a set of points: L = {(x, y, z) (x, y, z) = (p 1, p, p 3 ) + λ(d 1, d, d 3 ) for real λ} (6) The line L passes through the point (p 1, p, p 3 ) and has the line directions (d 1, d, d 3 ). The image plane point of a 3D point on the line L can be written as: (u, v) = (f x/z, f y/z) (7) = [f (p 1 + λd 1 )/(p 3 + λd 3 ), f (p + λd )/(p 3 + λd 3 )] where f is the focal length of the camera and λ is the line length or parameter. A vanishing point (u, v ) can be detected if λ and d 3 0; that is: (u, v ) = [f d 1 /d 3, f d /d 3 ] (8) The direction of the line L can now be uniquely determined from the above equation to be: (d 1, d, d 3 ) = ( u, v, f )/ u + v + f (9) With the above theory applied to the problem of pose estimation; the line directions of the X axis, which corresponds to the horizontal bar of the T, and the Y axis, corresponding to the vertical bar of the T, in the C.C.S. can firstly be determined. The line directions are denoted as d x, d y and d z, where the line direction of d z is determined by the cross product of d x and d y.

7 678 A.C. Tsai, P.W. Gibbens, and R.H. Stone Once the directions of the G.C.S. axes in the C.C.S. are determined the flight platform s attitudes with respect to the landing target can be determined via the following, which takes into account the transformation from C.C.S. to V.C.S.: cos α = (d x d Long. )/( d x d Long. ), cos β = (d y d Lat. )/( d y d Lat. ), (10) cos γ = (d z d Normal. )/( d z d Normal. ). d x, d y and d z are the direction cosines of the G.C.S. axes in the C.C.S. α, β, and γ are the angles of the G.C.S. axes in the V.C.S., which correspond to the roll, pitch and yaw angles of the flight vehicle respectively. The following figure shows the relations between the C.C.S., G.C.S. and I.C.S., and the line direction of the vanishing point in C.C.S. Fig. 3. Schematic diagram of the relations between I.C.S., C.C.S., and G.C.S., and the line direction of G.C.S. Y axis in the C.C.S. 5 Experimental Results Flight images taken during the flight testing of the T-Wing, a tail-sitter V.T.O.L. U.A.V., were used to test the accuracy, repeatability and computational efficiency of the fore mentioned theories of target recognition and 3D attitude estimation. Flight images are extracted from a 160 seconds period of the flight vehicle hovering over the target with the target always kept insight of the camera viewing angles. Figure 4 shows the three angles between the V.C.S. and the G.C.S. as the flight platform pitches, yaws and rolls during the hover.

8 Terminal Phase Vision-Based Target Recognition and 3D Pose Estimation 679 Fig. 4. Plots of the Alpha, Beta and Gamma angles ascertained from vanishing points and the respective roll, pitch and yaw angles of the vehicle The estimated attitudes were compared with filtered estimates from a NovAtel RTK G.P.S. unit, accuracy of down to cm and a Honeywell Ring-Laser Gyro. of 0.5º accuracy. The errors from the attitude estimates from the parallel line information compared favorably with R.M.S. errors of 4.8º in alpha, 4.º in beta and 4.6º in gamma. This range of error is deemed good by other work s standard[1]. Figure 5 shows the computed 1 st, nd, 3 rd and 4 th order invariant moments of the landing target, T, from the images captured during the vehicle hover. Knowing that the invariant moments are only invariant under D translation, scaling and rotation, all four orders of the invariant moments were tracked. The results show that the error of the invariants moments computed as the flight vehicle hovers above the target are larger than the errors associated with D motion; but by tracking of the first four orders of invariant moments, the target always had the smallest total normalized error (sum of all four percentage discrepancy to true value obtained from noiseless images divided by four) than other objects remaining in the images. The first four orders of invariant moments shows that the normalized errors were greater for the period where the vehicle undergoes greater pitching and yawing moments than periods where the vehicle is almost in a perfect vertical hover mode. There were images where noise was a big issue but nevertheless by tracking all the four orders of invariant moments, the landing target could still be distinguished. With regard to computational time, the filtering and thresholding of the images took approximately 11.9% of the time; component labeling and segmentation took around about 54.99%; attitude estimation algorithm needed 4.933% and the invariant moments calculations required 8.16% of the computational time when dealing with two objects.

9 680 A.C. Tsai, P.W. Gibbens, and R.H. Stone Fig. 5. The computed 1 st, nd, 3 rd and 4 th order invariant moments (in red --) compared with true values (in blue -); and the total normalized error 6 Conclusion In this paper, an algorithm to accurately identify landing targets and from those landing targets, using computer vision techniques, to obtain attitudes estimates of a tail-sitter V.T.O.L. U.A.V. undergoing 3D motion during the hover phase is presented. This method of 3D attitude estimation requires only a single image which is a more computationally effective technique than motion analysis which requires processing of two or more images. This paper also presented techniques of accurately determining landing targets via the invariant moments theorem while an air vehicle is undergoing 3D motion. The results show a good accuracy with target detection and pose estimation with previous other work in this field. Further developments of these techniques in the future can possibly see autonomous landing of manned helicopter onto a helipad and commercial aircrafts autonomously detecting runways during landing. A major issue requiring investigation is the estimation of attitude when the landing target is only partially visible in the image plane. We intend in the future to integrate the attitude, position and velocities estimates, to act as guidance information, with the control of the T-Wing especially during landing.

10 Terminal Phase Vision-Based Target Recognition and 3D Pose Estimation 681 Acknowledgements. J. Roberts for his technical expertise with setting up of the camera system and the wireless transmission for recordings. References [1] S. Saripalli, J. F. Montgomery, and G. S. Sukhatme, "Vision-based autonomous landing of an unmanned aerial vehicle.", IEEE International Conference on Robotics and Automation, ICRA'0 Proceedings, Washington, DC, 00. [] O. Shakernia, R. Vidal, C. S. Sharp, Y. Ma, and S. Sastry, "Multiple view motion estimation and control for landing an unmanned aerial vehicle.", IEEE International Conference on Robotics and Automation, ICRA'0 Proceedings, Washington, DC, 00. [3] Z. F. Yang and W. H. Tsai, "Using parallel line information for vision-based landmark location estimation and an application to automatic helicopter landing," Robotics and Computer-Integrated Manufacturing, vol. 14, pp , [4] T. K. O. Amidi, and J.R. Miller, "Vision-Based Autonomous Helicopter Research at Carnegie Mellon Robotics Institute ", American Helicopter Society International Conference, Heli, Japan, [5] Z. Zhang, "Flexible Camera Calibration by viewing a plane from unknown orientation.", International Conference on Computer Vision (ICCV'99), Corfu, Greece, pp , September 1999 [6] R. H. Stone, "Configuration Design of a Canard Configuration Tail Sitter Unmanned Air Vehicle Using Multidisciplinary Optimization." PhD Thesis, University of Sydney, Australia, [7] R. C. Gonzalez, R. E. Woods, and S. L. Eddins, Digital Image Processing Using MATLAB: Pearson Prentice Hall, Upper Saddle River, N.J., 004. [8] R. C. Gonzalez and R. E. Woods, Digital Image Processing, ed., Pearson Prentice Hall, Upper Saddle River, N.J., 00. [9] M. Hu, "Visual Pattern Recognition by Moment Invariants.", IRE Transactions on Information Theory, 196. [10] R. Sivaramakrishna and N. S. Shashidharf, "Hu's moment invariants: how invariant are they under skew and perspective transformations?", IEEE WESCANEX 97: Communications, Power and Computing. Conference Proceedings, [11] R. M. Haralick and L. G. Shapiro, Computer and Robot Vision, vol. II: Addison-Wesley, [1] C. S. Sharp, O. Shakernia, and S. S. Sastry, "A vision system for landing an unmanned aerial vehicle.", IEEE International Conference on Robotics and Automation, ICRA '01 Proceedings., 001.