Implementation of UAV Localization Methods for a Mobile Post-Earthquake Monitoring System

Transcription

1 Implementation of UAV Localization Methods for a Mobile Post-Earthquake Monitoring System Mitsuhito Hirose, Yong Xiao, Zhiyuan Zuo, Vineet R. Kamat, Dimitrios Zekkos and Jerome Lynch, Member, IEEE Department of Civil and Environmental Engineering University of Michigan, Ann Arbor, MI , USA mhirose@umich.edu, yongxiao@umich.edu, zzuo@umich.edu, vkamat@umich.edu, zekkos@umich.edu, jerlynch@umich.edu Abstract The goal of this paper is to develop an integrated mobile sensing system that enhances Unmanned Aerial Vehicles (UAV) to provide an autonomous means of locating and mapping damage characteristics of a structural system. The UAV brings mobility to the integrated sensor payload so that the sensors can access remote sites and collect data of structures and field features following an earthquake or a natural disaster. A light detection and ranging (LiDAR) sensor and high resolution camera are installed on a UAV to obtain highly detailed and accurate topological maps of physical surfaces from which structural cracks, surface chances and minor faulting can be detected. One of the challenges in automating the operation of the UAV in challenging operational environments such as GPS-denied environments (e.g., under bridges, indoor environments) is accurate navigation. In a GPS-denied environment, localization relies on on-board sensors, which has limited accuracy when positioning the UAV in a fixed coordinate system. This paper explores solutions to this problem with vision- and marker-based localization methods. The experimental results are based on a LiDAR and camera payload integrated with an octo-rotor UAV platform that has an onboard inertial measurement unit. The experimental results show that three dimensional navigation and geometric mapping in unknown operational environments are both feasible and accurate. Keywords Mobile sensors, UAV, LiDAR, localization and mapping, wireless sensor network I. INTRODUCTION In recent years, there has been growing interest in civil and commercial applications of unmanned aerial vehicles (UAVs). In particular, they have promise in being utilized to rapidly collect perishable data after natural disasters leading to improvements in post-event monitoring, assessment, and management. While UAV technology is not novel in and of itself, the rich and rapid development of flight control and autonomy through signal processing and machine learning technologies have provided a leap in the capabilities of UAVs. The development of mobile and autonomous UAVs offers an opportunity to advance natural hazard management practice, facilitating effective and accurate decision making during and after natural hazard events. This paper proposes a monitoring approach that will introduce a major paradigm-shift in how post-event site monitoring is conducted, data is disseminated, and decisions are made based on site information. Light detection and ranging (LIDAR) scans of field surfaces allow landslide topology, toppled structures, liquefied lateral spreads, and cracks to be accurately captured in a quantifiable manner that allows structural and geotechnical site characterization. Compared with previous infrastructure monitoring methods including fixed monitoring systems, the use of ground (autonomous) vehicles or human workers, comparatively less work has been devoted to the advancement of air-based robotic platforms for data collection. One of the earliest examples of the application of UAV in the field is the use of a remote controlled helicopter equipped with a camera and wireless interrogation method that could interrogate wireless sensors installed on bridges [1]. Other work includes Fumagalli, et al. [2] who proposed a UAV for performing visual inspection of complex pipe networks in power facilities. Marks, et al. [3] explored a UAV equipped with LIDAR instrumentation to inspect and map vegetation in close proximity to high voltage power lines. Jahanshahi, et al. [4] reports on pattern classification and image processing algorithms that can automatically detect cracks in civil engineering structures using high resolution camera pictures collected by UAV. Zekkos, et al. [5] reported on the feasibility of using an UAV for the autonomous acquisition of shear wave velocity measurements during reconnaissance studies following an earthquake. These previous studies have shown, in proof-of-concept manner, the potential for mobile sensing by UAVs. However, there remains more work to be done in terms of pose (position and orientation, especially the UAV yaw angle) estimation and localization of a UAV in order to fully automate the UAV operation for high resolution data collection. Many sensing tasks in infrastructure monitoring involve a GPS-denied environment (e.g. under bridges, indoor environments). In a GPS-denied environment, localization relies on on-board inertial measurement sensors, which have limited accuracy on estimating the positional state of a UAV. This project explores solutions to this problem with vision- and marker-based localization methods. The accuracy of the localization methods are evaluated based on how well a LiDAR sensor installed on a UAV can map and identify known structural geometries, including narrow-in-width surface features (such as cracks and minor faulting) with small elevation differences. II. HARDWARE/SOFTWARE SETUP A. Octo-rotor platform As shown in Fig. 1, an existing octo-rotor UAV platform (X8+ from 3D Robotics) was modified to have increased capabilities tailored for vision- and LiDAR-based localization

2 and mapping of surface geometry. The UAV was equipped with a downward-facing LiDAR instrument (Hokuyo, UTM 3LX) to map out physical objects within a 3 m range using a 18 point cloud in a 27 range oriented vertically from the bottom of the UAV. The LiDAR has a 4 Hz scanning rate. In addition, a 72p camera is mounted on a brushless gimbal (Tarot T-2D) that uses two-axis stabilization technology to ensure stable video in any flight condition. The UAV also has installed an inertial measurement unit (IMU) that employs accelerometers and gyroscopes for crude estimation of UAV position and pose. To further enhance the sensing capabilities of the UAV, an onboard navigation computing module was also installed (ODROID-U3 with an 1.7GHz ARM quad-core processor and 2 G Byte RAM from HardKernel). The on-board computer is in charge of collecting sensor data (LiDAR, camera, and IMU data) and executes a pose estimation algorithm to identify the UAV pose. The collected data is transmitted to a ground control station (GCS) through a 5 GHz WiFi interface. An autopilot system designed by the PX4 open-hardware project was installed as a flight controller of the UAV. Table 1 summarizes the detailed specifications of the extended UAV platform. B. Inter-process communications Light-weight communication and marshalling (LCM) are used for the communication of sensor data within the UAV computing architecture. LCM has an efficient broadcast mechanism using UDP multicast thereby enabling low-latency Flight controller inter-process communications. Messages can be transmitted between different processes using the publish/subscribe message passing system of LCM [6]. LCM automatically Camera Table 1. UAV platform specifications Size cm 3 Payload capacity.8 kg Vehicle weight with battery 2.5 kg Flight time 15 mins Battery 4S 14.8V 1mAh 1C Sensors on-board IMU, GPS, camera, LiDAR Flight mode Autonomous path following Altitude hold Stabilize (manual) On-board processor 1.7GHz ARM Quad-Core 2GB RAM Navigation board Flight controller LiDAR Gimbal&came Fig.1 Octo-rotor UAV with LiDAR and camera installed Ground PC ROS, LCM RC controller 2.4G 5G Wifi Fig2. System overview On board computer compiles message definition into language-specific bindings, making message language platform independent. Logging and replay functions are also available in LCM. Data collected on the onboard computer are subscribed to by the GCS so that onboard sensor data is available for debugging purpose. LCM runs on a Local Area Network to enable wireless communications between devices through a WiFi interface. ROS and MAVLink protocol were used for message passing between the flight controller and the on board computer. On board computer sends desired navigation command to the flight controller through MAVLink and collect IMU data in real-time. Fig.2 summarizes the system overview. III. MOBILE UAV LOCALIZATION UAV reconnaissance activities require knowledge of the location of the UAV relative to objects in proximity to the UAV. Position estimation (that estimates the UAV position relative to a known map given the UAV perception of its environment) is a challenging problem. This paper addresses the localization problem of a UAV using a single camera. A 72p HD resolution camera is installed on the UAV to observe a priori defined fiducial markers attached to known locations. With the help of the fiducial markers, the camera is able to measure the pose (position and orientation) of the UAV. In order to properly represent uncertainty inherent to UAV pose estimation, a Bayes recursive filter framework is used. Bayes filters are comprised of two steps: prediction and correction step as will be described in detail. A. Prediction step The prediction step uses a model of the UAV to compute a belief of the current UAV pose. As the dynamics of octo-rotors are only loosely coupled in the x, y, z and yaw angle, the UAV dynamics are modeled as four independent state variables [9]. The prediction model for filters is designed to accept the state,, of the UAV which is 8x1 and the control input vector, u, which is 4x1. Fundamentally, the control input is the linear accelerations of the octo-rotor. (1) (2) The prediction model stated in state-space form and takes measured acceleration as an input to predict the next state: (3) where (4) ROS MAVLink Pixhawk Flight controller Camera LCM UART USB USB LiDAR

3 and is the process noise modeled as Gaussian. B. Correction step Regarding the correction model, the AprilTags [7] visual fiducial system (as will be described) enables the UAV camera to measure relative position and orientation. The experiment assumes that the pose of the AprilTags with respect to the global frame is known and all fiducial AprilTags are attached on the same horizontal plane (e.g., on flat ground). Therefore the pose of the AprilTags marker m can be expressed as: (5) For the th AprilTags, the measurement obtained from the camera is expressed as follows: (6) where is the measurement noise modeled as Gaussian. A number of filter algorithms are available for utilizing these prediction and correction models for localization of the UAV. Several algorithms including optimal Gaussian filters (Kalman filter) and nonparametric filters (particle filter) were applied to perform UAV localization. C. Kalman Filter Since the prediction and observation models considered are linear, the Kalman filter was applied as a best linear unbiased estimator. The Kalman filter estimates the state of a discretetime controlled process that is governed by the linear stochastic differential equation. The discrete Kalman filter algorithm implemented is shown in Algorithm 1 below [1]. represent state vector and its covariance matrix at kth step in the prediction step, whereas represent those in the correction step. Algorithm 1 : Kalman filter 1: loop (3 Hz) 2: 3: 4: 5: 6: 7: end loop D. Particle Filter The particle filter (Algorithm 2) was implemented to the same prediction and correction model as an alternative to Gaussian-based estimation techniques. The particle filter is a more representative non-parametric filter that approximates posteriors by a finite number of values. As it is capable of representing multi-modal beliefs, the particle filter is valuable when a UAV has to cope with phases of global uncertainty or hard data association of landmarks. A low-variance sampler is used to resample particles according to weights (likelihood) of each particle [1]. is a zero-mean Gaussian likelihood probability of observing particle m given observations. Algorithm 2: Particle filter 1: loop (3 Hz) 2: for m = 1 to M (number of particles) 3: 4: 5: end for 6: 7: 8: for m = 1 to M (number of particles) // resampling 9: 1: while 11: 12: 13: end while 14: end for 15: end loop E. Vision marker calibration In order to provide position and orientation of the UAV in a correction step, a camera is mounted to observe landmarks in the environment. To provide 3D pose (position and orientation) of a camera, the visual fiducial system AprilTags [7], is used as visual landmarks of known position. AprilTags are similar to QR codes conceptually in that they are printed on paper and are composed of only black and white blocks oriented in a distinct pattern. The detection method is robust to lighting conditions and view angles even at long distances. The detection application can not only find tags in the image, but also identify the ID of the tags given their unique patterns. Moreover, with calibrated intrinsic parameters of the camera and the size of ApriTag, the camera is able to detect landmarks (identifying the ID) in the environments and also to compute the relative transformation T between the landmarks and the camera. Once the transformation matrix T between the camera and the tags are obtained, the position of the camera can be extracted from T while the orientation of the camera can be derived from T. The OpenCV computer vision library was used to calibrate the camera by taking a sufficient number of images (approximately 1) of AprilTags printed on a planar board. AprilTags detection is performed to find the tags and the corresponding corners (i.e., image points). By combining the image points and the corner points in the global coordinate frame, a calibration algorithm is carried out to find the camera matrix (projection matrix) and distortion coefficient of the camera. The Kalman filter algorithm and the vision observation algorithm was tested in a simple environment with 1 fiducial tags. The frame rate was set at 3 fps and the UAV based camera was moved along an eight-shaped trajectory on an x-y plane. Fig.3 shows the result of tracking the position of the camera. Note that position and velocity noise was assumed to be independent. The variance of motion noise corresponding to position was set to.1 while that of velocity was set as.1.

4 z position [m] Kalman Filter original observation the University of Michigan. The dimension of the cage is roughly 4.57 m wide, m long, and 5.49 m high; the cage is fully enclosed on all sides. In the experimental area, there exist a structures (i.e., a floating tank) which can be used for evaluating the 3D reconstruction of the LiDAR data obtained from the UAV. In addition, about 5 fiducial markers are attached to the ground in a U-shape around the structure. Tag locations are measured by measurement tape to obtain a precise location for each Fig. 3 Trajectory of the camera using Kalman filtering (blue squares denote AprilTag locations) z position [m] y position [m] Particle Filter original observation y position [m] x position [m] x position [m] B. Localization results In this experiment, the UAV acquired data has different sampling rates. The IMU sampling rate is 1 Hz, the camera frame rate is 3 Hz, and the LiDAR scan rate is 4 Hz. Each data stream was interpolated so that the pose estimation algorithm can run at 6 Hz. Fig.7 shows the KF filter results on a subset of the data; the total data contains 1, images and a subset was used for visualization and experiments. Based on a comparison of the KF trajectory and the initial trajectory estimated by the camera, it is concluded that the KF filter is able to provide a smooth and stable set of UAV poses. It is also shown that when only using the camera, the pose estimation is not smooth because of motion blur and the low resolution of the camera. When the UAV flies at above a certain speed, motion blur occurs and the camera will lose some of the tags as a result. As the number of tags detected changes, accuracy and resolution of the vision-based localization fluctuates in tandem. Regarding the flying height of the UAV, when the flying height Fig. 4 Trajectory of the camera using particle filtering (blue squares denote AprilTag locations) This allows more flexibility to a constant velocity model assumed in the prediction step. Measurement noise variance was set as 1 for each x, y, and z observations. This gives more belief on the prediction model and smooth out the errors of observation of the fiducial markers. Fig. 3 shows that the Kalman filter (KF) gives a better estimation of the pose trajectory of a camera. Note that the blue squares in the figure represent coordinates of the fiducial tags. Fig. 4 shows the same result but when using the particle filter (PF) algorithm. The number of particles was set to 8. The observation noise covariance was set as an identity matrix of magnitude.1. Motion noise was again assumed to be independent and variance corresponding to the y-direction was set as.1. The other portion of motion noise variance was set as.1. Fig. 5 Concept figure of the geometric mapping task IV. EXPERIMENTS A. Experimental Setup Fig. 5 shows the overall concept of the experiments. Firstly, the UAV collects data from its camera and IMU. This data is then used as input to the KF algorithm for pose estimation. This estimated UAV trajectory is utilized for registering the location and pose of the UAV LiDAR. To simulate an indoor GPSdenied environment, a net-caged area (Fig. 6) was constructed in a hydraulic laboratory (i.e., wave tank) with high ceilings at Fig. 6 Flying environment in the lab

5 and it does not provide the information of the UAV along its flying direction, the ICP method is employed to correct the x and z of successive scans. Fig. 1 shows the original point clouds before correction from the top view. 2D scans are lined side by side in order but there are significant distortions due to changes in the UAV pose. Fig. 11 shows the same scan lined in order but correct by the ICP algorithm in terms of x and z direction. Compared to the original point clouds in Fig. 1, it is shown that ICP corrects distortion of point clouds in x and z directions. Outlier points are due to the meshed net of the UAV cage (i.e., some laser emission light went through holes of mesh net). Fig. 7 KF estimated (blue) versus camera estimated (red) trajectories Fig.8 Tag detection performance (at an altitude of 4.8 m) Fig.1 Original point clouds (top view) Fig.9 Tag detection performance (at a low altitude of 3 m) of the UAV is greater than 5m, the camera will detect less tags compared to flying at a lower height (e.g. 3m) because of the image resolution of each tag. Therefore, the UAV flies at a lower height in collecting the data. Fig. 8 and Fig. 9 show the performance of the tag detection algorithm when the UAV flies at a high altitude and at a low altitude, respectively. When high, 9 out of 16 tags are detected while 11 out of 11 are detected when low. C. Iterative Closest Point (ICP) The ICP algorithm (Algorithm 3, 4) is usually utilized for estimating the 3D rotational and translation matrix between two LiDAR scans. However, since the laser can only get 2D scans Fig.11 ICP-corrected point clouds (top view) Algorithm 3: LiDAR 2D scan registration 1: 2: 3: for : endfor 5: return for

6 Algorithm 4: ICP for 2D scan 1: for 2: Initial transformation for k and k+1 scan 3: Iterative procedure to converge to local minima 1. find closest point 2. Transform, to minimize between each and 3. Terminate when change in error falls below a the preset threshold 4: Choose the best and for different initializations 5: endfor 6: Return and for D. Digital Terrain Model with LiDAR The LiDAR on the UAV presents a 2D-planar scan of space by measuring the phase lag of the returned light signal. The LiDAR generates 18 points per rotation. These points are mapped from the pose of the UAV utilizing range and angle information. The camera channel and LiDAR channel each published time stamps along with the data so that pose of the UAV and LiDAR data can be synchronized. Fig. 12 shows a result of reconstruction of airborne LIDAR data collected by the UAV. Each 2D planar scan point is projected and mapped based on the estimated location of the UAV. 3D geometric models were reconstructed from the LIDAR data; these images captured the general geometry of the target structure well. ACKNOWLEDGMENT The authors would like to acknowledge the funding provided by the National Science Foundation under Grant Numbers CMMI , CMMI , and CMMI REFERENCES [1] Taylor, Stuart G., Farinholt, K. M., Flynn, E. B., Figueiredo, E., Mascarenas, D. L., Moro, E. A., Park, G., Todd, M. D., Farrar, C. R., "A mobile-agent-based wireless sensing network for structural monitoring applications." Measurement Science and Technology 2(4): 4521, 29. [2] Keemink, A. Q. L., Fumagalli, M., Stramigioli, S., Carloni, R. "Mechanical design of a manipulation system for unmanned aerial vehicles." 212 IEEE International Conference on Robotics and Automation (ICRA), IEEE, 212. [3] Ax, M., Thamke, S., Kuhnert, L., Kuhnert, K. D., "UAV based laser measurement for vegetation control at high-voltage transmission lines." Advanced Materials Research. 614(1): , 213. [4] Jahanshahi, M. R., Masri, S., Padgett, C. W., Sukhatme, G. S., "An innovative methodology for detection and quantification of cracks through incorporation of depth perception." Machine Vision and Applications, 24(2): , 213. [5] Zekkos, D., Lynch, J. P., Sahadewa, A., Hirose, M., and Ellis, D. "Proofof-Concept Shear Wave Velocity Measurements Using an Unmanned Autonomous Aerial Vehicle." Geo-Congress, Atlanta, GA, 214. [6] Huang, A.S., Olson, E., Moore, D. C., "LCM: Lightweight Communications and Marshalling," 21 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Taipei, Taiwan, 21. [7] Olson E. AprilTag: A robust and flexible visual fiducial system, 211 IEEE International Conference on Robotics and Automation (ICRA), Shanghai, China, pp. 34-7, 211. [8] Benini, A., Mancini, A., Longhi, S. An IMU/UWB/Vision-based Extended Kalman Filter for Mini-UAV Localization in Indoor Environment using a Wireless Sensor Network, Journal of Intelligent & Robotic Systems, 7(1-4): , 213. [9] Meier, L., Tanskanen, P., Fraundorfer, F., Pollefeys, M., PIXHAWK: A system for Autonomous Flight using Onboard Computer Vision, 211 IEEE International Conference on Robotics and Automation (ICRA), Shanghai, China, pp , 211. [1] Thrun, S., and Burgard, W. Probabilistic Robotics. Cambridge, Mass.: MIT Press, 25. Fig.12 LiDAR data reconstruction V. CONCLUSIONS The paper has shown that UAVs are capable of implementing a LiDAR-based structural health monitoring system in a GPS-denied environment following a natural disaster. The localization algorithm implemented successfully register LiDAR scan data to reconstruct geometric feature of surrounding environment of the UAV. Future works include mapping of the unknown landmarks by SLAM methods and developing damage classification algorithm based on LiDAR point clouds data.