FINAL REPORT. Prepared for Evergreen Unmanned Systems and Shell International Exploration and Production Inc.

Transcription

1 FINAL REPORT Prepared for Evergreen Unmanned Systems and Shell International Exploration and Production Inc. Center for Collaborative Control of Unmanned Vehicles University of California, Berkeley PI: Prof. Raja Sengupta, Project Manager: John Connors Project Team: Ben Kehoe, Zu Kim, Tom Kuhn and Jared Wood Contains Proprietary Information belonging to: UC Berkeley and Evergreen Unmanned Systems

2 Table of Contents Executive Summary... 3 Search and Rescue... 3 System Overview... 4 Assessment of the ScanEagle as a platform for Autonomous Intelligent Control... 5 Computer Vision Detection... 6 Probabilistic Search... 8 Path Planning and Control Cursor-on-Target Interface Simulation Environment Flight Experiments Experiment Setup Day 1 (December 8 th, 2009) Day 2 (December 9 th, 2009) Day 3 (December 10 th, 2009) Day 4 (December 14 th, 2009) Conclusions and Analysis Future Work

3 Executive Summary The Center for Collaborative Control of Unmanned Vehicles (C 3 UV) at the University of California, Berkeley is an interdisciplinary research group focused on making unmanned vehicles operate autonomously, without extensive monitoring or intervention by human operators. C 3 UV, in conjunction with Evergreen Unmanned Systems (EUS), demonstrated a new technology to autonomously search and detect targets for use in search and rescue applications. An autonomously controlled, fixed-wing aircraft performed detection and localization of an orange kayak. The system maintained a probability density function (PDF) that expressed the likely location of the target. Video data from the aircraft was analyzed in real time to detect the target and update the PDF. The aircraft and camera sensor s paths were generated from the PDF to maximize the likelihood of detection of the kayak. Following this method, the aircraft searched the given space to detect and localize the target in an efficient manner. A series of flight tests were conducted at McMillan Airfield, Camp Roberts, CA on December 9 th -14 th. The aircraft were launched and operated by EUS. The automation technology was computed on a standard desktop computer operated by C 3 UV. The results of the experiment show positive advancement for autonomous search and rescue. Once the autonomous intelligence was adapted to the Scan Eagle system, the search and rescue target, i.e., an orange kayak, could be autonomously detected and localized to approximately 40 meters. The UAS flew at an altitude of 100 meters. We have also been able to identify a few simple ways in which the Scan Eagle can be improved to make it a robust platform for intelligent autonomous operation. These are summarized in the Section titled Future Work. Search and Rescue Search and Rescue (SAR) refers to missions performed by emergency crews to locate and retrieve people or equipment from dangerous situations. Typical scenarios include aircraft/vehicle accidents, natural disasters or acts of war. The two-fold challenge in search rescue is to find and locate those in need of rescue, and to access those individuals and provide support for their rescue. In traditional SAR mission, numerous human searchers require long periods of time in order to locate a target. Though the process of searching is not technically challenging to a human being, it can be very demanding and requires precision. Due to the repetitive and accurate nature of the exercise, it is an ideal problem to be solved through computer based techniques. Current manned SAR techniques, as employed by groups such as the Coast Guard, involves flying predefined patterns over a given search area. Examples of these patterns are outwardly travelling spirals, or transect paths. The limited number of patterns limits the search team s 3

4 ability to adapt to new scenarios, and may not incorporate information specific to each search. The system developed here can integrate prior knowledge of the search area and continuously reevaluates flight plans to produce the highest likelihood of detection. The system can also revisit areas of interest without interfering with pre-defined flight plans. In this work, the goal is to detect and localize an orange kayak (representing a life raft), within a 2 sq. km area. A fixed wing aircraft, the Insitu ScanEagle, will be automated to fly above the area with an electro-optical (EO) camera, and search for the target. The system will detect and localize the target in order to aid rescuers in the SAR task. System Overview The SAR system, shown in Figure 1, utilizes numerous science disciplines, including computer vision, mathematical modeling and controls. An EO camera on board the aircraft broadcasts an analog video signal to the system ground station. This video feed is digitized and each frame is made available to the C 3 UV system. Computer vision techniques analyze each video frame to detect the presence of the target. These algorithms map the image to the surface of the ground and estimate the location of the target. A likelihood function expresses the probability that the target exists at that location. These functions are merged into a larger map of the search area. The result is a probability distribution function (PDF) that expresses the probability that the target exists at any given location. As the target is not equally likely to be everywhere in the search area, some aircraft and sensor paths will be more likely to find that target than others. A desired aircraft and sensor path is created from the PDF and current aircraft telemetry in order to maximize the probability of detection. These paths are sent to the ground station through the Cursor-on-Target (CoT) interface, and broadcast to the aircraft. The aircraft follows the paths, and returns the new video stream as described above, closing the control loop on the SAR application. 4

5 C 3 UV System Vision Detection Probabilistic Search Likelihood Functions PDFs Path Planning Analog Video Telemetry Aircraft/Sensor Paths ScanEagle Aircraft Electro-Optical Camera ScanEagle Ground Station CoT Interface Figure 1: System Architecture for SAR Experiment Assessment of the ScanEagle as a platform for Autonomous Intelligent Control The ScanEagle system presents various restrictions that limit performance or flexibility. The majority of these issues are due to limitations of the CoT interface. On the ScanEagle aircraft, this interface is the only automated way to exert control over the aircraft. A direct interface to the autopilot is available on board the aircraft and may be a more appropriate way for future systems to interact. The telemetry data sent by the ground station demonstrated erratic behavior that made the integration process more difficult. Data packets were occasionally sent out of order or were duplicated. Additionally, some packets contained identical information with different timestamps. Because accurate vision detection relies on precise synchronization of telemetry with video, the lack of reliable telemetry reduces the accuracy with which the synchronization problem can be addressed. Control of the aircraft is performed through a limited CoT interface. Two modes of control are available: path following, or fixed-radius orbits. The current path-following techniques of the ScanEagle's autopilot present the greatest obstacles to increasing performance. The aircraft fails to implement a sufficient planning horizon, often flying direct line paths to each waypoint. Only after passing a waypoint does the aircraft attempt to turn toward the next. For this reason, the aircraft consistently overshoots the desired path and reduces the effectiveness of 3 rd party aircraft control. Because placement of the aircraft is a key factor in placing the sensor footprint, these behaviors limit the effectiveness of autonomous path planning. 5

6 The alternate control method is through the use of orbit points. Given a center point and radius, the aircraft will circle the point at the given radius. This feature can be a very useful way to gain lower level control of the aircraft as the path follower does not execute tight turns well. Unfortunately, the CoT interface implemented on ScanEagle does not allow the rotational direction of the orbit to be specified. The aircraft will therefore always orbit in the same direction, drastically decreasing the effectiveness of this control input. The ScanEagle provides a 2 axis, inertially stabilized gimbal system. The system provides accurate and stable images of a specified location. The system is limited however by the commands available through CoT. Ideally the interface would provide access to a lower level of control of the sensor. At present, the system is only capable of pointing the camera at a specific location. Due to the high rate at which the computer can process the vision data, the camera must be constantly re-commanded to point at a new location. The ground station limits the rate at which these updates may happen. The gimbal provides other operation modes that could be used to combat some of these issues. However, those capabilities are not available through the CoT interface. In addition to the control limitations, the gimbal provides poor telemetry feedback. The ground station reports gimbal orientation in terms of Earth coordinates. For computation purposes, this data must be translated in to relative gimbal angles. This is an unnecessary step as the true angles are available on the aircraft, but not provided through CoT. Computer Vision Detection Airborne sensors present unique challenges in real-time vision detection due to their perspective and relative speed. C 3 UV has focused on developing accurate vision-based detection algorithms for airborne systems. It is not possible to achieve perfect detection or localization performance, due to the intrinsic difficulty of the detection problem and sensor accuracy. The system accounts for the possibility of detection and localization errors by using target likelihood estimation algorithms to best approximate the target position based on the given error range. These likelihood functions can be used to update the probabilistic search framework to generate the PDF. The vision framework consists of two core components: target detection and target localization. First, each video frame is compared to a target model with a color- and shape-based target detection algorithm. The model is designed to detect a bright orange life boat in an ocean. The raft s distinctive color is a useful cue for target detection. A detection example is shows in Figure 2, where a color filter is applied for both the target and background. The dark background simulates that look of the Arctic Ocean from above, and provides good color contrast. A pixel grouping and shape analysis algorithm is used to filter out noise. The raft model is compared to real images by scaling the model based on altitude and zoom levels, thus filtering out objects that are not of a reasonable size. The use of these filters provides for a more robust algorithm and greater accuracy. 6

7 Figure 2: Color-based life boat detection algorithm: Original image (left), after color filter for target (middle), result of grouping (right). The real world coordinates of the detected target are calculated from the location and orientation of the sensor at the time the image was captured. Instead of specifying the target location as a single point, a likelihood map is created that can account for detection errors and sensor inaccuracies. Based on the map generated from it, a flight path is generated to find an efficient path for the search. Ground based real-time image processing for target search using the ScanEagle presents several challenges: Video quality: The wireless video downlink degrades video quality and produces various types of noise affecting color processing. At the flight altitude, the Scan Eagle video resolution also makes shape based detection difficult. Time synchronization: There exists an unknown time difference between the video processing computer and the ScanEagle. Inaccurate sensor readings: The attitude reading given by ScanEagle telemetry is not accurate. The attitude estimation results in an error greater than several degrees due to both sensor error and delay between different sensors. In addition, the ScanEagle telemetry only provides GPS based altitude, which is less accurate than barometric altitude. The GPSbased altitude estimate can have errors as large as 100 meters. We introduced new components to the image processing system to address these problems. The vision detection portion of the system uses improved color filters, and adds a layer of background detection. As an additional measure, sensor readings are incorporated to constrain the size of the projected target in the image. To adjust for time synchronization issues, a timing verification tool is used to tune the offset and delay between the aircraft and ground computers. A virtual point is superimposed on the video image based on aircraft telemetry. When synchronization is correct, the virtual point remains even when the camera is in motion, but with a synchronization offset, the point precedes or follows the camera motion. We compensate for the offset by manually manipulating the camera and observing the relative motion of this virtual point. This ensures the video images are correctly paired with telemetry data. Upon detecting a target, the vision components return an estimate of the target s location, calculated from the aircraft s telemetry data. When significant errors are present in this data, the localization of the target will not be accurate. The location estimate is fused with the PDF based 7

8 on a likelihood of that observation. The likelihood increases the probably of the target being in that location, while simultaneously decreasing the probability that the target is anywhere else. When error is present in the localization estimate, the likelihood functions place probability in the incorrect locations, as well as eliminated probability that may have been in the correct location. To reduce the effects of sensor error, the likelihoods returned from the vision detection are broader than in our on-board systems and represent greater uncertainty in the location of the target. The overlap of multiple target detections sharpens the localization of the target, without eliminating previous detection results. Multiple detections of the same target produce more accurate localization of the target. Probabilistic Search The purpose of a search is to find an object referred to as the target. When a search begins, the location of the target is unknown. Utilizing prior information about the last known location of the target is a key step in a successful search. Given an approximate location within some search space at some initial time, a distribution of the target s probable location, called the prior, can be constructed. For example, in Search and Rescue, the last known position of a ship may be given. A conservative distribution of the probable target location would assume the worst case propagation from the last known location (e.g., and model the probable target location as a Gaussian). Consequently, a search is initialized with a highly uncertain estimate of the target location. The goal of the search is then to decrease this uncertainty to the point that the target can be considered localized (i.e., found). The uncertainty in the target location distribution is decreased by making observations within the search space. An attribute of large scale probabilistic searches is that each observation covers a relatively small area as compared to the entire search area, so most observations will report that the target was not detected. Consequently there are two types of observations target detected and target not detected. Each type of observation (detection or no-detection) is modeled according to an observation likelihood function, where is the observation (target detected or target not detected). Figure 3 shows sample observation likelihood functions for both types of observation. The likelihood function is used to update the target s distribution of probable locations as which is known as Bayes' Rule. Figure 4 is a representation of the distribution of probable target locations after a sequence of no-detection observations have been made. Note that the distribution is represented as a contour plot with red being the highest level of probability and blue being the lowest. The data shown in Figure 4 and Figure 5 is from flight experiments performed on the ScanEagle while at Camp Roberts. The distribution is used to direct the effort of the search to highly probable target locations. How the aircraft is guided through the search space is determined by a path planner. 8

9 Figure 3: Observation likelihood functions for target not detected (left) and target detected (right). Figure 4: Distribution of probable target locations before a target has been detected. 9

10 Figure 5: Distribution of probable target locations after the target has been detected. Path Planning and Control The purpose of the path planner is to find an optimal path for the aircraft and sensor. Ideally, a path could be chosen such that its estimated reduction in entropy would be more than any other path and the dynamics of the aircraft would be accounted for such that it would be feasible for the aircraft to follow. However, it is not feasible to determine an optimal path for an infinite time horizon in real time. This leads to minimizing the cumulative predicted entropy over a path of finite length. The predicted entropy at some time into the future of a distribution is defined as The exact cumulative entropy cannot be minimized in real time. As such, a suitable alternative objective must be minimized such that the path produced approximates the entropy-optimal path. The objective we have considered is the probability of not detecting the target at every step in the path. Minimizing this is equivalent to maximizing the probability swept out by the path. The probability of not detecting the target over an entire path is 10

11 One side effect of using a finite horizon path is that special cases arise. The ScanEagle is a fixed wing aircraft, and as such it has a maximum bank angle, resulting in a minimum orbit radius. Consider the scenario in which the point of interest is to the left of the aircraft. If the path length is not long enough for the aircraft to perform a full turn-around, then the path planner will command the aircraft to make a hard left turn. If the point of interest is within the minimum orbit radius of the aircraft, the aircraft will end up in an orbit around the point, unable to fly over it. Let this area be termed the unreachable area. In reality this unreachable area is in fact reachable, but not with a constant bank angle. With a short path planning horizon, points within this area are essentially unreachable. However, points close to the plane can be imaged using the gimbaled camera. Figure 6 shows an approximation of the area. It is necessary to account for special cases and prevent them from occurring. Figure 6: Unreachable set. For our application, the point of interest is the distribution peak the best estimate of the target location. To account for the peak, the path planner not only minimizes the probability of not detecting the target over the path, but also guarantees that the peak will always be outside of the unreachable area (thus ensuring that the aircraft will never enter an infinite orbit around the peak). In typical flight conditions the planner will determine a finite-horizon path through the probability distribution, while heading roughly toward the peak. The planner guarantees that the aircraft will head roughly toward the peak by checking if peak is within a cone over the heading of the aircraft. If the peak happens to leave this heading cone, then the planner will initiate a hard turn path toward the peak. The path planning algorithm is shown in Figure 7. Notice that there are additional cases that the path planner checks. These cases are 1. The entropy of the distribution is low enough to consider the target localized 2. The aircraft will head outside of the search space unless it makes a hard turn. 3. The peak is unreachable and the aircraft is heading outside of the search space. 11

12 For case 1, if the target can be considered localized, then the path of the aircraft should be directed to orbit the target so that it can be kept in sight this is a "tracking" path. The target is considered localized when the estimate distribution s entropy drastically decreases. Figure 8 shows this decrease in entropy for one of the experiments. For case 2, most real searches will involve airspace restrictions. These restrictions must be accounted for and checked by the path planner. In order to guarantee that the aircraft will not cross airspace boundary, the planner must check if part of the unreachable area will cross the airspace boundary. Figure 9 represents how the unreachable area is related to the aircraft potentially crossing the airspace boundary. In Figure 9, is the unreachable space to the left of the aircraft. Similarly is the unreachable space to the right of the aircraft. Figure 9 shows crossing the airspace boundary. In this situation, the aircraft is commanded to make a hard left turn toward the peak. This is what is meant by "head to PDF peak". Case 3 is a combination of the peak being unreachable and the aircraft is heading outside of the search space. In this case the aircraft cannot simply make a hard turn toward the peak because the peak will be unreachable. This is what is meant by "unreachable turn-around path". In normal conditions, if the peak is unreachable, then the control is simply relaxed, allowing the aircraft to travel forward, removing itself from the neighborhood of the peak. This is what is meant by "straight forward path". Figure 7: High level view of path planning algorithm. 12

13 Figure 8: A drop in target estimate standard deviation (squared error sense) due to target detection. Figure 9: Detection of airspace boundary crossing. Cursor-on-Target Interface The ScanEagle provides a subset of the Cursor-on-Target (CoT) interface to allow automated control and telemetry interfaces. To implement the CoT standard, a new component was added to the system to handle the interface to the ScanEagle. The interface component receives a desired sensor path from the path planning component, and turns this sensor path into a series of waypoints for the aircraft and a target for the camera. The data is sent over IP to the ScanEagle 13

14 ground station via CoT. This component also receives telemetry data from the ground station, parses and translates the data, and makes it available to the other components in the system. Simulation Environment Due to the high cost of flight testing, extensive simulations are run to maximize the likelihood of success and minimize required flight time. To provide the most accurate results, simulations are kept as close to a real scenario as possible. The system was run on the same computer used for flight tests with as much of the real system as possible. The vision detection portions were simulated as actually video data is not available indoors. Software from the manufacturer of the ScanEagle simulated the flight characteristics of the aircraft, as well as the CoT interface. Running the system with the simulated aircraft provided the most realistic testing environment that could be achieved without access to an aircraft. Each component of the system was tested for errors and was tuned to increase performance for the ScanEagle. Flight Experiments Experiment Setup Flight tests for this project were conducted at McMillan Airfield at the Camp Roberts National Guard base in California. The maintenance, launch and recovery of the aircraft was performed by EUS. As stated previously, the system developed here communicated with the ScanEagle ground station through the CoT interface over IP. All software for this system resided on a standard off the shelf desktop computer, connected through a single Ethernet cable to the ScanEagle ground station. The only additional hardware was a USB frame grabber used to digitize the video down link from the aircraft. The search area was defined by a two square kilometer rectangle approximately centered about the west end of the runway. The target was an orange inflatable kayak approximately ten feet in length and was located toward the west end of the runway. Numerous locations were tested, including brush and dirt, but ultimately the dark runway best simulated the look of the arctic sea. The aircraft flew at a height of approximately 100 m AGL. Video data was transmitted to the system through a wireless radio and USB frame grabber. Control command and aircraft telemetry were communicated through the CoT interface. 14

15 Figure 10: Search space above Camp Roberts, CA. Day 1 (December 8 th, 2009) The primary goal of the first flight day was to establish proper communication with the aircraft, understand the movement of the gimbal, and gather images of the life raft from the aircraft. Simulation testing had established a working communication link with the ScanEagle system, but some time was required to verify and adjust gimbal commands that were not testable in simulation. Several experiment trials were performed without the use of vision techniques. The purpose of these tests was to understand the behavior of the aircraft, and gimbal system. The path planning algorithms and vision control were updated and tuned in order to compensate for the behavior of the ScanEagle command center. At multiple times throughout the day, test data was gathered of the life raft. That aircraft flew manually set flight plans while the camera was focused on the target. These tests were performed at the same altitude and zoom level as would be used during detection scenarios. Data was gathered throughout the day in order to have training data in various lighting conditions. The results of the first day of testing proved positive, verifying all underlying mechanisms of the system. The C 3 UV system was able to control the aircraft and gimbal, receive and process vision data, and perform target estimation and path planning. The test data gathered that day provided the basis for the life raft classifier used throughout the experiment. 15

16 Day 2 (December 9 th, 2009) Having established that all system components were functioning, the second set of flight experiments aimed to vision algorithms developed from the test data. Using video data collected on Dec. 8 th, a computer vision classifier was developed to detect the orange kayak provided for this experiment. The real-time detection performed very well, consistently identifying the target. Some false-positive detections were seen due to features of the terrain, such as shrubs and ground effects. Under the assumed conditions of an arctic search and rescue, these features would not be present and would not contribute to the false detection rate. The unique way in which the PDF stores and updates information protects against false detections corrupting the system. The system will attempt to revisit a detection to ensure the accuracy and location of that detection. Generally, false-positives are not consistently generated by these ground anomalies and will not be redetected when the system images the area again. The nature of the PDF will remove that target likelihood and the restore the relative mass of data from before the false detection. Having established accurate vision detection, the entire system was tested using a search and rescue scenario. The system performed well, searching the space, updating the PDF and detecting the target. The localization of the target proved to be less accurate than desired due to several issues inherent in the ScanEagle system. The main source of error was the clock skew between the aircraft s system clock, and that of the C 3 UV ground based system. In order to localize a target, the video frame containing the target detection is compared to telemetry information describing the location and orientation of the aircraft and sensor at the time the image was taken. The ground location of the video frame can be extrapolated, giving the ground location of the target. Because the telemetry data is time stamped using the aircraft clock, and the video data is linked to the ground clock, and offset between the clocks will result in video frames being matched with incorrect telemetry data. The target detection is thereby localized using the incorrect set of data, and the location becomes in accurate. These errors vary based on the behavior of the aircraft at the time of detection and therefore do not result in consistent localization errors. Therefore, when the system attempts to verify the location of the target, the miscalculated target location will fail to produce an additional detection, and a new location may be found if the system detects the true target with new system error. Additional inaccuracies in the aircraft s gimbal angle readings add further error to the localization process. The results of this testing indicated a need for additional tools and techniques to compensate of the timing and measurement angles present in the system. Day 3 (December 10 th, 2009) While no flight tests were performed on this day, the issues of clock offset and data delays were investigated. Of principal interest was whether these timing issues were deterministic and consistent. A test aircraft was used for ground based tests. The gimbal was held at a fixed angle to establish a steady-state point. By commanding a quick change in the gimbal position, an event was created that would appear in both the video and data streams. The event was recorded in both data sets using the local clock. By locating these events in both data sets, the relative time of each could be compared to determine the offset between the two clock domains. Several sets of data were gathered consisting of 4 or 5 events each and testing two different aircraft. An analysis of the data indicated that the offset remained fairly consistent over time, but that jitter in 16

17 transmission delays, coupled with idiosyncrasies in the data rate from the aircraft introduced jitter on the order of several hundred milliseconds. This conclusion suggested that the offset could be accounted for, but precise synchronization would not be possible in the short term. The clocks could be compensated to account for the offset, while the jitter could be dealt with by modifying the likelihood functions returned from the vision detection component. A broader detection likelihood suggests greater uncertainty in the location of the detection and produce less defined peaks in the PDF. With broader peaks, a previous detection will not be eliminated if the same location fails to produce a new detection during the next visit. Multiple broad detections in the same general area (limited in size by the errors in the system) overlap to create a more accurate prediction of the target location. Figure 11: Timing measurements show that timing error remain fairly consistent over time. Tests 1 and 2 were conducted on a different aircraft than test 3, and shows variance from one airframe to another. Figure 12: Timing measurements, centered about the average delay, indicated clock jitter of several hundred milliseconds. 17

18 Day 4 (December 14 th, 2009) Having established the functionality of the system, the goal of these flights was to increase the performance of the localization process by compensating for the timing issues discovered in previous tests. A new technique was implemented to measure and compensate for the timing offset. Using the telemetry data, a fixed GPS point was projected on to the video stream. As the aircraft and gimbal moves, the projection of the point within the image moves accordingly. These movements also result in changes in the video data, and a correlation of the two images provides a convenient way to adjust and verify the offset. The process produced reasonable results but had to be performed several times throughout the day due to drift in the clocks. After adjusting for the timing offset and modifying the likelihood functions, the performance of the target localization increased dramatically and provided consistent results. Detections were localized to accurately enough to reinforce previous detection and provide sufficient probability of target presence and location. Several tests were performed of the full search and rescue scenario. The aircraft was tasked to search for the target within the specified area and consistently detected and located that target to enough accuracy to allow a rescue team to reach the location. Conclusions and Analysis The data fusion algorithm performed very well and demonstrated robustness in the presence of localization errors. Throughout the final flight testing week, the precision in the localization of the search target improved. We believe tit finally reached the best accuracy possible with the ScanEagle system, i.e., we could search and localize the target with an error less than 40 meters. The GPS location of the target was recorded before the experiments so as to properly evaluate the accuracy of the target location estimated by the system. The process iterated between 1) experimentally flight testing the target estimation and path planning algorithms, analyzing the behavior of the aircraft, and determining how the algorithms should be modified; and 2) implementing modifications into a new version of the algorithms. After each set of flights, the error in the estimated target location was calculated and the performance of the path planning algorithm analyzed. Figure 13 shows the evolution of the target estimation algorithm s accuracy as it was enhanced throughout the experiments. In Figure 13, each point is an average of several target detections made in several experiments for a given algorithm version. The maximum and minimum detection errors are represented by the error bars. Figure 13 shows that the error in the target estimate for the last version of the algorithm was reduced to approximately 40 meters. Figure 14 shows the path of the aircraft during one of the experiments flying the final version of the path planning algorithm. In Figure 14 you can see that the aircraft heads directly toward the areas of highest probability to find a target. In its first pass through the area, the target is not detected. The aircraft then returns to observe the new highest probability area and detects the target. Finally the aircraft comes back to track the target. The accuracy of the target position estimate was ultimately dependent on 1) the communication delay of the camera orientation and aircraft state packets (less than 5 Hz) as well as on the synchronization of video time with ScanEagle autopilot time and calibration of the camera; and 2) false detections. 18

19 The use of a gimbaled sensor instead of a fixed sensor provides increased search coverage and greater planning flexibility, but the movement of the sensor introduced additional errors in the measured camera orientation. The primary sources of error are due to the low frequency of aircraft and camera position and orientation data from the ground station. Due to the low frequency of data packets, the error in camera orientation could be as high as 10 degrees. This error is determined by the speed of the gimbal and the frequency of the telemetry packets. When applied to the projection of the image, this orientation error leads to significant localization errors. Consequently, the accuracy of the target localization estimates is degraded. Figure 13: Improvement of error in the target location estimate over successive flight experiments. 19

20 20 Figure 14: Path of the aircraft for the final version of the algorithm.

21 Future Work We find the Scan Eagle a promising platform for autonomous Search and Rescue. To realize this promise we have identified a few areas for a phase 2 of this project. These are listed below. Onboard Computing For the purposes of this experiment, all detection, planning and control was performed on a ground based computer. Transitioning this computing to the aircraft will provide faster control feedback and aid in the resolution of clock synchronization issues. Low latency access to the aircraft s clock would allow synchronization of the sensor and telemetry data for greater accuracy in localization. Furthermore, direct access to the autopilot, or different autopilot hardware, could eliminate many of the limitations imposed by CoT. This would allow greater control over the aircraft and sensor as well as minimize errors in aircraft data and lead to great improvements in performance. The use of onboard computing would also eliminate the need for a wireless video link, resulting in a clean video image, and allowing for the use of higher resolution cameras. Multiple Aircraft The focus of much of C 3 UV s work is the coordination and management of multiple aircraft. The technologies deployed for these experiments were originally developed for scenarios requiring multiple autonomous agents. Distributed Data Fusion allows multiple aircraft to participate in the same search by sharing likelihood functions. Each aircraft maintains its own PDF, but becomes aware of the observations made by other agents from likelihood functions broadcast by each plane. In this way, an area can be search more efficiently, as multiple sensors aid in the same search. Because aircraft do not rely on the information of other agents, the system is robust to failure, communication issues or the loss of an aircraft. Search and Rescue with a multi-sensor payload: The aim here is to enhance the ocean search and localization module by fusing SAR and infrared sensors for robust detection in actual ocean operating environments and higher altitude flight. The C3UV control and sensor fusion algorithms could be transitioned to multi-sensor payloads such as those developed by the University of Alaska. The multi-sensor autonomy would drive the Cursor-on-Target implementation and deliver more precise gimbal control. Likelihood Functions and Multi-Target Distributions The data fusion work use for this experiment can be expanded to include several new features. One addition would modify the likelihood functions to account for the error in gimbal orientation measurement. The certainty of the likelihoods should adapt in real-time based on the sequence of gimbal orientation measurements that have recently been received. If the orientation hasn t changed much over a couple time steps, then the likelihood should be fairly certain. However, if there is significant difference in recent orientation measurements, suggesting greater gimbal movement, then the likelihood should have less certainty 21

22 Another area of development involves extended the data fusion to generalize the probability distribution for multiple targets. This extension would make preliminary localization of targets faster. The overall distribution would be less effected false target detection. Additionally, the localization would allow searching for a potentially unspecified number of targets. This extension would require modification of the current distribution estimation algorithm and restrict the likelihood functions coverage to be within the camera image. Path Planning Algorithm Improvement The final path planning algorithm performed very robustly, however, the algorithm could be improved in a couple ways. One approach would add multiple high level behavioral modes. These high level modes would be perform explore, search, and track functions. In explore mode, the path would be optimized for searching less observed areas. In search mode, the path would be optimized for observing areas with a high probability of detection. In track mode, the path would be optimized to keep the highest probable area in view as much as possible. These modes could even be dynamically switched between a team of aircraft if multiple aircraft were used for the search. Vision-Based Object Detection The video feed used for this work was transmitted from the aircraft through an analog video transmitter which introduced various types of noise, such as color distortion, degrading the contrast of the life raft. Additionally, the pixel color of an object in each image varies greatly under different illumination conditions and varying background colors. In many cases, the color of the life raft is not a significantly distinctive color for vision detection to be effective based entirely on color filters. In the original image in Figure 2, the color of the raft is very similar to the surrounding earth. Therefore, although color is a good cue for fast prototyping and also useful in general as a supplementary cue, it should not be the ultimate main cue for object detection. For future work, advanced methods using wavelet or histogram of oriented gradients, should be used for shape-based object recognition. 22