Vision-based GNC and Decision Making for an Unmanned Helicopter. presented by Frank Thielecke

Transcription

1 Vision-based GNC and Decision Making for an Unmanned Helicopter presented by Frank Thielecke

2 Outline The ARTIS Family System Concept for More Autonomy - On-Board Components for Supervisory Control - Route and Task Planner Vision-based Navigation to compensate GPS failures - Positioning by Ground Feature Tracking Vision-based Collision Avoidance - Stereo Camera and on-board Image Processing Summary Vision-based GNC and Decision Making for an Unmanned Helicopter > Slide 2

3 Autonomous Rotorcraft Testbeds for Intelligent Systems Future of ARTIS? Vision-based GNC and Decision Making for an Unmanned Helicopter > Slide 3

4 ARTIS - The VTOL UAV Family maxiartis Main Rotor Diameter: 3 m Max Weight : 25 kg Engine: 1 Turbine 4,5 kw 4 kg Navigation and Flight Control System Max 6 kg Experimental Payload ARTIS miniartis Main Rotor Diameter: 1,9 m Empty Weight: 6 kg Engine: 1 Two-Stroke, 15 cm³, 2 hp 4 kg Navigation and Flight Control System 2 kg Experimental Payload Main Rotor Diameter: 54 cm Empty Weight: 370 g Engine: Brushless DC 200 g Navigation and Flight Control System 50 g Experimental Payload Vision-based GNC and Decision Making for an Unmanned Helicopter > Slide 4

5 Components of the ARTIS Flight System Magnetometer GPS-Antenna Digital camera Sonar Flight computer Power supply Vision computer Telemetry GPS & IMU Vision-based GNC and Decision Making for an Unmanned Helicopter > Slide 5

6 General System Setup Autonomy GPS Navigation & Flight Control System Data Link I Data Link II Data Link I Data Link II GPS Reference Ground Computer RS-232 Serial Ground Control Station R/C RC Interface RC Receiver RC Transmitter Actuators On-board Avionics Safety Pilot Standard setup requires two operators (GCS, Safety Pilot) Two redundant data links (serial, TCP/IP Ethernet) R/C electronics and the custom avionics have independent power supplies to ensure operational safety Vision-based GNC and Decision Making for an Unmanned Helicopter > Slide 6

7 Hardware-in-The-Loop Simulation Environment Real-Time Simulation Computer dspace Sensor Simulation Flight State ARTIS Flight Dynamics Simulation Model Control Inputs Actuator Simulation Sensor Data Actuator Data Sensor Interface Sensor Data Navigation Filter State Estimate Flight Controller Trajektory Planning Control Input Actuator Interface Flight Control Computer QNX Real-Time OS Telemetry Commands Ground Control Software MAESTRO User Interface, Mission Planning GCS Computer Windows XP Vision-based GNC and Decision Making for an Unmanned Helicopter > Slide 7

8 Haptic Multi-modal Interaction Sidesticks Stirling Dynamics - Active Sidestick with adaptive force feedback - Haptic Interaction with UAV system and sensor-/vision-system - Support of Easy Handling Voice Input - Direct voice recognition and input - Definition of a syntax Displays iobjects - Visualization and new displays - Improve automation and situation awareness Vision-based GNC and Decision Making for an Unmanned Helicopter > Slide 8

9 Man-Machine-Interaction in Highly Automated Systems Recognize - Act Cycle Situation awareness Planning Situation awareness Planning Monitoring Supervisior? Supervision Decision making Execution Execution UAV-Agent Operator Vision-based GNC and Decision Making for an Unmanned Helicopter > Slide 9

10 Onboard Planning and Decision Making System What does autonomy stand for? - Dictionary decision-making authority or independent Autonomous Automatic 3 2 Mission Manager (Planners, Supervisor) Guidance System 1 Control System Autonomy in UAVs: - Machine-based decision making capabilities to solve problems under uncertainty at runtime, while minimizing the dependency on external (human) decisions. Vision-based GNC and Decision Making for an Unmanned Helicopter > Slide 10

11 High Level System Architecture Digital 4D World Model 4D Mission Planning and Execution Monitoring Mission Manager Supervisory Control System Sensor Fusion Object Recognition Object Tracking Obstacle Detection + Avoidance Agent Sensor status and reactive commands Health Management System Diagnosis Fault Detection Behavior Command Sequence Task Planner Route Planner Module Manager Behavior Command Sequence Resources Cooperation Communication Sequence Control System Event Handler Behavior Pool Behavior Synthesis Direct Commands High Level Commands User generated Behavior Sequ. Low Level Commands Human Operator UAV-Agent Adaptation Reconfiguration Flight Controller (Standard Mode) Remote Control Vision-based GNC and Decision Making for an Unmanned Helicopter > Slide 11

12 Behavior Synthesis for Sequence Control System Generating mission plans (sequences) using predefined behaviors: Task-specific Planner (Mission Manager component) Surveillance Search Explore Transport Sequencing behaviors into a plan (waypoints and tasks) global time Photo Photo Take Off Fly To Hover Hover To Hover Fly To Hover Land t 0 t 1 t 2a t 2b t 3 t 4a t 4b Elementary capabilities (Behaviors) Take off Hover Hover To Fly To Pirouette t 5 t 6 t 7 Example: Mosaiking mission. Landing/ Take off Hover Fast Flight Controller Configuration (Task IDs) Vision-based GNC and Decision Making for an Unmanned Helicopter > Slide 12

13 Sequence Control Sequence Control System State Chart Model - Executes plans (i.e. behavior sequences) - Nested automation describing states relations and transitions - Interactive remote control (joystick input, manually generated behavior sequences) Task-specific Planner (Mission Manager component) Surveillance Search Explore Transport Sequencing behaviors into a plan (waypoints and tasks) Photo Photo globa l time Take Off Fly To Hover Hover To Hover Fly To HoverLand t 0 t 1 t 2 t 3 a 2 Elementary capabilities (Behaviors) b t t 4 a t 4 b t 5 t 6 t 7 Take off Landing/ Take off Hover Controller Configuration (Task IDs) Hover Hover To Fly To Pirouette Fast Flight Remote Control Vision-based GNC and Decision Making for an Unmanned Helicopter > Slide 13

14 High Level System in Action Vision-based GNC and Decision Making for an Unmanned Helicopter > Slide 14

15 Mission Manager: 3D Path Planning Module Vision-based GNC and Decision Making for an Unmanned Helicopter > Slide 15

16 Path Planning Concept #1 Path planning is performed in three steps Step 1: Roadmap generation - A number of sample points in free space are chosen - Pairs of waypoints with a collision-free path (e.g. a straight line) are connected - The resulting structure is a graph, called Probabilistic Roadmap (PRM) in which nodes are the sample points and edges indicate collision-free paths Step 2: Roadmap search - Uses known graph search algorithms (Dijkstra, A*) - Tends to be quick - Multiple searches within the same graph are possible: step 1 needs not be repeated Step 3: Path smoothing - Waypoints are removed if there is a collision-free path between its neighbors - Splines may be fitted over a path consisting of straight line segments Vision-based GNC and Decision Making for an Unmanned Helicopter > Slide 16

17 Path Planning Concept #2 Known issues - Random samples may fail to find points within narrow passages, limiting the algorithm's performance in constrained environments - A method called bridge test is then used to generate samples within those areas Vision-based GNC and Decision Making for an Unmanned Helicopter > Slide 17

18 Sample results of 3D Path Planning PRM average building runtime (1000 samples): 1.8s Graph search average runtime: 48 ms, Smoothing runtime: 9 ms Top: planned path (orange) and smoothed one (red). Bottom: underlying PRM Vision-based GNC and Decision Making for an Unmanned Helicopter > Slide 18

19 Task Planning Problem Allocate tasks to multiple agents - Agent and task heterogenity - Agent cooperativeness capacity - Close-to-optimal requirement Missions accomplished earlier With respect to economy (e.g. energy economy) Take Photos Artis 1 Capabilities: Photo camera 15 min flight endurance Search Area Start positions Artis 2 Capabilities: Photo camera 30 min flight endurance Waypoint Artis 3 Capabilities: Live Video 15 min flight endurance Vision-based GNC and Decision Making for an Unmanned Helicopter > Slide 19

20 Centralized Task Planning and Distribution Group of UAVs (Agents) Agent 1 Agent 2 Agent 3 Agent 4 Multi-Agent Mission Input Tasks Search Task 1 Search Task 2 Mosaiking 1 Live Video 1 Task Refinement Task Assignment (Ant Algorithm) Fast run-time: e.g. 1.4s for 3 agents w/ 20 tasks Task Sharing Planner Output Mission Schedule (Group Plans) Agent 1 Agent 2 Agent n T 11 T 12 T 1k T 21 T 22 T 2l T n1 T n2 T np Vision-based GNC and Decision Making for an Unmanned Helicopter > Slide 20

21 Task Assigment using an Ant Algorithm Used for finding optimal task ordering for each agent Each iteration returns best ordering found Optimization Algorithm returns best ordering so far Initialization at algorithm start Task Ordering Best of Iteration Update Performance saturation at algorithm end Best of Run Vision-based GNC and Decision Making for an Unmanned Helicopter > Slide 21

22 Task Planner Example Scenario Shared task (Two UAV s) Search Task Artis 1 Artis 2 Artis 3 Timeline display Search task En-route Waypoint-based task Vision-based GNC and Decision Making for an Unmanned Helicopter > Slide 22

23 Fault Detection, Reconfiguration & Decision Vision-based GNC and Decision Making for an Unmanned Helicopter > Slide 23

24 Vision-based Navigation to compensate GPS failures Vision-based GNC and Decision Making for an Unmanned Helicopter > Slide 24

25 Vision-based Positioning Positioning relative to ground features Error accumulation in helicopter position Assumptions Feature-rich ground texture Features lie in horizontal ground plane ( ) g g h g c h c h P, k = F k V, k + B r r T T r r Static feature position Sufficient light conditions Vision-based GNC and Decision Making for an Unmanned Helicopter > Slide 25

26 Positioning using Vision c CCD xs x c c 1 CCD rs = ys = y f 1 f CCD CCD c c c c x y rv = zv rs = zv,, 1 f f T Vision-based GNC and Decision Making for an Unmanned Helicopter > Slide 26

27 Ground Feature Tracking Monocular digital camera, monochrome, frame rate 30 Hz, resolution 640 x 480 pixel Lucas-Kanade algorithm E ε Feature Selection 2 Lx( x, y) Lx( x, y) Ly( x, y) 2 (, ) (, ) (, ) px+ ω p x y+ ωy = x= px ωx y= p L y ωy x x y Ly x y Ly x y Tracking p + ω px+ ωx y y I J ( d, ) ( (, ) (, )) 2 x dy = L x y L x+ dx y+ dy x= p ω y= p ω x x y y ε ( d ) d = 0 Vision-based GNC and Decision Making for an Unmanned Helicopter > Slide 27

28 Zoom Vision-based GNC and Decision Making for an Unmanned Helicopter > Slide 28

29 Navigation filter (EKF) State equation: Measurement equation: Extended state vector: Measurement vector: Inputs: ( ) xt &() = f xt (), ut () + Fwt () ( ) z() t = h x(), t u() t + Gv() t g g g g g g x = x b = x, y, z, u, v, w, q0, q1, q2, q3, T bax, bay, baz, bp, bq, br ; b& = 0 g g g g g g z = xgps, ygps, zgps, ugps, vgps, wgps, Hx, Hy, Hz,1.0, i i N N hson, xcam, ycam,..., xcam, y Cam ; i = 1,..., N u = pm, qm, rm, axm, aym, azm T T Vision-based GNC and Decision Making for an Unmanned Helicopter > Slide 29

30 Latency / Data Synchronization Real-time data processing Stable control requires permanent state feedback with low latency Data synchronization and accounting for image data latency in the navigation filter Sensor Latency Data rate IMU < 10 ms 100 Hz Image data < 150 ms Hz ( k ) ( k, k, d) xˆ = x% + K z y% k k k k y% = h x% u τ k GPS position GPS velocity Magnetometer Sonar ~ 80 ms ~ 120 ms < 50 ms < 20 ms 20 Hz 15 Hz 10 Hz Vision-based GNC and Decision Making for an Unmanned Helicopter > Slide 30

31 Test in Hardware-In-The-Loop Simulation Simulation Computer Flight Control Computer Vision Computer Sensor emulation Sensor raw data Time stamp Sensor driver Time stamp Camera driver Camera State Sensor data Synchronization UDP Sensor data Helicopter model Control surface deflection Missionplanner Navigation filter State estimation Flight controller Commands SPICE Actuator model Actuator signal Actuator driver MCP DIP Vision-based GNC and Decision Making for an Unmanned Helicopter > Slide 31

32 HITL-Results Flight path WP5 (0, 10, -9) - z, m WP4 (10, 10, -6) WP3 (10, 0, -6) WP2 (0, 0, -5) WP1 (0, 0, -3) y, m 0 True Nav.filter x, m -5 0 Vision-based GNC and Decision Making for an Unmanned Helicopter > Slide 32

33 Flight Test Results (1) Flight path Attitude 10 - z, m y, m 0 GPS ref Nav sol x, m Phi, deg Theta, deg Psi, deg Nav sol time t, s Vision-based GNC and Decision Making for an Unmanned Helicopter > Slide 33

34 Flight Test Results (2) x, m y, m z, m GPS ref 10 Nav sol. 5 Position time t, s z, m x, m y, m Vision On error time t, s (Error < 1.5 m, 10 %) Vision-based GNC and Decision Making for an Unmanned Helicopter > Slide 34

35 Flight Test Results (3) Velocity Vision On ug, m/s vg, m/s wg, m/s GPS ref Nav sol time t, s ug, m/s vg, m/s wg, m/s error time t, s (Error < 0.9 m/s) Vision-based GNC and Decision Making for an Unmanned Helicopter > Slide 35

36 Flight Test Video Vision-based GNC and Decision Making for an Unmanned Helicopter > Slide 36

37 Vision-based Tracking of a Moving Target Development and evaluation of pattern tacking methods Test in HITL simulation incl. cameras Flight tests with ARTIS helicopter Vision-based GNC and Decision Making for an Unmanned Helicopter > Slide 37

38 Vision-based Collision Avoidance Vision-based GNC and Decision Making for an Unmanned Helicopter > Slide 38

39 Collision Avoidance UAVs need to be integrated in public airspace for civil markets The predominant use of UAVs is in unknown areas under exceptional conditions Typical scenarios for UAVs require the capability for low level flights and operation in urban terrain. The availability of mature collision avoidance systems is crucial for UAV operations. Vision-based GNC and Decision Making for an Unmanned Helicopter > Slide 39

40 Threat Analysis Ground Objects Controlled Flight Into Terrain (CFIT) Near Miss (Mid Air Collision) Runway Incursion Vision-based GNC and Decision Making for an Unmanned Helicopter > Slide 40

41 Requirements for Collision Avoidance Systems A pilot performs sensing, prediction, judgment, path planning and reaction A UAV component must be simply at least as good as a pilot to be certified and accepted Vision-based GNC and Decision Making for an Unmanned Helicopter > Slide 41

42 Concepts active passive cooperative non cooperative (emitting) T-CAS Flarm Mode S Radar Ladar (emission free) Databases TV IR small mass small power consumption cheap multi purpose Vision-based GNC and Decision Making for an Unmanned Helicopter > Slide 42

43 Stereo Camera-based Collision Avoidance Vision-based GNC and Decision Making for an Unmanned Helicopter > Slide 43

44 Stereo Camera Image Acquisition Synchronization of image pairs Correction of lens distortions Minor image differences due to different perspective Vision-based GNC and Decision Making for an Unmanned Helicopter > Slide 44

45 HITL Simulation Environment Environment for development, testing, tuning, and analysis Same camera hardware and image processing software as in flight test Original interfaces Stereo vision simulation Simulation of flight characteristics Simulated or real controller for avoid maneuver Vision-based GNC and Decision Making for an Unmanned Helicopter > Slide 45

46 Depth Image Computation (based on simulated data) Measure for different object positions in image pairs: disparity Disparity can be converted to distance value for each object Here: near objects appear bright Vision-based GNC and Decision Making for an Unmanned Helicopter > Slide 46

47 Recommended evasion direction Projection of planned flight path plus safety distance into depth image red cross: planned direction without collision avoidance green cross: recommended direction, regarding safety distance and old direction Vision-based GNC and Decision Making for an Unmanned Helicopter > Slide 47

48 Sensor Visualization and Object Recognition Angle and range For several sensors and camera systems Allows better interpretation of virtual flight tests Processed depth images Projected in visualization Allows assignment of recognized objects Vision-based GNC and Decision Making for an Unmanned Helicopter > Slide 48

49 Summary The ARTIS Family Base Agent 2 WP 2 Agent 1 WP 1 System Concept for More Autonomy WP 3 Search Area 1 Agent 1 Agent 2 Mission Manager with Modules for Supervisory Control Route and Task Planner Vision-based Navigation to compensate GPS failures Vision-based Collision Avoidance using Stereo Cameras Vision-based GNC and Decision Making for an Unmanned Helicopter > Slide 49

50 Contact Information Prof. Dr. Frank Thielecke Head of Systems Automation Department German Aerospace Center (DLR) Institute of Flight Systems Lilienthalplatz Braunschweig Germany phone: fax.: frank.thielecke@dlr.de Vision-based GNC and Decision Making for an Unmanned Helicopter > Slide 50