Autonomous Vehicles: Research, Design and Implementation of Autonomous Vehicles Research Group - GPVA http://www.eletrica.unisinos.br/~autonom Tutorial page: http://inf.unisinos.br/~osorio/palestras/cerma07.html html Dr. Fernando S. Osório - Applied Computing Post-Grad. Program PIPCA Dr. Christian R. Kelber - Electrical Engineering / Computer Eng. Dr. Cláudio R. Jung - Applied Computing M.Sc.. Program PIPCA M.Sc. Farlei Heinen - Computer Engineering B.Sc. (Director) Grupo de Pesquisas em Veículos Autônomos Autonomous Vehicles Research Group - Unisinos 1
Introduction CERMA 2007 / Tutorial Veículos Autônomos Inteligentes Robotic: Automatons, Mobile Robots and Autonomous Robots Perception, Action, Locomotion e Communication and Intelligence Intelligent Vehicles Technologies for Vehicle Automation pyramid Intelligent of Autonomous Vehicles : Computational Architectures Simulation of Autonomous Vehicles Computer Vision Practical Applications Agenda 2
CONTROL: : Computational Architectures => From where do I start? Modeling and Simulation Models: - Sensorial Models - Actuator Models - Kinematics Models - Environment Models - A.I. Models (Path Planning, Agents,...) Simulation: - Validate models - Test robustness - Improve design : Computational Architectures 17
CONTROL: : Computational Architectures => From where do I start? Modeling and Simulation Scientific American January 2007 : Computational Architectures 17
CONTROL: : Computational Architectures C1 C2 C3 C4 Sensorial Models Kinematics Models C0 M1 M2 C5 Sensorial Model: C7 C6 Sonar Infrared Kinematics Model: Differential φ Radar, Compass, Odometer Aeckerman Y Aeckerman θ X θ = V / L * Sin (Φ) X = V * Cos (Φ) * Cos (θ) Y = V * Cos (Φ) * Cos (θ) : Computational Architectures 18
CONTROL: : Computational Architectures Sensorial Models Kinematics Models Robotic : * Reactive * Deliberative * Hierarchical * Hybrid Environment Maps * Building Maps * Path Planning * SMPA - Sense Model Plan Act Problems: * Complex tasks * Avoid Obstacles: Static / Mobile - Unexpected obstacles * Robot actual position estimation - Where am I? : Computational Architectures 17
CONTROL: : Computational Architectures Complexity... P.Bessièrre * Action Planning * Ability to Perceive the Environment * Ability to Decide * Ability to Act * High Level Tasks Planning * Reaction: Sensorial-Motor * Estimate Actual and Future States * Adaptation and Learning * Robustness * Unexpected Situations => From where do I start??? : Computational Architectures 20
CONTROL: : Computational Architectures Complexity... : Computational Architectures 21
CONTROL: : Computational Architectures Complexity... Simplify! How? : Computational Architectures 22
CONTROL: REACTIVE Architecture Complexity... Simplify! How? MIT - OCW Reactive: Sensorial-Motor Integration Able to Act Able to Perceive the Environment Able to React Reactive Architecture 23
CONTROL: REACTIVE Architecture Reactive: Sensorial-Motor Integration Reactive S0 S1 M1 S2 S3 M2 S4 S5 IF S1 < Threshold and S2 < Threshold and S3 < Threshold and S4 < Threshold THEN Action (Go_Forward) S7 S6 IF S1 < Threshold and S2 < Threshold and S3 > Threshold and S4 > Threshold THEN Action (Turn_Left) IF S2 > Threshold and S3 > Threshold and S2 > S3 and S1 > S4 THEN Action (Turn_Right) Sensorial-Motor: Perceive => Act Reactive 24
CONTROL: REACTIVE Architecture Reactive: Sensorial-Motor Integration Reactive Robotic Lawn Mowers - Toro imow - Husqvarna Auto Mower - Automower Electrolux Sensorial-Motor: Avoid Obstacles, Wall Following, Wander Electrolux Trilobite Robotic Vacuum Cleaner ZA1 http://www.onrobo.com/reviews/at_home/vacuum_cleaners www.onrobo.com/reviews/at_home/vacuum_cleaners/ Reactive 25
CONTROL: REACTIVE Architecture Reactive: Sensorial-Motor Integration Reactive Sensorial-Motor: - Avoid Obstacles - Wall Following - Wander Simple behaviors Robustness? Complex tasks? Reactive 26
CONTROL: DELIBERATIVE Architectures Deliberative: Planning + Action Deliberative SIMROB (2D) - Map - Configuration Space - Visibility Graph - Optimized Path (Dijkstra) Robotic Arm: Pre-defined paths Deliberative Architecture 27
CONTROL: DELIBERATIVE Architectures Deliberative: Planning + Action Deliberative Tarefas Complexas... Robustez? Imprevistos? Ambiente pouco conhecido? Deliberative Geometric Map based Navigation: Planning: Graph+Dijkstra, A* Grid based Navigation: Planning: A* 28
CONTROL: HIERARCHICAL and HYBRID Architectures Combining: Deliberative + Reactive Hierarchical and Hybrid Hierarchical and Hybrid Hierarchical : - Layers - Priorities - Information Exchange Figures From: Brooks, R. A. MIT A.I. Memo 864 Sept. 1985 Brooks - Subsumption Architecture 29
CONTROL: HIERARCHICAL and HYBRID Architectures Building the Environment Map: SMPA - SENSE / MODEL / PLAN / ACT MODEL Hierarchical and Hybrid S E N S E PLAN: AStar Dijkstra ACT! Hierarchical and Hybrid Sebastian Thrun / CMU 30
CONTROL: Simple HYBRID Architectures Hybrid Farlei Heinen PLAN: Dijkstra ACT & ReACT Hybrid 31
CONTROL: Simple HYBRID Architectures Hybrid Farlei Heinen References: SEVA2D / SEVA3D Autonomous Vehicle Parking TASK PLANNING & CONTROL: Finite State Automata (FSA) Artificial Neural Net (ANN) ACTION: Sense, Act React (change state) SEVA-A (Automaton) Farlei Heinen SEVA-N (Neural) Farlei Heinen Fernando Osório Luciane Fortes Milton Heinen Publications: SBRN 2002 WCCI 2006 Hybrid 32
CONTROL: Simple HYBRID Architectures Hybrid SimRob3D Simulation SEVA3D 3D World Kinematics: Estimation of Position and Orientation Perception: Robot Sensor Model Simulation Motor Actions Sensorial Information Commands Sensors : SEVA3D-A (FSA) SEVA3D-N (Neural) Visualization Hybrid 32
CONTROL: : Computational Architectures Sensorial Models Kinematics Models Robotic : * Reactive * Deliberative * Hierarchical * Hybrid Environment Maps * Building Maps * Path Planning * SMPA - Sense Model Plan Act Problems: * Complex tasks * Avoid Obstacles: Static / Mobile - Unexpected obstacles * Robot actual position estimation - Where am I? Robust Hybrid 33
System Task Execution PROBLEMS: * Avoid Obstacles - Known Obstacles - Unknown Obstacles (static / no movement) - Unknown Obstacles (dynamic / moving objects) * Positioning - How to determine the exact actual position of the robot? - How to maintain the control of exact position after displacement? - Error and Imprecision: Move forward / Rotate Robust Hybrid 34
PROBLEMS: * Avoid Obstacles CERMA 2007 / Tutorial System Task Execution - Known Obstacles - Unknown Obstacles (static / no movement) - Unknown Obstacles (dynamic / moving objects) Robust Hybrid 35
PROBLEMS: * Positioning System Task Execution - How to determine the exact actual position of the robot? - How to maintain the control of exact position after displacement? - Error and Imprecision: Move forward / Rotate Robust Hybrid 36
Robust Hybrid COHBRA / HyCAR Robot [SimRob3D] Sensors Actuators Layers Behaviors Environment Representation Maps Positioning Estimator (Monte Carlo) Sequencer Polygonal Modules Grid Path Planning Topological and Semantic Shared Memory Robust Hybrid COHBRA e Híbrido de Robôs Autônomos HyCAR - Hybrid for Autonomous Robots 37
Robust Hybrid COHBRA / HyCAR [SimRob3D] Simulation using SimRob3D Robust Hybrid 38
Robust Hybrid COHBRA / HyCAR [SimRob3D] Simulation using a static environment Position estimation based on Monte Carlo Localization Method Robust Hybrid 39
Robust Hybrid COHBRA / HyCAR [SimRob3D] Simulation using a static environment Position estimation based on Monte Carlo Localization Method Robust Hybrid 40
Robust Hybrid COHBRA / HyCAR [SimRob3D] Simulation using a static environment Environment was changed related to the original map Internal robot representation is different from actual world configuration Robust Hybrid 41
Robust Hybrid COHBRA / HyCAR [SimRob3D] Simulation using a dynamic environment (mobile obstacles) Robust Hybrid 42
Robust Hybrid COHBRA / HyCAR [SimRob3D] Position estimation based on Monte Carlo Method: Robot was moved, starting in a new and unknown position Robust Hybrid 43
Robust Hybrid COHBRA / HyCAR [SimRob3D] Virtual Environment: 3D Realistic Environment SimRob3D Simulation Tool Robust Hybrid 44
Robust Hybrid COHBRA / HyCAR [SimRob3D] Virtual Environment: 3D Realistic Environment SimRob3D Simulation Tool Robust Hybrid 45
Robust Hybrid COHBRA / HyCAR [SimRob3D] SEVA 3D SimRob3D Simulation Tool Robust Hybrid 46
Intelligent Autonomous Robots and Vehicles << Intelligence >> * Task and Actions Planning * Ability to Perceive the Environment * Ability to Decide * Ability to Act * High Level Tasks Planning * Reaction: Sensorial-Motor Integration * Estimate Actual and Future States: Environment + Behavior = Interaction * Adaptation and Learning * Robustness: Unexpected Situations Next steps... GPVA 47
Intelligent Autonomous Robots and Vehicles << Intelligence >> * Task and Actions Planning * Ability to Perceive the Environment * Ability to Decide * Ability to Act * High Level Tasks Planning DARPA Challenge - Desert (2004, 2005) * Reaction: Sensorial-Motor Integration DARPA Challenge - Urban (2007) * Estimate Actual and Future States: Environment + Behavior = Interaction * Adaptation and Learning * Robustness: Unexpected Situations Next steps... GPVA 47
Intelligent Autonomous Robots and Vehicles << Intelligence >> * Task and Actions Planning * Ability to Perceive the Environment * Ability to Decide Computational Vision * Ability to Act * High Level Tasks Planning * Reaction: Sensorial-Motor Integration * Estimate Actual and Future States: Environment + Behavior = Interaction * Adaptation and Learning * Robustness: Unexpected Situations Next steps... Computational Vision 48
Computational Vision CERMA 2007 / Tutorial Intelligent Autonomous Robots and Vehicles << Intelligence >> Path following: - Follow Me, Lane Follow Avoid danger situations: going out of the track - Lane Detection Obstacle detection: pedestrians, cars, etc Traffic signs detection and recognition Visual Navigation (Based on Images) Computational Vision 49
Intelligent Autonomous Robots and Vehicles << Intelligence >> Computational Vision Lane Follow Lane Departure Detection Follow Me Computational Vision 50
Image Database: Path defined by a sequence of image CERMA 2007 / Tutorial Visual Navigation Reference ImagemAtual NCC (match) PróximaImagem Next Imagemcapturada Robot acquired image pelorobô Navigation based on Monochromatic Images [Jones et al. 1997] Algorithm: NCC Normalized Cross-Correlation Correlation Visual Navigation 51
Visual Navigation [Matsumoto et al. 1996] Matlab Code IR: Reference Image ICR: Image Captured by the robot [Righes 2004, 2005] Visual Navigation 52
Visual Navigation [Righes 04] Visual Navigation 53
Visual Navigation Mobile Robot Localization and Mapping with Uncertainty using Scale-Invariant Visual Landmarks Stephen Se, David Lowe, Jim Little (UBC, CA) Algorithm: SIFT Reference Int. Journal of Robotics Research Vol. 21, No. 8, August 2002, pp. 735-758, Visual Navigation 54
Omnidirectional Cameras CERMA 2007 / Tutorial Visual Navigation Visual Navigation 55
Aerial Visual Navigation Vision System for Unmanned Aerial Vehicles Correlation: Satellite image and Helicopter Results... Not good at all! Visual Navigation 55
Aerial Visual Navigation Vision System for Unmanned Aerial Vehicles Referential Correlation in the Crossing Point Using helicopter only images Very Good Match! Visual Navigation
Vehicle Visual System Vision system used to identify traffic signs [F.Alves 2003] RGB Color HSV Color [H]SI Color based sign segmentation Artificial Neural Network Recognition Visual Systems
Multiple Vehicles: Fire fighting squad Planning, Navigation, + Strategy, Cooperation Visual Systems
http://www.eletrica.unisinos.br/~autonom Google: Veículos Autônomos