InFuse: A Comprehensive Framework for Data Fusion in Space Robotics

Transcription

1 InFuse InFuse: A Comprehensive Framework for Data Fusion in Space Robotics June 20 th, 2017 Shashank Govindaraj (Space Applications Services, Belgium)

2 Overview 1. Motivations & Objectives 2. InFuse within the context of other OGs 3. Reference Scenarios: Planetary & Orbital 4. Taxonomy and Preliminary selection of Data Fusion Techniques 5. CDFF Architecture Core, Support and Dev 6. Sample Data Fusion Processing Compound 7. CDFF Release and Exploitation 8. Validation assets and partner competencies 9. Conclusion

3 Motivations 1. To develop a modular, portable and robust data fusion system for robotics applications in space environments. 2. Address the limited availability of data fusion (DF) frameworks for evaluation of methods and facilitate moving it to target system. 3. Applicable to planetary and orbital space applications with a spin out effect on multiple domains of research and commercial robotics.

4 Objectives Addressing Planetary and In-Orbit applications Adapted to Structured and Unstructured environments Providing Navigation (localization) and Perception (environment modeling) functions

5 Objectives Addressing all stages of data fusion and sensors data processing - from raw data to complex models, feature extractions to symbolic representations Software compatible with stringent constraints of space qualified avionics => CPU, Memory, FDIR, RTEMS etc. Providing convenient interfaces to OG2 (Autonomy Framework) and to users CDFF is being designed to remain agnostic of robotics middleware. Enabling deployment towards robotics middleware - ESROCOS, ROS, ROCK, YARP or GenoM.

6 Objectives Addressing data originating from a wide range of sensors Data Types Sensor Type Data Types Sensor Type Image Data Close-up High-Resolution Camera Angular Range Data Multi Angle Radar Image Data Light field camera (2-D images, ext. DoF) Range Data Narrow Angle Radar Extended Image Data Depth Image Data Hyper / Multispectral Camera TOF Camera Range Data Force Data Ultrasonic Sensor Contact Sensor Depth Image Data Structured Light Vision Torque Data Force/Torque Sensor Depth Image Data Visible Stereo Camera Angular Data Joint Position Encoder Depth Image Data Depth Data IR Stereo Camera 3D LIDAR Angular Pose Data Star Tracker Planar Depth Data 2D LIDAR Angular Pose Data Sun Sensor

7 InFuse within the context of other OGs

8 Reference Scenarios: Planetary Focuses on surface exploration, autonomous navigation, scientific inspection and rendezvous. Credit: Magellium Creation of an initial panorama Production and updating navigation maps (DEMs) Path planning Path execution and hazard detection/avoidance Rover self-localization (odometry / SLAM) Detection and visual servoing towards the POI Soil sample acquisition Global localisation using orbiter data Going back to lander - Visual servoing towards the lander Visual servoing of the robotics arm for sample transfer.

9 Reference Scenarios: On-orbit servicing Focuses on rendezvous for on-orbit servicing operations such as refueling, deorbiting, reconfiguration or repair of hardware modules. Detection of a target satellite from far away (it might be done by ground observation to determine its orbit) Credit: DLR Bearing tracking Initial approach 3D modeling of the satellite 3D tracking Final approach Visual servoing of the robotics arm Docking and berthing.

10 InFuse coverage of scenarios Planetary Exploration Rover localization in its environment Relative localization w.r.t a fixed or moving asset Production of DEM and navigation maps Production of a panorama Detection of point of interest On-orbit servicing Bearing only localization for approach (target position and direction estimation w.r.t. chaser / Long range) Localization w.r.t satellite for rendezvous and orbital parameter estimation (target pose estimation w.r.t. chaser close perception) Target pose estimation w.r.t. chaser / Docking (Visual servoing) 3D reconstruction of a target.

11 InFuse General Approach

12 Data Fusion Combinations Matrix

13 Taxonomy of Data Fusion Techniques Purpose of establishing a taxonomy of DF processes get a complete understanding of the nature of the entire perception layer. For fusing image data, image stitching methods using RANSAC feature matching and Bayesian filtering are available for Multiple View-Point reconstruction. To fuse images and depth data - 3D visual odometry with image based orientation determination (OR) 3D reconstruction fused with colored texturing - texture mapping method. Depth data can be fused with depth data by merging reconstructions either as raster or point cloud, or by constraining a disparity search by use of a different depth sensor.

14 Taxonomy of Data Fusion Techniques Visual data can be combined with angular pose - monocular localization fused with angular pose from an IMU or sun tracker. Contact and force/torque sensors can be fused by measurement of forces and torques at different contact points/joints to build a coarse map by just sensing surfaces and provide pose corrections(ex. collisions causing wheel slippage or while overcoming obstacles). Pose, velocity and acceleration data can be fused using a multi-sensor fusion approach combining 6D force/torque, 6D acceleration, angular velocity, and joint angle data to estimate set of inertial parameters. Fusion of inertial sensors is commonly and easily done by means of Kalman Filter variants, for noise reduction, sensor redundancy and failure detection.

15 Preliminary selection of DF techniques Category of Algorithm Selected Algorithms Optional Algorithms Category of Algorithm Selected Algorithms Optional Algorithms Optical Feature Detection Hough Transform; Harris Detector; ORB Descriptor SIFT; SURF Non- Probabilistic State Estimation Flow-Based Estimation; Fuzzy Logic; Dempster-Shafer Dezert- Smarandache Point Cloud Feature Detection Recognition & Registration Harris 3D; SHOT Descriptor ICP; RANSAC PFH/FPFH; SIFT 3D; SURF 3D Levenberg- Marquardt Probabilistic State Estimation Data Filtering and Pre- Processing Extended Kalman Filter; Unscented Kalman Filter FFT Band Filters Variance Filter; Decimation; Normalization Particle Filter/SMC - General Data Association K-Nearest Neighbours; Linear Classifier; Bayesian Classifier Dense Registration Outlier Removal Methods Interquart. Range; Mahalanobis Dist.; One Class SVM GMMs; K-Means; Min. Vol. Ellipsoid

16 CDFF Architecture CDFF Core Provides core state of the art data fusion algorithms applicable to Planetary and Orbital RIs, as a collection of ready-to-use packages. At the core of the CDFF architecture lie the Data Fusion Nodes (DFN) - core libraries with a standardized interface performing a specific subtask

17 CDFF Architecture - CDFF Support Provides necessary tools for instantiation and execution of Data Fusion Processing Compounds. These are connected via a Common Interface forming Data Fusion Processing Compounds (DFPCs). The DFPCs are regulated by the main management component, the Orchestrator, which is also handles external data product requests. The Data Product Management (DPM) tool provides data products storage and retrieval services.

18 CDFF Architecture CDFF Dev The CDFF-Dev provides the tools to develop, test, visualize, and perform analysis on data fusion products. The Common Interface are python bindings provided for the DFN common interface. The Middleware Facilitator provides the CDFF the capability to partially convert a DFPC from the designer's environment to the target RCOS. The Logs and Data Flow Management allows injecting logged data to a data fusion process in a chronological order. The Data Analysis and Performance Tools - statistical analysis tools and graphical representations for comparison of data fusion products from different DFPCs. The Visualizer is responsible for presenting graphical representations of the different data products (e.g. 2D/3D plots, maps, camera images) The Calibration Tools provide a framework for automatic (re-)calibration.

19 Data Fusion Node Common Interface Methods to receive and output data Methods for configuration of internal DFN parameters (optional) Methods for handling of Meta-Information of the processed data (optional). Spatiotemporal information (transformation and timestamp) Source information (sensor + parameters) Processing steps gone through Allows to capture information regarding processing requirements. Allows to capture information regarding timing requirements (e.g. periodicity, synchronicity) Provides optional interfaces for control of the internal states by the orchestration

20 CDFF Infrastructure

21 Sample DFPC

22 CDFF Release and Exploitation Open Source Strategy: A final version of the CDFF will be made publicly available as open source. A second package will be released as a legacy support for upcoming Space Robotics Challenge projects with potentially some license restricted subparts Exploitation: InFuse provides software components, hardware support and contribution to the robotics community. Incorporate additional scenarios - precision landing and autonomous navigation on comets. Commercial: Self-localisation for autonomous vehicles (aerial, terrestrial, underwater, humanoid) or high-precision pose estimation and tracking, e.g. for industrial manipulation tasks.

23 CDFF Release and Exploitation

24 Planetary / Orbital Partners & Competencies Coordinator; avionics requirements; middleware; avionics EGSE; outreach; Lead partner for Orbital RI; Orbital Perception & Navigation; Orbital EGSE (internal testing); Lead partner for DF framework supporting tools; open-source release; Perception & Navigation; Lead partner for Planetary RI; essential DF functions; Navigation & Perception; exploitation; Lead partner for essential DF functions; Navigation & Perception; datasets collection; dissemination; Planetary Perception & Navigation; DF framework supporting tools; Planetary EGSE (internal testing)

25 Planetary RI (internal testing) Orbital RI (internal testing) Assets and facilities to validate InFuse DLR s OOS simulator and RDV simulator with lighting conditions DFKI s HIL RDV simulator SPACEAPPS Leon 4 target platform, DLR s & STRATH s FPGA DFKI s Asguard 4 CNRS-LAAS fully equipped outdoor platform CNES SEROM mars yard (through Magellium) STRATH s rover SPACEAPPS mobile manipulator

26 Conclusion 1. Infuse CDFF provides techniques to estimate location and model the surroundings of a robotic platform in order to support the planning and execution. 2. Aims to provide tools to perform fast prototyping with core data fusion libraries before their final integration on the target platform. 3. Bring improvements in the parameterisation of DF nodes and will probably enable an autonomous reconfiguration of DFPCs

27 Project Funded under the H2020 PERASPERA SRC Space Robotics Thank you