Intelligent Robots for Handling of Flexible Objects IRFO Vision System Andreas Jordt Multimedia Information Processing Institute of Computer Science University Kiel
IRFO Vision System Overview 2) Sensing of Material Properties Estimating Material Parametes by Haptic / Visual Sensing Tasks: Object Scanning Manipulation Density Simulation Visual Sensing by Range Camera 1) Sensing of object shape Physical Modelling Shape Prediction 3) Modeling and Predicion Execution Learning 4) Manipulation and Learning Object Tracking 2
IRFO Vision System Overview Depth/Color Sensors Scanning Stage Interaction Stage Conveyor 3
Scanning Stage Depth/Color Sensor: Kinect 640 x 480 Pixel @ 30Hz Framerate Simultaneous Color and Depth Video 4
Scanning Stage: 3D Model Generation Task: Acquire a 3D model Generate 3D surface from depth image Remove conveyor surface (Optional): Colorize 3D surface 5
Scanning Stage Deformation Estimation Grasp calculation Database Deformation Tracking 6
Deformation Tracking Task: Determine Object Deformation Define deformation model Find model parameters describing the deformation 7
Deformation Tracking: Deformation Model NURBS Surfaces A continuous surface function defined by a set of 3D control points. 8
Deformation Tracking: Deformation Model Define a vector positions. NURBS Surfaces, which contains all 3D control point Register the 3D mesh from the scanning stage to such a NURBS surface. 9
Registration Consider every 3D point of the given model Find the closest point on the surface for each vertex Describe every object point relative to the NURBS x d z y u v 10
1 1
With a deformable 3D model at hand, how do we determine the correct deformation parameters for every frame in a give sequence? Render deformation guesses and compare them to the input data Define a fitness function to measure the difference between the results of a deformation and the input images.. 12
The Fitness Function How is such a function defined? Color fit (normalized luminance) Depth fit Surface penalty (Regularization) 13
Fitness function evaluation Controlpoints C For every object point: Recursive NURBS evaluation Transformation to D Projection into D Apply lense distortion Compare to depth value C Transformation to C Projection into C W D Apply color lens distortion Normalize and compare color Evaluate distance points Calculate distances 14
Fitness function evaluation D C Build NURBS Cache Transform CPs Undistort Images Warp C into D Pre-normalize Color For every object point: Recursive NURBS evaluation Transformation to D Projection into D Apply lense distortion Compare to depth value Transformation to C Projection into C Apply color lens distortion Normalize and compare color Evaluate distance points Calculate distances 15
CMA-ES Optimization problem has e.g. 48 dimensions for 4 x 4 control points High dimensional optimization problem with no derivation available => Covariance Matrix Adaptation (Evolution Strategy) Samples (Individuals) per Iteration: 4 + floor ( 3 * log(dim) ) e.g. 10 for dim=100 16
18
19
20
Thanks Thank you for your attention. 21