Tactile-Visual Integration for Task-Aware Grasping Mabel M. Zhang, Andreas ten Pas, Renaud Detry, Kostas Daniilidis
Molyneux s Question William Molyneux s famous question to John Locke in 1688: Suppose a Man born blind, and now adult, and taught by his touch to distinguish between a Cube, and a Sphere of the same metal, and nighly of the same bigness, so as to tell, when he felt one and t other; which is the Cube, which the Sphere. Suppose then the Cube and Sphere placed on a Table, and the Blind Man to be made to see. Quaere, Whether by his sight, before he touch d them, he could now distinguish, and tell, which is the Globe, which the Cube. 2
Development of Touch Streri 1986, 1987, 1988, 2000, 2003, 2004 Newborns, 2- and 5-month-olds touch-only shape discrimination vision-touch transfer Meltzoff 1979 1-month olds; oral touch Cowey 1975 Rhesus monkeys food/sand in the dark Icons by Freepik, Smashicons from flaticon.com 3
Sensory Substitution & Active Exploration Bach-y-Rita 1969, White 1970 Also: active exploration 4
Why touch? Perception Manipul Darkness Transparency Complex task Underwater Inside a bag Slippag Smoke Reflection Adaptation Occlusion Nonprehe Image sources journalstar.com whaleshark.org.au brede-art.com alibaba.com kjpargeter from Freepik goir/shutterstock.com IROS 2017 Manip. Challenge Hang 2016 7
Broadly-Related Work Transplant vision techniques to touch Schneider 2009 BoW Pezzementi 2011 BoW, moments, SIFT Strub 2014 moments Luo 2015 tactile SIFT Luo 2016 ICP, BoW Yuan 2017 CNN Calandra 2017 CNN Hollis 2018 compress 8
Intuition of End-effector pose Approach: Object poseindependent; Geometric; Holistic; Sparse contacts (low cost $) Contact point (x, y, z) 1 Contact point (x, y, z) 3 Contact point (x, y, z) 2 Triangle (l 0, l 1, a 0 ) Zhang et al. IROS 2016 9
Histogram of Triangles Build 3D histogram of triangle parameters Move hand; Repeat Zhang et al. IROS 2016 Classifier 10
Avg accuracy over 100 train-test splits Avg accuracy over 100 train-test splits Histogram Parameters Mesh Cloud Accuracy for Various Histogram Parameters Physics Simulation Accuracy for Various Histogram Parameters Number of bins per 3D histogram dimension Highest: 90.3% from (l1, l2, a2) and 10 bins Number of bins per 3D histogram dimension Highest: 73.9% from (a1, a2, l0) and 20 bins Zhang et al. IROS 2016 11
Qualitative Contact Clouds Zhang et al. IROS 2016 12
Pairwise Distances Least similar objects Most similar objects (bottle) In-between objects Zhang et al. IROS 2016 13
Intuition of Approach: Objects share similar local features; there s distribution observation z 1 observation z 2 Actions are associated with local geometric observations wrist pose p 1 action a Zhang et al. IROS 2017 wrist pose p 2 14
Active Pose Selection Poses selected at test time (recognized in 2-9 moves): Zhang et al. IROS 2017 15
Active Pose Selection Zhang et al. IROS 2017 16
On a Continuum Manipulator Simulation (recognition under 3 wraps) Real (no sensors) Mao*, Zhang*, et al. IROS 2017 18
Enclosure contacts only Drawbacks Lederman & Klatzky 1987 exploratory procedures 19
Visuotactile Integration (2x3) x 3 TakkTile barometric sensors 20
Problem: Grasp success from vision + touch, with Task semantics (beyond pick and place) 21
Visuotactile Representation 1. Spatial Correspondence? 2. Leverage state of the art in vision? Input CNN 0 1 FC Grasp Success Conv 22
Related Work Yuan CVPR 2017 Fabric material classification Kinect + camera-based touch Calandra CoRL 2017 Grasp success probability RGB + camera-based touch Varley arxiv 2018 3D CNN on voxels Depth + tactile point cloud 23
Visuotactile Representation 3. Semantic task? Implementation from Detry et al. IROS 2017 24
Visuotactile Representation Correspondence: Camera frame Output: Grasp success 25
6DOF Tactile Grasp Collection Random Scene Point Cloud RGB View Off-the-shelf Grasps Tactile Simulation Grasp and Lift 26
Tactile Input & Grasp Label Good Grasp Bad Grasp Contact readings Grasp Success Label Good Lift Slipped Lift 27
Task Label Binary task labels in CAD (Detry et al. IROS 2017): pour handover Transform contact points to object frame Task label from CAD Grasp 28
Tactile Heatmap Visualization Successful grasps Unsuccessful grasps 29
Dex-Net 2.0 Planar Grasps Crop to local grasp; center row of pixels aligned to grasp axis Successful grasps Unsuccessful grasps Mahler et al. RSS 2017 30
Simulated Tactile Heat Map 2D normal + thickness Successful grasps 0.1 + Unsuccessful grasps 0 thickness normal - -0.1 31
Results on Dex-Net Adv-Synth Subset 32
Ongoing Work Currently collected 10,000 grasps 10x? Adv-Synth has 189,300 grasps, 1/6 of Dex-Net 2.0 33
Challenges Visuotactile representation Sim-to-real transfer Sensory input Physics Rusu 2016 James 2017 Inoue 2017 34
Future Work Courtesy of K. Queen 35