Tactile-Visual Integration for Task-Aware Grasping. Mabel M. Zhang, Andreas ten Pas, Renaud Detry, Kostas Daniilidis

Transcription

1 Tactile-Visual Integration for Task-Aware Grasping Mabel M. Zhang, Andreas ten Pas, Renaud Detry, Kostas Daniilidis

2 Molyneux s Question William Molyneux s famous question to John Locke in 1688: Suppose a Man born blind, and now adult, and taught by his touch to distinguish between a Cube, and a Sphere of the same metal, and nighly of the same bigness, so as to tell, when he felt one and t other; which is the Cube, which the Sphere. Suppose then the Cube and Sphere placed on a Table, and the Blind Man to be made to see. Quaere, Whether by his sight, before he touch d them, he could now distinguish, and tell, which is the Globe, which the Cube. 2

3 Development of Touch Streri 1986, 1987, 1988, 2000, 2003, 2004 Newborns, 2- and 5-month-olds touch-only shape discrimination vision-touch transfer Meltzoff month olds; oral touch Cowey 1975 Rhesus monkeys food/sand in the dark Icons by Freepik, Smashicons from flaticon.com 3

4 Sensory Substitution & Active Exploration Bach-y-Rita 1969, White 1970 Also: active exploration 4

5 Why touch? Perception Manipul Darkness Transparency Complex task Underwater Inside a bag Slippag Smoke Reflection Adaptation Occlusion Nonprehe Image sources journalstar.com whaleshark.org.au brede-art.com alibaba.com kjpargeter from Freepik goir/shutterstock.com IROS 2017 Manip. Challenge Hang

6 Broadly-Related Work Transplant vision techniques to touch Schneider 2009 BoW Pezzementi 2011 BoW, moments, SIFT Strub 2014 moments Luo 2015 tactile SIFT Luo 2016 ICP, BoW Yuan 2017 CNN Calandra 2017 CNN Hollis 2018 compress 8

7 Intuition of End-effector pose Approach: Object poseindependent; Geometric; Holistic; Sparse contacts (low cost $) Contact point (x, y, z) 1 Contact point (x, y, z) 3 Contact point (x, y, z) 2 Triangle (l 0, l 1, a 0 ) Zhang et al. IROS

8 Histogram of Triangles Build 3D histogram of triangle parameters Move hand; Repeat Zhang et al. IROS 2016 Classifier 10

9 Avg accuracy over 100 train-test splits Avg accuracy over 100 train-test splits Histogram Parameters Mesh Cloud Accuracy for Various Histogram Parameters Physics Simulation Accuracy for Various Histogram Parameters Number of bins per 3D histogram dimension Highest: 90.3% from (l1, l2, a2) and 10 bins Number of bins per 3D histogram dimension Highest: 73.9% from (a1, a2, l0) and 20 bins Zhang et al. IROS

10 Qualitative Contact Clouds Zhang et al. IROS

11 Pairwise Distances Least similar objects Most similar objects (bottle) In-between objects Zhang et al. IROS

12 Intuition of Approach: Objects share similar local features; there s distribution observation z 1 observation z 2 Actions are associated with local geometric observations wrist pose p 1 action a Zhang et al. IROS 2017 wrist pose p 2 14

13 Active Pose Selection Poses selected at test time (recognized in 2-9 moves): Zhang et al. IROS

14 Active Pose Selection Zhang et al. IROS

15 On a Continuum Manipulator Simulation (recognition under 3 wraps) Real (no sensors) Mao*, Zhang*, et al. IROS

16 Enclosure contacts only Drawbacks Lederman & Klatzky 1987 exploratory procedures 19

17 Visuotactile Integration (2x3) x 3 TakkTile barometric sensors 20

18 Problem: Grasp success from vision + touch, with Task semantics (beyond pick and place) 21

19 Visuotactile Representation 1. Spatial Correspondence? 2. Leverage state of the art in vision? Input CNN 0 1 FC Grasp Success Conv 22

20 Related Work Yuan CVPR 2017 Fabric material classification Kinect + camera-based touch Calandra CoRL 2017 Grasp success probability RGB + camera-based touch Varley arxiv D CNN on voxels Depth + tactile point cloud 23

21 Visuotactile Representation 3. Semantic task? Implementation from Detry et al. IROS

22 Visuotactile Representation Correspondence: Camera frame Output: Grasp success 25

23 6DOF Tactile Grasp Collection Random Scene Point Cloud RGB View Off-the-shelf Grasps Tactile Simulation Grasp and Lift 26

24 Tactile Input & Grasp Label Good Grasp Bad Grasp Contact readings Grasp Success Label Good Lift Slipped Lift 27

25 Task Label Binary task labels in CAD (Detry et al. IROS 2017): pour handover Transform contact points to object frame Task label from CAD Grasp 28

26 Tactile Heatmap Visualization Successful grasps Unsuccessful grasps 29

27 Dex-Net 2.0 Planar Grasps Crop to local grasp; center row of pixels aligned to grasp axis Successful grasps Unsuccessful grasps Mahler et al. RSS

28 Simulated Tactile Heat Map 2D normal + thickness Successful grasps Unsuccessful grasps 0 thickness normal

29 Results on Dex-Net Adv-Synth Subset 32

30 Ongoing Work Currently collected 10,000 grasps 10x? Adv-Synth has 189,300 grasps, 1/6 of Dex-Net

31 Challenges Visuotactile representation Sim-to-real transfer Sensory input Physics Rusu 2016 James 2017 Inoue

32 Future Work Courtesy of K. Queen 35

33