Towards an Improved Understanding of Human In-hand Manipulation Capabilities

Transcription

1 Towards an Improved Understanding of Human In-hand Manipulation Capabilities Learning from Demonstration Eugen Richter Faculty of Mathematics, Informatics and Natural Sciences Technical Aspects of Multimodal Systems 20. December 2016 E. Richter 1 / 33

2 Outline 1. Introduction 2. Capturing human motion 3. Integration of object feedback 4. Data analysis 5. The road ahead E. Richter 2 / 33

3 Introduction In-hand manipulation In-hand manipulation of objects is one of the most complex and expressive biological capabilities of the human being It requires: Stabilization of manipulated object through contact Effective control of a large number of degrees of freedom In-hand manipulation can be described as repositioning of a grasped object within a hand while maintaining a stable grasp at any given time E. Richter 3 / 33

4 Introduction Manipulation sequence In-hand manipulation consists of a sequence of stable grasps Transition between grasps must be possible (synthesis vs. demonstration) Dynamic nature of in-hand manipulation can lead to unstable states of object grasp during transition Adaptive strategy required A few of the categories of in-hand manipulation movements are: Rolling Sliding Finger gaiting E. Richter 4 / 33

5 Introduction Robotic hands E. Richter 5 / 33

6 Introduction Challenges Unstable states during grasp transitions High dimensionality (at least 20 DOF just for the hand) Non-linearity and high coupling of finger motion Generalization of capabilities to objects of different size Generalization to unknown objects Very difficult to acquire good tactile feedback Human finger motion is fast A lot of possibility for occlusion Real-time requirements E. Richter 6 / 33

7 Introduction Potential applications Robotics / Automation Dexterous object manipulation Human-Robot Interaction Manufacturing / Assembly Medical Prosthetics Exoskeletal prosthetics Assistance E. Richter 7 / 33

8 Introduction Natural hand motion Hand/finger motion synthesis is a highly active research field Primary focus is grasp stability However, not every stable grasp is natural or efficient Collecting data from human demonstration makes sense Human in-hand manipulation capabilities are highly evolved Captured motion sequences constrain the problem search space Recorded data is mostly collision free Good starting point for grasp planning, grasp optimization (adjustment) E. Richter 8 / 33

9 Introduction Curse of dimensionality The human hand has 27 DOF (including the wrist) Robotic hands usually have less Not possible to extract all degrees of freedom (especially of the thumb) Natural human hand motion is highly coupled allowing dimensionality reduction E. Richter 9 / 33

10 Introduction Trends in related work Lots of research on grasping, not so much on in-hand manipulation Despite availability of dexterous robot hands several research focused on more limited mechanisms Less fingers Underactuated systems Some focus on data from the human, others on synthetic data Available results are often presented using task-specific manipulators Applicable in strictly controlled environments Real-time requirements are still hard to fullfil Difficult to adapt to uncertain tasks or unstructured environments E. Richter 10 / 33

11 Capturing human motion Motion capture Marker-based, passive (optical) Usually done with retro-reflective marker objects attached to parts of human body But anything else is good too if the image processing approach can handle it Less invasive, yet still restrictive Marker-based, active (optical) Actively driven LEDs, pulsed or non-pulsed Unique pulse sequences simplify identification, non-pulsed LEDs processed similar to passive markers More invasive due to wiring and potentialy larger markers E. Richter 11 / 33

12 Capturing human motion Motion capture Markerless (optical) Based on 2D/3D information from stereo vision setups or RGB-D sensors Processing usually requires an existing dataset Non-invasive, but computationaly more involved Other means of motion capture: Magnetic systems (e.g. Polhemus, Ascension) Mechanical systems (e.g. CyberGlove) Inertial systems (e.g. Xsens) E. Richter 12 / 33

13 Capturing human motion Motion capture issues Occlusion and accuracy (relative, absolute) Ambient conditions (lighting), noise Sensor resolution, range of motion Calibration accuracy Broken trajectories (Occlusion) Phantom markers (Reflection) Blooming (two-for-one), flickering (intensity, volume boundaries), merging Crossover (close passing markers) Tracking continuity E. Richter 13 / 33

14 Capturing human motion Motion capture issues Marker identification problem. Loss of marker data (Noise/Occlusion). E. Richter 14 / 33

15 Capturing human motion PhaseSpace Impulse X2E Active marker-based motion capture/tracking system Very high frame capture rate (up to 960 Hz) Detector FOV is close to 60 Almost instant identification Tracking of markers and rigids Real-time filtering, cleaner data Not very user friendly API largely undocumented E. Richter 15 / 33

16 Integration of object feedback Role of tactile feedback Touch is one of the main senses used by human beings for interaction with their environment [WJ84] have shown lack of touch to introduce great difficulty of maintaining stable grasps Tactile information is indispensable for adaptive manipulation behavior Detection of object slippage Handling of deformable objects Lack of tactile feedback in today s industrial/service robots restricts their use to highly structured environments E. Richter 16 / 33

17 Integration of object feedback On-hand sensing Sensing touch/force using sensors on the hand is difficult HANDLE project data glove [HA1] Sensing patches (Tekscan) [HA1] E. Richter 17 / 33

18 Integration of object feedback Instrumented objects Estimation of contact points with object is also difficult Idea is not new - but there is not much research Different shapes resembling everyday objects IMU for orientation information Markers (AprilTag, LEDs,... ) for position tracking Tactile sensors (e.g. FSR, optical?) for force sensing Force-Torque sensor(s) for wrench information between parts... E. Richter 18 / 33

19 Integration of object feedback Shadow Can Cylindrical object (from the HANDLE project) Upgraded with IMU for orientation and relative motion 40x FSR sensors (mostly non-linear response, hysteresis) Coarse resolution of the surface area Close enough to cups, bottles, object handles, canisters, etc. Too large for true in-hand manipulation experiments E. Richter 19 / 33

20 Integration of object feedback Custom-built objects Everyday objects (e.g. kitchen) are difficult to customize Primary issues Integration of force sensing elements, F/T sensors Tracking marker integration 3D-printed objects present a convenient option Generically-shaped objects should be a good start Cylinder, cube, tetrahedron,... Good spatial resolution is key Difficult issue on curved surfaces 3D-printed object [WA16] E. Richter 20 / 33

21 Data analysis What to do with all the data? Dataset built from series of in-hand manipulation sequences Pose information of arm, hand, fingers and objects Finger contact point information Forces applied at contact points Perform hand/finger motion segmentation Search for manipulation actions, motion primitives Combine motion primitives for a full manipulation task e.g. through a graph-based approach Map motion primitives to robotic hand Current focus: Automatic segmentation (and annotation) E. Richter 21 / 33

22 Data analysis Automatic motion segmentation Inspired by [ZTH08] and [KTWZ10] and [VKK14] In-hand manipulation sequence as input Temporal segmentation in an unsupervised fashion Identify distinct motion (manipulation) actions in recorded sequence e.g. reach, pre-grasp, initial grasp, pick-up, object rotation (in-hand), put-down, release Classify segments inbetween actions as transitions Decompose manipulation actions into motion primitives Identification of manipulation actions and motion primitives using the concept of self-similarity E. Richter 22 / 33

23 Data analysis Segmentation into actions Input: Sequence M as points p i,..., p m, each represented by F = (f 1,...f N ) F is a vector of multi-modal features (pose information, contact forces, etc.) k-d tree search for local neighbors of each p i in feature space S i is the resulting neighborhood of p i with distance given in frames Sets S i are converted into a self-similarity matrix (SSM) SSM is an M x M matrix containing local neigborhood distances for each frame E. Richter 23 / 33

24 Data analysis Self-similarity Self-similarity matrix representation is used to determine action boundaries Region growing applied to self-similarity matrix through: Forward step starting at M1,1 Backward step starting at Mn,n Areas in between actions are considered transitions Example: SSM / Region growing [VKK14]. E. Richter 24 / 33

25 Data analysis Segmentation into motion primitives Construction of a neighborhood graph G M for the set of all neighborhoods S Every neighbor represents a single node Connections between nodes are established using dynamic time warping Each diagonal in the SSM translates to a connected component in the graph Adding one start/end node connected to first/last reduces warping path search to shortest path problem Each connected component in G M represents a local similarity and is thus considered a motion primitive E. Richter 25 / 33

26 Data analysis Resulting data Motion primitives can be clustered Determines frequency Indicates temporal relationship Cluster graph G C with motion primitives as nodes Search for valid warping paths between pairs m i and m j of motion primitives Construction of neighborhood graph using neighbors in the rectangle spanned around m i and m j Search for shortest path from top to bottom of spanned submatrix Shortest path satisfying inclusion criteria (length, slope) results in edge between nodes Each strongly connected component represents a cluster E. Richter 26 / 33

27 The road ahead Outlook: Motion tracking PhaseSpace tracking system is installed and operational Main LED driver is currently causing problems (Re-) Calibration is non-trivial Tests show accuracy issues outside the sweet-spot of the calibrated capture volume Standard markers too large for finger placement Custom calibration object would be awesome (in the making) So would be a custom alignment object (in the making) E. Richter 27 / 33

28 The road ahead Outlook: Motion tracking MultiLED boards required to drive LEDs directly Reverse engineering of the control sequence not very promising Placing markers on skin is very inconvenient Hand glove as thin and robust as possible Current choice: cheap running gloves Has some stretch, fits well, LEDs can be sewn onto the surface Thin enough, touch feels normal Potential issue: fingertip friction E. Richter 28 / 33

29 The road ahead Outlook: ROS integration Possible to use PhaseSpace API directly (e.g. standalone applications) C/C++, Python and even a Unity package Current state of ROS integration allows streaming of 3DOF marker and 6DOF rigid data API offers data filtering - highly undocumented (not integrated yet) Calibration must be done with PhaseSpace tools No plans to integrate into ROS - student project maybe? E. Richter 29 / 33

30 The road ahead Outlook: Instrumented objects Shadow Coke-Can too large, but still interesting for initial tests Needs to be upgraded due to unstable data from the on-board controllers Build a smaller version of the Coke-Can with higher surface resolution Minimal size is constrained by size of LED driver Placing AprilTags for pose tracking is difficult Possible to use further sensors (Kinect, Stereo vision) for object pose reconstruction E. Richter 30 / 33

31 The road ahead Outlook: Dataset recording ROS package for hand glove tracking is in the making Uses hand model from my diploma thesis Current work is focused on filtering of data dropout due to occlusion Measures to offset the joint marker positions based on data from medical studies Ultimate goal for the dataset is to make it public Several datasets [HBLS16] exist but are often not dealing with in-hand manipulation (at finger motion level) are unstructured (e.g. change of task domain, or even the sensors for capturing of hand/finger motion) E. Richter 31 / 33

32 The road ahead Outlook: Data analysis Presented approach more or less implemented, but not tested Trying to segment old data from my thesis did not perform well due to strong occlusion yielding hard to understand SSMs Exploit multi-modal nature of the capture framework to minimize data dropout Additional information obtained from the instrumented object should improve rate of discrimination during segmentation of motion primitives Will need subjects for experiments :) E. Richter 32 / 33

33 The road ahead References [HA1] [HBLS16] [KTWZ10] [VKK14] [WJ84] [ZTH08] HANDLE project (eu/fp7), Y. Huang, M. Bianchi, M. Liarokapis, and Y. Sun, Recent datasets on object manipulation: A survey, Big Data, B. Krueger, J. Tautges, A. Weber, and A. Zinke, Fast local and global similarity searches in large motion capture databases, 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation, A. Voegele, B. Krueger, and R. Klein, Efficient unsupervised temporal segmentation of human motion, 2014 ACM SIGGRAPH/Eurographics Symposium on Computer Animation, G. Westling and R. S. Johansson, Factors influencing the force control during precision grip, Experimental Brain Research 53 (1984), no. 2, F. Zhou, F. De La Torre, and J. K. Hodgins, Aligned cluster analysis for temporal segmentation of human motion, IEEE Conference on Automatic Face and Gestures Recognition, E. Richter 33 / 33