Object Action Complexes (OACs, pronounced oaks)

Transcription

1 CoTeSys ROS Fall School on Cognition enabled Mobile Manipulation, 4. November, 2010, Munich Object Action Complexes (OACs, pronounced oaks) Tamim Asfour and PACO-PLUS Consortium Karlsruhe Institute of Technology, Institute for Anthropomatics, Germany INSTITUTE FOR ANTHROPOMATICS, DEPARTMENT OF INFORMATICS KIT University of the State of Baden-Wuerttemberg and National Research Center of the Helmholtz Association

2 Building Humanoids BuildingHumanoids = BuildingHuman Centered Technologies Assistants/companions for people in different ages, situations, activities and environments in order to improve the quality of life Key technologies for future robotic systems Experimental platforms to study theories from other disciplines 2

3 Ultimate goals Integrated complete 24/7 humanoid robot systems rich with sensorimotor capabilities and able to act and interact in humancentered environments and to perform a variety of tasks as an indispensible requirement to implement cognitivecapabilities capabilities in technical systems Reproducible complete humanoid systems in terms of mechanical design, mechantronics, hardware and software architecture Autonomous humanoid systems 3

4 Humanoid KIT ARMAR, 2000 ARMAR-II, 2002 ARMAR-IIIa, 2006 ARMAR-IIIb, 2008 Collaborative Research Center 588: Humanoid Robots Learning and Cooperating Multimodal Robots (SFB 588) Funded by the German Research Foundation (DFG: Deutsche Forschungsgemeinschaft ) karlsruhe.de/ 4

5 ARMAR I and ARMAR II HURO 1999, ICAR199 Humanoids 2000, 2001 IROS 2003, 2004 ISER 2004 STAR2005 First demonstrator of the SFB 588 Demo at CEBIT

6 ARMAR-IIIa and ARMAR-IIIb 7 DOF head with foveated vision 2 cameras in each eye 6 microphones 7-DOF arms Position, velocity and torque sensors 6D FT-Sensors Sensitive Skin 8 DOF Hands Pneumatic actuators Weight 250g Holding force 2,5 kg 3 DOF torso 2 Embedded PCs 10 DSP/FPGA Units Holonomic mobile platform 3 laser scanner 3 Embedded PCs 2 Batteries Fully integrated autonomous humanoid system 6

7 ARMAR IV 63 DOF 170 cm 70 kg 7

8 Three key questions Grasping and manipulationin in human centered and open ended environments Learning through Observation of humans and imitation of human actions Interaction and natural communication SFB 588, Karlsruhe 8

9 Grasping is Integral Part for Humanoids in the Real lworld 9

10 Grasping and manipulation system Offline Analysis Execution asd Grasp planning Global model database Object models Vision system (Object recognition and localization) Grasp set for each object Arm path planner Grasp selection and execution 10

11 Object recognition and localization Colored objects (IROS 2006, IROS 2009) Segmentation by color Appearance-based recognition using a global approach Model-based generation of view sets Combination of stereo vision and stored orientation information for 6D pose estimation Textured objects (Humanoids 2006, IROS 2009) Recognition using local features Calculation of consistent features with respect to the pose of the object using the Hough transform 2D-localization using image point correspondences 6D pose estimation using stereo vision Integrating Vision Toolkit: 11 Correspondences between ee learned view and view in scene

12 Object Modeling KIT Object model database Generation of point 3D clouds using a laser scanner Post processing with triangulation results in high quality meshes Stored in several file formats (Open Inventor, VRML, Wavefront) Generation of multiple object views using stereo cameras Developed within the German Humanoid Robotics Project (SFB 588) 12 Web database

13 Object representation 13

14 Offline grasp analysis Feasible grasps for every object are computed offline and stored together with the object models. dl A grasp is defined by: Grasp type Grasp starting point Approaching direction Hand orientation Simulation environment: GraspStudio and OpenGrasp: opengrasp.sourceforge.netsourceforge net Hook Cylindrical Spherical Pinch Tripod 14

15 Execution: Position based Visual Servoing Differential Kinematics + θ = J (θ ) x θ Joint Controller Object Pose Δx Target Pose Target Object Position & Orientation (Visual Estimation) θ Hand Pose + - Joint Sensor Values Hand Orientation (Kinematic Approximation) Hand Position (Visual Estimation) Perception x target Target Pose x tcp Hand Pose Wrist Pose θ + = J (θ ) x x t = = x x t vision t + 1 t + 1 tcp x kinematic x + t kinematic t tcp 15

16 Interactive tasks in kitchen environment Object recognition and localization Vision based grasping Hybrid position/force control Vision based selflocalisation Combining force and vision for opening and closing door tasks Learning new objects, persons and words Collision free navigation Audio visual user tracking and localization Multimodal humanrobot dialogs Speech recognition for continuous speech Software framework MCA2: 16

17 Bimanual grasping gand manipulation Stereovision for object recognition and localization VisualServoing for dual hand grasping Zero force control for teaching of grasp poses Humanoids 2009 Loosely coupled dual armtasks arm Tightly coupled dual arm armtasks 17

18 Limitations and shortcuts Problem: Representational differences between high level AI planning and low level robotics/vision. Continuous Robotics vs. Discrete AI Planning Previous efforts have resulted in largely ad hoc solutions. Claim:Object Action Complexes (OACs) can be used as a lingua franca to bridge this representational divide. Formalize the requirements for an artificial system to approach some level of cognitive complexity. Grounding of entities in the motor sensory domain Data structures, which can be used on all levels EU project PACO PLUS: plus.org org 18

19 Main Paradigm Objects and Actions are inseparably intertwined Visually based object recognition fails Visual information is sparse and limited Activity involving the object decreases the uncertainty about theobject's nature considerably! CMU Graphics Lab Motion Capture Database / 19

20 Intuitions on Object Action Complexes Affordances (J.J. Gibson) Objects affords actions Object Action Complexes (OACs) Actions define the meaning of Objects and Objects suggest Actions OACs are associations of objects and affordances Affordances can be expressed by STRIPS rules comprising: Preconditions and Deletions/additions 20

21 Object Perspective OAC Thing & Attributes Movement Thing & changed Attributes Objects & Actions have arisen if this process is successful Code-similarity emerges only through the fact that both codes describe the same physical entity Human Neuronal code Robot code OACs are code-independent Success? Measured against: Consistency with world Novelty, Drives, etc. 21

22 Action Perspective OAC movement & Attributes Thing movement & changed Attributes Objects & Actions have arisen if this process is successful Code-similarity emerges only through the fact that both codes describe the same physical entity Human Neuronal code Robot code OACs are code-independent Success? Measured against: Consistency with world Novelty, Drives, etc. 22

23 Object, Action and Task Perspective Object Perspective Container (full) tilting Container (empty) Action Perspective Grasp Pen Pinch Grasp Success? Success? Consistency w. world Novelty, Drives, etc. Consistency w. world Novelty, Drives, etc. Grasp Bottle Power Grasp Task Perspective Success? drink Consistency w. world Novelty, Drives, etc. Grasp Cup Handle Grasp put on table Success? Grasp Cup From-top Grasp Consistency w. world Novelty, Drives, etc. Success? Consistency w. world Novelty, Drives, etc. 23

24 Formalization of OACs 24

25 Six design principlesp Attributes [P1] Aspects of the world required for and effected by the OAC Downward congruency between levels Attributes are measured and this measurement can be noisy or wrong Prediction [P2] Expectation of change through the OAC Usually defined on subspaces of the total state space Execution [P3] Every OAC must be performable in the real world Requires an embodiment 25

26 Six design principlesp Evaluation [P4] Evaluating the differences between the predicted state and the actual observed state Downward congruency must guarantee that the results are interpretable at the OAC s own level Learning [P5] State and action representations are dynamic entities that are refined and extended by learning Measuring Reliability [P6] Embodied physical experiences ( experiments ) with actions, predictions, and outcomes deliver the basis for learning 26

27 Formal definition of an OAC Krüger et al A Formal Definition of Object Action Complexes and Examples at Different Levels of the Processing Hierarchy (availableat at plus.org) plus.org) 27

28 Functions associated with OACs In addition to id,t and M, the following class functions need to be defined for each OAC. Furthermore, we define a system level and other functions for each OAC 28

29 The learning cycle 29

30 Limitations and shortcuts Objects Complete model knowledge (shape, color, texture) Only visual representation is used How to learn new objects? How to acquire multi sensory representations of objects? Actions Kinematics based approaches as place holders for learned primitive actions. How to learn new actions? How to adapt actions to new situations? How to chain different actions to achieve complex tasks? 30

31 How to relax the limitations in our scenario? Autonomous Exploration: Visually guided haptic exploration Visual object exploration and search Learning actions of objects Coaching and Imitation Learning from Observation Goal directed Imitation 31

32 Hand: available skills Direct/Inverse Kinematics Position/force control Detection of contact and objectness Assessment of deformability and object size Failed grasp no object Precision grasps: Distance between fingertips Power grasps: Distance between fingertips and palm 32

33 Tactile Object Exploration Potential field approach to guide the robot hand along the object surface Oriented 3D point cloud from contact data Extract faces from 3D point cloud in a geometric feature filter pipeline Parallelism Minimum face size Face distance Mutual visibility Association between objects and actions (grasps) Symbolic grasps (grasp affordances) RCP TCP 33

34 Examples: Bottle 34

35 Visually guided haptic exploration on ARMAR In simulation Physics extension for Open Inventor/VRML modeling of complex mechanical systems Modeling of virtual sensors VMC-based inverse kinematics 35

36 Active Visual Object Exploration and Search Generation of visual representations through exploration Application of generated representations in recognition tasks. 36

37 Visual exploration of unknownobject 37

38 Visual exploration of unknownobject Background object and hand object segmentation Generation of different views through manipulation 38

39 Representation Aspect Graph Multi view appearance based representation Each node corresponds to one view Edges describe neighbor relations Feature Pool Compact representation of views with prototypes Grouping based on visual similarity Vector quantization through incremental clustering 39

40 Active visual search Active Search Object search using perspective and foveal camera Scene memory Integration of object hypotheses in an ego centric representation ICRA 2010 Humanoids 2009 ICRA

41 Learning by Autonomous Exploration: Pushing Learning of actions on objects (Pushing) Learning relationship between point and angle of push and the actual movement of an object y world y obj angle of contact point of contact x obj actual object movement (x vel, y vel, Φ vel ) x world Use the knowledge in order to find the appropriate point and angle of push in order to bring an object to a goal 41

42 Pushing within the OAC concept Requirements: Initial motor knowledge: the robot need to know how to move the pusher along straight lines Assumed visual processing: segment and localize objects The pushing OAC oac push learns the prediction i function updatet, which is implemented as a feedforward neural network. This network represents a forward model for object movements that have been recorded with each pushing action. 42

43 Pushing for grasping Object independent pushing (generalization across objects). Learning relationship between point and angle of push and the actual movement of an object Direct association between the binarized object image and the response of the object with respect to the applied pushing action. Use the knowledge in order to find the appropriate point and angle of push in order to bring an object to a goal Joint work with Damir Omrcen and Ales Ude; (Humanoids 2009) 43

44 Learning of actions on objects Pushing for grasping y world y obj x obj Learning relationship between point and angle of push and the actual movement of an object angle of contact t point of contact actual object movement (x vel, y vel, Φ vel ) x world Use the knowledge in order to find the appropriate point and angle of push in order to bring an object to a goal Joint work with Damir Omrcen and Ales Ude; (Humanoids 2009) 44

45 How to relax the limitations in our scenario? Autonomous Exploration Visually guided haptic exploration Visual object exploration and search Learning actions of objects Coaching and Imitation Learning from Observation Goal directed Imitation 45

46 Learningg from human observation Reproduction Observation Ge e a at o Generalization 46

47 Stereo based 3D Human Motion Capture Capture 3D human motion based on the image input from the cameras of the robot s head only Approach Hierarchical Particle Filter framework Localization of hands and head using color segmentation and stereo triangulation Fusion of 3d positions and edge information Half of the particles are sampled using inverse kinematics Features Automatic Initialization 30 fps real time tracking on a 3 GHz CPU, 640x440 images Smooth tracking of real 3d motion Integrating Vision Toolkit: 47

48 Reproduction Tracking of human and object motion Visual servoing for grasping Generalisation? 48

49 Action representation Hidden Markov Models (HMM) Humanoids 2006, IJHR 2008 Extract key points (KP) in the demonstration Determine key points that are common in multiple demonstrations (common key points: CKP) Reproduction through interpolation between CKPs Dynamic movement primitives (DMP) ICRA 2009 Ijspeert, Nakanishi & Schaal, 2002 Trajectory formulation using canonical systems of differential equations Parameters are estimated using locally weighted regression Spline based representations fifth order splines that correspond to minimum jerk Humanoids 2007 trajectories to encode the trajectories Time normalize the example trajectories Determine common knot points so that all example trajectories are properly approximated. Similar tovia point, key points calculation. 49

50 Action representation using DMPs canonical system: nonlinear function: transformation system: τ u = α u P i ψ i(u) ( ) w i u f(u) = P i ψ i(u) τ v = K(g x) Dv +(g x 0 )f τẋ = v h i (u c i ) 2 ψ i (u) =e f target = w i Locally Weighted Learning K(g y)+dẏ + τÿ g x 0 canonical system u nonlinear function f target transformation system y, ẏ, ÿ 50

51 Action representation using DMPs canonical system: nonlinear function: transformation system: τ u = α u P i ψ i(u) w i u f(u) = P i ψ i(u) τ v = K(g x) Dv +(g x 0 )f τẋ = v h i (u c i ) 2 ψ i (u) =e x 0,g canonical system u nonlinear function f f(u) ) f target w i Locally Weighted Learning transformation system ŷ, ˆy, ẏ, ˆ ẏ, ÿˆ ÿ 51

52 Learning from Observation Library of motor primitives Markerless human motion tracking Object tracking Action representation Dynamic movement primitives for generating discrete and periodic movements Adaptation of dynamic systems to allow sequencing of movement primitives i i Associating semantic information with DMPs sequencing of movement primitives Planning 52

53 Learning from Observation Periodic movements: Wiping Extract the frequency and learn the waveform. Incremental regression for waveform learning Joint work with Andrej Gams and Ales Ude, Humanoids 2010 Ude, Gams, Asfour, Morimoto. Task-Specific Generalization of Discrete and Periodic Dynamic Movement Primitives. IEEE Trans. on Robotics, vol. 26, no. 5, pp , October,

54 Learning from Observation Human grasp recognition, mapping and reproduction Joint work with Javier Romero, Danica Kragic; Humanoids

55 API of the ARMARs Robot interface (API) allows access to the robot s sensors and actors via C++ variables and method calls Skills: implemented atomic capabilities of the robot SearchForObject Grasp Place HandOver Open/Close door Tasks: combination of several skills for a specific purpose Bring object from Put object on Stack objects Software framework MCA2: Scenario Task 1 Task 2 Task m Skill 1 Skill 2 Skill n Robot Interface Robot Hardware 55

56 Connecting High level Task Planning to Execution PKS planner (STRIPS-like planner ) Joint work with Ron Patrick and Mark Steedman, University of Edinburgh 56

57 OAC Relation to Existing Concepts and Representations OACs combine three elements: The object (and situation) orientedoriented concept of affordance (Gibson 1950; Duchon, Warren and Kaelbling 1998; Stoytchev 2005; Fitzpatrick et al. 2003; Sahin et al. 2007; Grupen et al. 2007, Gorniak and Roy 2007); The representational and computational efficiency for planning and execution monitoring (the original Frame Problem) of STRIPS rules (Fikes and Nilsson 1971; Shahaf and Amir 2006; Shahaf, Chang and Amir 2006; Kuipers et al. 2006; Modayil and Kuipers 2007); The logical clarity of the situation/event calculus (McCarthyand and Hayes 1969; Kowalski and Sergot 1986; Reiter 2001; Poole 1993; Richardson and Domingos 2006; Steedman 2002) Machine Learning is involved in all three 57

58 What OACs add OACs are congruent between levels of representation. This is necessary for them to be both grounded and modular (Brooks 1986). This is crucial ilto plan monitoring i to deal with ihnon determinism and failure. It follows that OACs are initially defined from the bottom up. but with more innate structure assumed than in other work (e.g. the EU projects MACS and RobotCub). and with higher levels subsequently proposing new composite macro actions top down. In a robot system the formalization of OACs enforces that actions are always grounded, the agent s model of actions and their effect on the world is subject to permanent modification and improvement, relevant data for generalization and bootstrapping processes are generated and memorized while the system is operating. 58

59 OACs vs. Affordances Affordances are unidirectional Objects affords actions OACs Csae are bidirectional bd ecto Object affords actions Actions suggest objects OACs can be chained (new complex OACs from simpler OACs Tasks from skills = Planning ) OACs (unlike affordances) are fully formalized and (partly) implemented. 59

60 More Grasping OACs Box decomposition for grasping Pre grasp manipulation Grasping unknown objects 60

61 Grasping Known Objects: A Box Based Approach Object shape approximation through Box Decomposition [Huebner et al. 2008] Grasp hypothesis generation on objects, evaluated in GraspIt! grasp simulator part based grasping : Heuristical Selection (Geometric Reduction & Grasp Quality Learning) Shape Approximation Final grasp for the task show object Joint work with Danica Kragic and Kai Huebner, KTH, Sweden, ICAR

62 Pre grasp manipulation Flat objects are on the table are difficult to grasp Pre grasp manipulation to adjust the object before final grasping Joint work with Lillian Chang and Nancy Pollard (CMU), Humanoids 2010 Interface MCA2 and ROS 62

63 Grasping unknown objects Co planarity relation between visual entities define potential grasping affordances Surprising result: A success rate between 30 40% is already achievable by such a simple mechanisms. One reason is high level mechansism for hypotheses rejections through motion planning There is an autonomous success evaluation based on force/haptic information Collision, no success, unstable, stable. Joint work with Norbert Krüger and Mila Popovic, University of Southern Denmark 63

64 Thanks to the PACO PLUS Consortium Kungliga Tekniska Högskolan, Sweden J. O. Eklundh, D. Kragic University of Göttingen Germany F. Wörgötter Aalborg University Denmark V. Krüger University of Karlsruhe, Germany R. Dillmann, T. Asfour PACO PLUS PLUS Perception, Action and Cognition through Learning of Object Action Complexes plus.org plus org ATR, Kyoto, Japan M. Kawato & G. Cheng University of Liege Belgium J. Piater University of Southern Denmark N. Krüger University of Edinburgh United Kingdom M. Steedman Jozef Stefan Institute Slovenia A. Ude Leiden University Netherlands B. Hommel Consejo Superior de Investigaciones Científicas, Spain C. Torras & J. Andrade Cetto 64

65 Thanks Humanoids KIT Rüdiger Dillmann Stefan Ulbrich David Gonzalez Manfred Kröhnert Niko Vahrenkamp Julian Schill KiWlk Kai Welke Ömer Terlemez Alex Bierbaum Martin Do Stefan Gärtner Markus Przybylski Tamim Asfour Isabelle Wappler Pedram Azad (notin the picture) Paul Holz (not in the picture) Collaborators Members of the SFB 588, PACO PLUS PLUS and GRASP Ales Ude (JSI, Slovenia), James Kuffner (CMU,USA ), Danica Kragic (KTH, Sweden), Norbert Krüger (Denmark), Stefan Schaal and Peter Pastor (USC, USA), Nancy Pollard (CMU), Atsuo Takanishi (Waseda University, Japan), Gordon Cheng (TUM, Munich), Erhan Oztop (ATR, Japan), Florentin Wörgötter (BCCN, Germany), Mark Steedman (Edinburgh, UK) 65

66 Thank you for your attention. This work has been conducted within: the German Humanoid Research project SFB588 funded by the German Research Foundation (DFG) the EU Cognitive Systems projects funded by the European Commission PACO PLUS plus.org GRASP project.eu 66