6th European PhD School in Robotic Systems. Models and Knowledge. Herman Bruyninckx University of Leuven Eindhoven University of Technology

Transcription

1 6th European PhD School in Robotic Systems Models and Knowledge Herman Bruyninckx University of Leuven Eindhoven University of Technology Leuven, August 4 8, 2014 Overview Knowledge representation & s structure & behaviour mechanism & policy decomposition/decoupling & composition/coupling examples Model Driven Engineering for knowledge representation: meta meta s meta s & Domains Specific Languages (DSLs) -to- transformations & reasoning Homeworks identifying structure & mechanism turning the insights into DSLs connect DSLs to your code European PhD School in Robotic Systems H. Bruyninckx Page 2 Overview Knowledge representation & s structure & behaviour mechanism & policy decomposition/decoupling & composition/coupling examples Model Driven Engineering for knowledge representation: meta meta s meta s & Domains Specific Languages (DSLs) -to- transformations & reasoning Homeworks identifying structure & mechanism turning the insights into DSLs connect DSLs to your code Knowledge engineering is still an immature science...! Basically no knowledge is already available in formal ways...! European PhD School in Robotic Systems H. Bruyninckx Page 2

2 Knowledge representation = Modelling Objective: to what one knows about the world State of the practice in knowledge representation: State of the practice in ling for engineering: European PhD School in Robotic Systems H. Bruyninckx Page 3 Knowledge representation = Modelling Objective: to what one knows about the world integration of large amounts of knowledge, from various expertise s structure is necessary... also the understanding by humans is better when they (are forced to...!) formalize it. (Check this out, this week!) use by robots requires formal representations... if they need to exchange or to reason about it... State of the practice in knowledge representation: State of the practice in ling for engineering: European PhD School in Robotic Systems H. Bruyninckx Page 3 Knowledge representation = Modelling Objective: to what one knows about the world integration of large amounts of knowledge, from various expertise s structure is necessary... also the understanding by humans is better when they (are forced to...!) formalize it. (Check this out, this week!) use by robots requires formal representations... if they need to exchange or to reason about it... State of the practice in knowledge representation: Glossary: list of definitions, for humans. Taxonomy: tree of properties, for humans and computers. Ontology: graph of relationships between concepts. State of the practice in ling for engineering: European PhD School in Robotic Systems H. Bruyninckx Page 3

3 Knowledge representation = Modelling Objective: to what one knows about the world integration of large amounts of knowledge, from various expertise s structure is necessary... also the understanding by humans is better when they (are forced to...!) formalize it. (Check this out, this week!) use by robots requires formal representations... if they need to exchange or to reason about it... State of the practice in knowledge representation: Glossary: list of definitions, for humans. Taxonomy: tree of properties, for humans and computers. Ontology: graph of relationships between concepts. State of the practice in ling for engineering: Simulink, LabView,... : algorithms, single-thread systems. 20Sim, Modelica,... : s + algorithms for physical systems. RobotStudio (ABB), KUKASim, RobotExpert (Siemens),... : s for robot kin&dyn and geometric trajectories. European PhD School in Robotic Systems H. Bruyninckx Page 3 Knowledge representation = Modelling (2) Types of knowledge: symbolic: manipulation needs a hand on an arm,... discrete: go left, then right, then stop,... continuous: the dynamic relations of deburring,... European PhD School in Robotic Systems H. Bruyninckx Page 4 Knowledge representation = Modelling (2) Types of knowledge: symbolic: manipulation needs a hand on an arm,... discrete: go left, then right, then stop,... continuous: the dynamic relations of deburring,... Knowledge at system level: must integrate symbolic, discrete and continuous knowledge! must be composable, also at runtime in robotics, open world assumption must be designed in! European PhD School in Robotic Systems H. Bruyninckx Page 4

4 Knowledge representation = Modelling (2) Types of knowledge: symbolic: manipulation needs a hand on an arm,... discrete: go left, then right, then stop,... continuous: the dynamic relations of deburring,... Knowledge at system level: must integrate symbolic, discrete and continuous knowledge! must be composable, also at runtime in robotics, open world assumption must be designed in! Discussion: how to ground formal knowledge? ( enumeration, verified implementations,... ) how to compose knowledge? ( follow the natural hierarchical structures,... ) European PhD School in Robotic Systems H. Bruyninckx Page 4 5X Meta to represent structure Composition "contains", "connects" Concern 4 Concern 3 Concern 2 Concern 1 Vertical RHS: natural hierarchy of decoupled concerns Horizontal: composition of decoupled concerns, via: mereology/ contains relationship: ling parts and their containment in wholes. and/or topology/ connects relationship: ling interconnection of parts and wholes. European PhD School in Robotic Systems H. Bruyninckx Page 5 5X Meta to represent structure (2) Note: sometimes, there is no natural hierarchy, and the composition reduces to a mere aggregation. For example: Robot World Model Task specification Common Knowledge Object Affordances What are the robots' motion capabilities? What can exist in the world, and where is it? What does the application want the robot to achieve? What should every robot system know? How can objects be sensed and manipulated? European PhD School in Robotic Systems H. Bruyninckx Page 6

5 Example of grounding & composition Kinematics & dynamics of robots enumeration of platforms : verified implementations: European PhD School in Robotic Systems H. Bruyninckx Page 7 Example of grounding & composition Kinematics & dynamics of robots enumeration of platforms : mobile: bicycle, differentially driven, car,... serial arms: SCARA, industrial 6-DOF with structure, LWR-like 7-DOF, pan-tilt sensor head,... parallel arms: 3-DOF Delta, 6-DOF flex picker,... mobile manipulators: composition of two or more of the above humanoids: HRP-4 type, Asimo type,... The enumeration is structured knowledge in itself! verified implementations: European PhD School in Robotic Systems H. Bruyninckx Page 7 Example of grounding & composition Kinematics & dynamics of robots enumeration of platforms : mobile: bicycle, differentially driven, car,... serial arms: SCARA, industrial 6-DOF with structure, LWR-like 7-DOF, pan-tilt sensor head,... parallel arms: 3-DOF Delta, 6-DOF flex picker,... mobile manipulators: composition of two or more of the above humanoids: HRP-4 type, Asimo type,... The enumeration is structured knowledge in itself! verified implementations: we have correct implementations for the forward/inverse kinematics and dynamics of all these platforms. And for (some of the many ways) how these robots can apply these instantaneous behaviour to create trajectories. European PhD School in Robotic Systems H. Bruyninckx Page 7

6 Example of grounding & composition Kinematics & dynamics of robots (2) Robots systems have two complementary, natural structures: physical structure (hierarchical: contains ) motion capabilities (non-hierarchical: connects ) (Multi-) chain Composition LWR, 321, Justin, youbot, PR2,... Kinematic joint Mechanical joint Actuator motion constraints via revolute or prismatic joints,... transmission, limits, friction, inertia,... electric, hydraulic, pneumatic,... Power convertor electric-electric, electric-hydraulic,... (Multi)robot motion composition Manipulation dexterity Orientation dexterity Holonomic position mobility fingers,... wrist,... arm, shoulder, torso,... Non-holonomic position mobility wheels, legs,... (Discussion: other examples of structure...?) European PhD School in Robotic Systems H. Bruyninckx Page 8 Knowledge: graphs of connected concepts graphs: nodes & edges = binary relationships e.g.: tree, multi-tree, DAG,... hypergraphs: multi-node edges = n-ary relationships e.g.: factor graphs, info box,... X Y hidden observed A B C hierarchical hypergraphs: every containment level = a context but a context should not be a boundary of information hiding! e.g.: state machine, Bond Graph,... State 2.1 E_1 State1 State2 E_21 E_22 State 2.1 E_23 E_2 E_24 State 2.2 European PhD School in Robotic Systems H. Bruyninckx Page 9 Intermezzo: Context (Scope, namespace, frame, situation,... ) statement s semantics depends not only on statement itself, but especially on context in which it is used. for intelligent behaviour, context is everything! E.g., knowledge that Robot X can move at a speed of 1m/s can mean danger in one context, but opportunity in another context. The meaning is a property of the composition knowledge context, not of the knowledge relationship itself. all extra information needed to be able to decide (formally, by a computer algorithm) whether two different terms have the same meaning or not, or one single term has a different meaning or not in the two different contexts names/terminology: it is very difficult to avoid giving names that reflect one particular context ( looking up/outside, in the context ), instead of reflecting only the properties of the term ( looking down/inside ) implicit context is a major error in coding! European PhD School in Robotic Systems H. Bruyninckx Page 10

7 Knowledge: graphs of connected concepts (2) RDF/OWL triples: most popular approach in the semantic web links three concepts: relationship, subject, object. Example: a Wikipedia article about Tony Benn : European PhD School in Robotic Systems H. Bruyninckx Page 11 Knowledge: graphs of connected concepts (2) RDF/OWL triples: most popular approach in the semantic web links three concepts: relationship, subject, object. Example: a Wikipedia article about Tony Benn : advantage: lots of tools and documentation available. European PhD School in Robotic Systems H. Bruyninckx Page 11 Knowledge: graphs of connected concepts (2) RDF/OWL triples: most popular approach in the semantic web links three concepts: relationship, subject, object. Example: a Wikipedia article about Tony Benn : advantage: lots of tools and documentation available. disadvantages: exclusively symbolic level! ( description logic ) limitation to triples makes it difficult to represent tree and graph like representations in an easy syntax! Lisp represents trees. European PhD School in Robotic Systems H. Bruyninckx Page 11

8 Knowledge: graphs connected concepts (3) Wikipedia Info-boxes: structured tables (n-ary relationships!) with concepts and links as named (not led!) properties, in each article. Wikidata: n-ary relationship inverse of Info boxes, i.e., with centralized representation. Xlink: W3C standard to add single and n-ary relationships ( links ) to XML documents. More in particular, to make a link a first-class concept in itself, allowing to make statements about the relationship. More: KnowRob: (University of Bremen) combines multiple languages, for different purposes good! focus on (expression) trees a bit less good... misses hierarchical hypergraph natural structures the best one can get in robotics, for now... European PhD School in Robotic Systems H. Bruyninckx Page 12 Knowledge: graphs connected concepts (4) Topic Map ISO standard, n-ary relationship between concepts and contexts and occurrences (Source: Wikipedia) Not very popular. Open source topic maps database exists: Model-Driven Engineering paradigm: see later. European PhD School in Robotic Systems H. Bruyninckx Page 13 Reification Each relationship is a concept in itself too (knowledge_representation): statements can be made about statements! leads naturally to ( natural???) hierachies! Example: There is a 50% probability that the statement Robot X is in Room A is true. Importance: this ability/insight to treat each relationship/statement/knowledge as a primitive itself, about which statements can be made, is one of the keys to knowledge engineering in open worlds. European PhD School in Robotic Systems H. Bruyninckx Page 14

9 Open world assumptions Needs for open world assumption: one can not everything in advance. s must be updated at runtime = knowledge is added contexts come and go at runtime Types of open world assumption: 1. every rule that is not led is true. E.g.: semantic web s OWL language. 2. every rule that is not led as true is false. (= closed world assumption ) 3. a DSL that s knowledge can be given explicit tags to indicate where its world is open, that is, it needs other DSLs to complement itself. Examples: in robotics : Collada; in general: XCore. Counter examples in robotics: URDF, Gazebo s SDF. European PhD School in Robotic Systems H. Bruyninckx Page 15 Cartesian frame Primary example of knowledge graph Kinematic chains joint structure: mereology + topology mechanism: inward/outwards sweeps, to compute motion, articulated inertias, and forces joint axis branching joint link root behaviour: mass, friction, elasticity, controllers, estimators, geometry(?) (shape + location) policy: scheduling of behaviour to execute during sweeps, and tolerance of results European PhD School in Robotic Systems H. Bruyninckx Page 16 Cartesian frame Primary example of knowledge graph Kinematic chains joint structure: mereology + topology mechanism: inward/outwards sweeps, to compute motion, articulated inertias, and forces joint axis branching joint link behaviour: mass, friction, elasticity, controllers, estimators, geometry(?) (shape + location) policy: scheduling of behaviour root to execute during sweeps, and tolerance of results Similar examples:: probabilistic graphical s, control diagrams, Finite State Machines, multiple Level-of-Detail world s,... European PhD School in Robotic Systems H. Bruyninckx Page 16

10 Knowledge must be composable! Mainstream bad practice = API hierarchies Example: extremely non-composable library: inverse dynamics v1: Newton-Euler, by inward/outward sweeps over kinematic chain with ideal 1DOF joints: tau = ID NE (q, qdot, F) European PhD School in Robotic Systems H. Bruyninckx Page 17 Knowledge must be composable! Mainstream bad practice = API hierarchies Example: extremely non-composable library: inverse dynamics v1: Newton-Euler, by inward/outward sweeps over kinematic chain with ideal 1DOF joints: tau = ID NE (q, qdot, F) v2: what about posture control? tau = ID NE PC (q, qdot, F,tau p) European PhD School in Robotic Systems H. Bruyninckx Page 17 Knowledge must be composable! Mainstream bad practice = API hierarchies Example: extremely non-composable library: inverse dynamics v1: Newton-Euler, by inward/outward sweeps over kinematic chain with ideal 1DOF joints: tau = ID NE (q, qdot, F) v2: what about posture control? tau = ID NE PC (q, qdot, F,tau p) v3: and damped least-squares singularity robustness? tau = ID NE PC DLS (q, qdot, F,tau p,lambda) European PhD School in Robotic Systems H. Bruyninckx Page 17

11 Knowledge must be composable! Mainstream bad practice = API hierarchies Example: extremely non-composable library: inverse dynamics v1: Newton-Euler, by inward/outward sweeps over kinematic chain with ideal 1DOF joints: tau = ID NE (q, qdot, F) v2: what about posture control? tau = ID NE PC (q, qdot, F,tau p) v3: and damped least-squares singularity robustness? tau = ID NE PC DLS (q, qdot, F,tau p,lambda) v4: what about joint limit avoidance? tau = ID NE PC DLS JL (q, qdot, F,tau p,lambda,k) European PhD School in Robotic Systems H. Bruyninckx Page 17 Knowledge must be composable! Mainstream bad practice = API hierarchies Example: extremely non-composable library: inverse dynamics v1: Newton-Euler, by inward/outward sweeps over kinematic chain with ideal 1DOF joints: tau = ID NE (q, qdot, F) v2: what about posture control? tau = ID NE PC (q, qdot, F,tau p) v3: and damped least-squares singularity robustness? tau = ID NE PC DLS (q, qdot, F,tau p,lambda) v4: what about joint limit avoidance? tau = ID NE PC DLS JL (q, qdot, F,tau p,lambda,k) v5 v : what about N-DOF joints? mobile platforms? configuration of all parameters? trees? constraints? various trajectories?... European PhD School in Robotic Systems H. Bruyninckx Page 17 Inverse dynamics API bad practice It will get even worse... When sensing & control are added... : Cartesian point trajectory soft Cartesian point trajectory joint limits centre of gravity trajectory Cartesian link trajectory sliding contact with environment rigid connection to environment sensor space European PhD School in Robotic Systems H. Bruyninckx Page 18

12 Inverse dynamics API bad practice It will get even worse... When sensing & control are added... : Cartesian point trajectory soft Cartesian point trajectory joint limits centre of gravity trajectory Cartesian link trajectory sliding contact with environment rigid connection to environment sensor space Observations: main API problem: exposes behaviour coupled with structure, and policy coupled with mechanism! sensing & control are not libraries, but concurrent activities! European PhD School in Robotic Systems H. Bruyninckx Page 18 Model Driven Engineering (MDE) Knowledge representation via meta s M3 metameta conforms to M2 M1 meta (DSL) conforms to meta (DSL) DSL Designer DSL User DSL: Domain Specific Language realisation of M0 Real-world systems conforms-to : every relationship in the satisfies the constraints represented in the above. European PhD School in Robotic Systems H. Bruyninckx Page 19 Model-Driven Engineering (2) M3 = meta-meta (no -knowledge!) M2 = meta (Domain Specific Language) M1 = ( encoded in DSL) M0 = implementation (in specific programming language(s)) M3 M0: is an ontology! (And it extends traditional ontology design approaches with an extra meta meta for ontologies!) Conforms-to vs instance-of: (Discussion!) fundamental difference is composability: constraints compose, behaviour does not! European PhD School in Robotic Systems H. Bruyninckx Page 20

13 Model-Driven Engineering (2) M3 = meta-meta (no -knowledge!) M2 = meta (Domain Specific Language) M1 = ( encoded in DSL) M0 = implementation (in specific programming language(s)) M3 M0: is an ontology! (And it extends traditional ontology design approaches with an extra meta meta for ontologies!) Conforms-to vs instance-of: (Discussion!) fundamental difference is composability: constraints compose, behaviour does not! Meta levels: essential part of the Less is more objective! European PhD School in Robotic Systems H. Bruyninckx Page 20 Model-Driven Engineering (2) M3 = meta-meta (no -knowledge!) M2 = meta (Domain Specific Language) M1 = ( encoded in DSL) M0 = implementation (in specific programming language(s)) M3 M0: is an ontology! (And it extends traditional ontology design approaches with an extra meta meta for ontologies!) Conforms-to vs instance-of: (Discussion!) fundamental difference is composability: constraints compose, behaviour does not! Meta levels: essential part of the Less is more objective! Composition of knowledge: put semantic ( symbolic ) links between MDE s RDF, OWL, TopicMaps,...! Symbolic, Discrete & Physical/Continuous s: robotics needs this combination! ( difficult...!) European PhD School in Robotic Systems H. Bruyninckx Page 20 Model Driven Engineering: example of classes (Source: European PhD School in Robotic Systems H. Bruyninckx Page 21

14 Model Driven Engineering: other examples M3 M2 meta (DSL) metameta conforms to conforms to meta (DSL) XML Collada 1.5 URDF M1 youbot s realisation of M0 Real-world systems European PhD School in Robotic Systems H. Bruyninckx Page 22 Model Driven Engineering: other examples (2) M3 M2 meta (DSL) metameta conforms to conforms to meta (DSL) XML (metameta) XHTML, SVG languages (metas) M1 particular web page realisation of M0 this web page in your browser European PhD School in Robotic Systems H. Bruyninckx Page 23 Model Driven Engineering: classes vs s Class hierarchies are for: classification of behaviour inheritance: every subclass must behave as any of its superclasses multiple inheritance is extremely difficult Java allows inheritance of interfaces, i.e., of a, not a behaviour! Model hierarchies are for: knowledge representation knowledge is added by adding new relationships, anywhere a need not behave like its meta, but just satisfy all the latter s relationships, if applicable s are a lot more composable! European PhD School in Robotic Systems H. Bruyninckx Page 24

15 Should be: Intermezzo: Meta meta s small: full meaning must be graspable by human in five minutes. stand-alone: meaning should be clear from meta meta itself, and from the other imported meta meta s. fully formal: reasoning is possible, on wellformedness and meaningfulness. abstraction ( class vs instance ) is possible. categorization is possible. relationships are first-class concepts. European PhD School in Robotic Systems H. Bruyninckx Page 25 Should be: Intermezzo: Meta meta s small: full meaning must be graspable by human in five minutes. stand-alone: meaning should be clear from meta meta itself, and from the other imported meta meta s. fully formal: Examples: reasoning is possible, on wellformedness and meaningfulness. abstraction ( class vs instance ) is possible. categorization is possible. relationships are first-class concepts. hierarchical hypergraphs for lumped-parameter, port-based systems. physical units: QUDV, QUDT, UDUNITS, QUOMOS, Unified Code for Units of Measure,.... geometric frames: http: // software patterns: inversion of control, composite component,... European PhD School in Robotic Systems H. Bruyninckx Page 25 Intermezzo: Meta meta s (2) QUDT Quantities, Units, Dimensions and Data Types Semantic Sensor Network ontology: Semantic_Sensor_Net_Ontology describes sensors, the accuracy and capabilities of such sensors, observations, methods used for sensing, operating and survival ranges uses DOLCE-UltraLite3 (DUL) upper ontology. Dublin core: ontology of authorship & provenance of documents European PhD School in Robotic Systems H. Bruyninckx Page 26

16 Intermezzo: Meta meta s (3) Geospatial data: interconnected set of standards/ontologies: Geography Markup Language: where are things on earth. ISO standard: ISO 19136:2007. KML (Keyhole Markup Language): adds views to the geographic, and COLLADA as world language. CityGML: s how cities look like. IndoorGML: s how indoors of buildings look like. BIM (Building Information Models): s physical and functional characteristics of places. https: //en.wikipedia.org/wiki/building_information_modeling XDMF: s grids in 1D, 2D, 3D and above. European PhD School in Robotic Systems H. Bruyninckx Page 27 Intermezzo: Meta meta s (4) UML as ling language European PhD School in Robotic Systems H. Bruyninckx Page 28 Intermezzo: Meta meta s (5) UML as ling language (cont d) UML-specific problems: fine for class-instance relationships ( generalization of behaviour ). poor for link relationships ( associations ): a link is just a string, not a first-class ling primitive for humans, not robots! set of available relationships too limited: composition = is-a ; aggregation = has-a. no hyperedge relationships. Generic Observations: way too much language lock-in exists: people only use the features (for ling and processing of knowledge) that their familiar language and its tooling ecosystem provide. the XML (web services community) and RDF/OWL (X3C semantic web community) ecosystems are the two major competitors, with Lisp/Prolog (Artificial Intelligence community) a good ( better!) third. European PhD School in Robotic Systems H. Bruyninckx Page 29

17 Composition of meta s/knowledge M3 metameta conforms to metameta 2 M2 meta (DSL) conforms to meta (DSL) M1 realisation of M0 Real-world systems Discussion: compare to class multiple inheritance...! European PhD School in Robotic Systems H. Bruyninckx Page 30 Model Driven Engineering: transformations conforms to Source Language Meta Source Model conforms to Metaing Language Metameta conforms to Transformation Definition conforms to Transformation conforms to Target Language Meta conforms to Target Model Source Target M2-level transformations are key to integration, composition, harmonization, standardization! Discussion: advantages, disadvantages...! European PhD School in Robotic Systems H. Bruyninckx Page 31 Domain Specific Languages (DSL) DSL = artificial language in one particular. = one particular form of ontology. (At least, it should be... ) Necessary(?) parts in a DSL: Primitives: concepts, objects,... Relationships: between Primitives. Constraints on properties of Primitives and Relationships. Tolerances: on ( soft ) Constraints. European PhD School in Robotic Systems H. Bruyninckx Page 32

18 Domain Specific Languages (DSL) DSL = artificial language in one particular. = one particular form of ontology. (At least, it should be... ) Necessary(?) parts in a DSL: Primitives: concepts, objects,... Relationships: between Primitives. Constraints on properties of Primitives and Relationships. Tolerances: on ( soft ) Constraints. Discussion: which of the already mentioned examples is really a DSL? That is, do they have a M3-M0 formalisation? is it really useful to make DSLs that robots can understand & use...? European PhD School in Robotic Systems H. Bruyninckx Page 32 Examples: DSLs Contexts Constraints The talking horse flies through the ocean is well-formed (= syntactically correct, following the relationships) but its meaning depends on the context. the correctness of the syntactically correct statement The robot should not move faster than 1m/s to be safe depends on the context. European PhD School in Robotic Systems H. Bruyninckx Page 33 Examples: DSLs Contexts Constraints The talking horse flies through the ocean is well-formed (= syntactically correct, following the relationships) but its meaning depends on the context. the correctness of the syntactically correct statement The robot should not move faster than 1m/s to be safe depends on the context. Context provides the most important difficulties for formal language design: it brings constraints that come from above, and not from the sentence itself. computer tools can only work with context-free languages. complexity of context requires graph-shaped sets of DSLs context-dependent languages! European PhD School in Robotic Systems H. Bruyninckx Page 33

19 Hierarchical Hypergraphs for structural s NPC = Nodes, Ports, Connectors Structural primitives are: contains relationship: hierarchy. connects, via lumped-parameter ports. Example: connects(connector-i, port-p), connects(connector-i, port-s), connects(connector-j, port-n), connects(connector-j, port-q), i connects(connector-j, port-r), n B p connects(port-n, node-b), j connects(port-p, node-b), s q connects(port-q, node-c), r C X D connects(port-r, node-d), and A connects(port-s, node-x). contains(node-a, node-b), contains(node-a, node-c), and contains(node-a, node-d). European PhD School in Robotic Systems H. Bruyninckx Page 34 Meta /DSL example(s): robot kinematics, dynamics, motion & control kinematic joint mechanical joint + limits geometric frame kinematic chain + kinematics rigid body hybrid dynamics position, velocity, force motion specification motion control... l 6 l 4 = l 5 = 0 q 5 X 4 = X 5 = X 6 Discussion: why don t these DSLs exist already...? l 3 l 2 l 1 q 3 q 2 {ee} wrist centre q 6 point X 7 Y 7 X 1 X 3 X 2 q4 Y 4 = Y 5 = Y 6 Y 3 q 1 Y 2 Z 0 Y 1 X 0 Y 0 {bs} European PhD School in Robotic Systems H. Bruyninckx Page 35 Meta /DSL example(s): HH for robot motion stack actual motion Cartesian constraint motion constraints perception data bus Cartesian motion joint motion sensors actuator value Cartesian control joint constraint joint control actuator constraint actuator control joint & actuator constraints actuator constraints "current" setpoint action data bus actuators motion control stack software hardware Discussion: where does the motion stack stop, and the application start? European PhD School in Robotic Systems H. Bruyninckx Page 36

20 Meta meta robot device capabilities Topological structure Perception Software Adaptation Sensing Modelling Learning Planning Mechanism Control Actuation ENVIRONMENT Hardware Discussion: capabilities must still be grounded! How...? European PhD School in Robotic Systems H. Bruyninckx Page 37 Meta meta Robot app Task specification Platform capabilities Models Common knowledge Robotics Application World Object affordances Algorithms and Data... perception... planning reasoning... kin& control dyn learning trajectory adaptation generation... Software Activities configuration calls Configurator constraint constraint flow flow functional Computation monitor Computation HW/SW platforms App framework SW framework Operating Syst. MoveIt,... ROS, Orocos, OpenRTM, OPRoS,... Linux, ecos, VxWorks, QNX, Windows,... Coordinator events HW framework PC architecture, data buses, FPGA, GPU,... Discussion: each robot vendor dreams of making its app platform into the app platform... European PhD School in Robotic Systems H. Bruyninckx Page 38 Intermezzo: Robot app literature coverage Task specifications Robot platforms Models Robotics Application Knowledge base World Algorithms... perception... planning reasoning... kin& control dyn learning trajectory adaptation generation... Software Activities data in functional Computation data out HW/SW platforms App framework SW framework Operating Syst. MoveIt,... ROS, Orocos, OpenRTM, OPRoS,... Linux, ecos, VxWorks, QNX, Windows,... HW framework PC architecture, data buses, FPGA, GPU,... Too many publications on details... Hardly any research results on integration! European PhD School in Robotic Systems H. Bruyninckx Page 39

21 Meta meta Perception networks HH-NPC + lots of behaviour/policy mission robot task env obj1 obj2 fea1 fea2 fea3 sen1 sen2 Discussion: information flows in all directions...! European PhD School in Robotic Systems H. Bruyninckx Page 40 Meta meta Cognitive sensori-motor control networks pre-cognitive preview robot feedback & feedforward memory/ focus Control obj1 task obj2 fea1 fea2 fea3 cause1 mission cause2 intention/ motivation understanding env Perception reaction interpretation Discussion: should this be the meta to guide robot software development...? European PhD School in Robotic Systems H. Bruyninckx Page 41 Task { { Platform Conclusions Learn to decompose first! But in the context of structural composition! most algorithms in robotics are already composites, or even worse, applications, since they contain (often implicitly) control, sensing and world ling aspects! For example: SLAM motion planning... definition of an algorithm : sequential computation without any side effects. Design is about removing as many features as possible from (the of) a sub-system, without compromising its composability! European PhD School in Robotic Systems H. Bruyninckx Page 42

22 Discussion + Homework How can we bring more structure to robotics? (For example, via Could it be done via by hierarchical structures like: action mobile manipulator action perception mobile manipulator perception pick-and-place move to pick location locate object to pick align for pick... pick... lift move to place... place... object affordance: relationship between some of the above... European PhD School in Robotic Systems H. Bruyninckx Page 43 References RDF: http: //en.wikipedia.org/wiki/resource_description_framework Topic Maps: Wikipedia InfoBox: Ontology engineering: WikiData: Semantic MediaWiki: RoboEarth: Using Semantic Technologies to Describe Robotic Embodiments, Alex Juarez, Christoph Bartneck, Loe Feijs. Proceedings of the 6th ACM/IEEE International Conference on Human-Robot Interaction, Lausanne pp /semanticTechnologyRobotEmbodiments/ European PhD School in Robotic Systems H. Bruyninckx Page 44 References (2) A robot ontology for urban search and rescue, Craig Schlenoff, and Elena Messina. KRAS 05 Proceedings of the 2005 ACM workshop on Research in knowledge representation for autonomous systems Integrating ontological knowledge into a robotic DSL, Gaëlle Lorta, Saadia Dhouib and Sébastien Gérard. Proceedings of the 2010 international conference on Models in software engineering, pp , Events/2010/RoSym/presentations/Integrating_Ontological_ Domain_Knowledge_into_a_Robotic_DSL.pdf Robot Ontology Formation: index.php/robot_ontology_formation Upper ontologies: general concepts that are the same across all knowledge s. European PhD School in Robotic Systems H. Bruyninckx Page 45

23 References (3) OBO RO: with a -specific list of relations: // //purl.obolibrary.org/obo/merged/ro OBO RO Wiki: OpenCyc: general knowledge base and commonsense reasoning engine. temporal relations, spatial relations, mereological relations UMLS: online_learning/sem_001.htm BFO applications: OBI: Investigations European PhD School in Robotic Systems H. Bruyninckx Page 46 References (4) Cognitive Paradigm Ontology Meronymy, holonymy, mereology Moritz Tenorth and Michael Beetz, KnowRob: A knowledge processing infrastructure for cognition-enabled robots. The International Journal of Robotics Research, 32(5): , John Hallam, and Herman Bruyninckx, An Ontology of Robotics Science. First European Robotics Symposium, pp. 1 14, Jean Bézivin, On the unification power of s. Software and Systems Modeling, 4(2): , European PhD School in Robotic Systems H. Bruyninckx Page 47