UML Robotics: SUBTLE MURI. Dan Brooks, Abe Shultz, Holly Yanco University of Massachusetts Lowell

Transcription

1 UML Robotics: SUBTLE MURI Dan Brooks, Abe Shultz, Holly Yanco University of Massachusetts Lowell

2 Funded by the Army Research Office

3 SUBTLE Pipeline

4 Parsing, Semantics, Pragmatics Parsing Separates parts of speech Semantics Assigns meaning Determines Question Under Discussion Pragmatics Infers tone

5 PARs Parameterized Action Representation Agent of action (actor) Objects of action Context of action Applicability and termination conditions Parameterized Action Representation and Natural Language Instructions for Dynamic Behavior Modification of Embodied Agents, Badler et al, 1999

6 Linear Temporal Logic MissiOn Planning LTLMoP

7 Planner Integration

8 Invocation of Robot Behaviors LTLMoP has actions which may be activated when certain conditions are met Activated actions are passed on to the robot for execution Successful execution is reported in updated state information received by LTLMoP

9 Robot Behaviors Driving Commands GO to [location] SEARCH for [item,... ] FOLLOW [target] STOP Auxiliary Commands RADIO when you find [item,...] DEFUSE [target]

10 JavaScript Object Notation (JSON) A flat text format for representing objects and data structures Bindings in many languages, including C++ (MARC) and Python (LTLMoP)

11 Integration Implementation

12 Example JSON Conversation

13 Robot Receives commands from planner Executes commands in real world Records results

14 Robot Requirements Interface with LTL Planner User Interface Robot hardware drivers Perform Summarization Record large amounts of data Classify and present data to the user

15 MARC Architecture

16 Messaging System Common type definitions called Parcels Header - time stamp, author, uuid Body complex / abstract data type Serializable Many-to-Many Publish/Subscribe Topic Channels Message Broker Client Library

17 Technologies Protocol Buffers

18 What Unix Sockets want to be when they grow up Multiprotocol (IPC, TCP, Multicast) Multilanguage (C/C++, Java, Python,... ) Multiplatform (Windows, OSX, Linux) Multipattern (Pub/Sub, Pipeline, Request - Reply)

19 Protocol Buffers think XML, but smaller, faster, and simpler Language-Neutral (C/C++, Java, Python) Platform-Neutral (Windows, OSX, Linux) Serializable Data Structures

20 Protocol Buffers message Joystick { optional sint32 y = 1; optional sint32 x = 2; optional sint32 z = 3; } Joystick.set_x(-15);

21 Player/Stage Widely used hardware abstraction layer Player provides convenient interfaces to robot hardware Stage provides a simulation environment for rapid development

22 Message Store All content stored in basic types Metadata stored with messages

23 Database Storage Custom object-relational mapping Protocol buffers converted to database entries Query data using SQL

24 State Summarization The process of generating concise, detailed, and useful representations of the robot s current state and world model.

25 Use Cases Summarizing in-progress autonomous runs Summarizing completed autonomous runs Anomaly detection and reporting Failed autonomous run recovery

26 Out-of-the-Loop Problem Occurs when operator is removed from control loop Operators are slow or unable to recover from failure modes

27 3 Parts of Summarization Logging and indexing of information Deciding what information the user needs Interfacing with the user

28 Internal Behavior and State Representation Robot has set of predefined behavior patterns Listens for commands and returns state information Everything gets recorded into a database Sensor values Decisions made dictating behavior Current system state

29 Keeping Track of Data

30 Reference Trees

31 Making Data Useful Characterize the data to make finding important information more accurate Uses data metrics that are calculated at runtime Current Metrics: Abstractness and Interestingness

32 Metric: Abstractness More Abstract Less Abstract Autonomous Plan Autonomous State Autonomous Behaviors Pose / Waypoints Fiducial Information Laser / Camera Data

33 Metric: Interestingness Degree of relevance to current mission Arrival at destination Detection of sought object Difference from expected sequence of events Deviation from commanded course Detection of dangerous situation

34 Deja Vu Similarity with previous experience Trap detection Dynamic environments Confused navigation Loss of Situational Awareness

35 Interfaces and Audiences Linguistic Interface - Take advantage of the SUBTLE pipeline to complete human-robot conversation Visual Interface - Takes advantage of multimedia to create rich explanations

36 Provides more detailed information Users include remote observers at a command center and operators in training Visual Interface Use Cases: Recovering failed autonomous runs Summarizing completed autonomous runs

37 An Example Interface

38 Linguistic Interface Quick access to the most important information Users include co-located robot operators, bystanders Use Cases: Updates during autonomous runs Anomaly detection and reporting Photo From:

39 Demo Video

40 The End