Robotic Perception and Action - Project. Augmented Reality for the Man in the Loop. M. De Cecco - Robotic Perception and Action

Transcription

1 Robotic Perception and Action - Project Augmented Reality for the Man in the Loop

2 Jaakko Hyry, Max Krichenbauer, Goshiro Yamamoto, Takafumi Taketomi, Christian Sandor, Hirokazu Kato, Petri Pulli, Design of Assistive Tabletop Projector- Camera System for the Elderly with Cognitive and Motor Skill Impairments M. De Cecco - Robotic Perception and Action An example of setup from literature

3 The user quest for help (voice, gesture or virtual button on the table) The system: 1. Item := 1; 2. Detect the item to manipulate 3. Highlight it to the user 4. Animate the action to be performed 5. Verify that the item was moved 6. Ask if the first step was performed correctly 7. Item = item If not end- procedure, goto point 2 Project interaction sequence

4 Project Modules Description Software to use Groups State Machine > Main software that manages the HMI interaction. > Main projected interface. HW: virtual buttons with retroreflective material C# - UNITY Objects Localization with AR Toolkit Skeleton acquisition > Markers placed on the objects extraction (position and location on the projector space). > Calibration of camera/projector/furnitures HW: kitchen simulated furnitures C# - UNITY > Skeleton acquisition with ToF (kinect) > Control of task execution UNITY Animations Objects Localization with ToF camera Sensor Fusion Development of proper anmations (for example with Blender) > Point cloud extraction and its use for objects recognition and localization > Sensor Fusion of Objects locations estimated with Ar Toolkit and the ToF camera UNITY - Blender C++ (PCL) UNITY Project modules to develop

5 1. State Machine

6 In gaming design the gameplay is usually the main goal that everyone thinks of when first undertaking the creation of a game. After the game is well along its development, a realization begins to appear that there s more to a game than just playing it Putting a game on the market means it has to include things such as beginning title screen, game options, player settings. The same is true when we are designing a human machine interface in AR! Why the State Machine

7 An object created from a class in C# is called an instance of the class In UNITY, an object created from a class is called a component There is only one main difference: A class we write in UNITY has to be attached to a GameObject A class we write for the State Machine will not be attached to a GameObject UNITY hides the code that is necessary to create an object because knows how to create it from the GameObject For the State Machine we have to write the code necessary for the object Instance and component

8 The beauty of the state machine is its simplicity: you probably spent at least the first 14 years of your life living in a state machine environment (and still ): As a child your parents were always making you take a bath, go to school, do your homework,, go to sleep When you were in school you were in school state. Where you allowed to do anything else??? When you were taking a bath you were in bath state. Where you allowed to do anything else??? what about the do your homework The basic concept of the state machine

9 This is exactly what we are going to do in UNITY to manage the dynamic UI The following is the basic concept of a State Machine framework for controlling a UNITY project: UNITY calls the Update() method every frame The StateManager script is the heart of the State Machine, a Component that has the Update() method The code of Update() delegates control to the state that is active States are regular C# instances (and not GameObject components) The active state determines what is displayed

10

11 The StateManager script is the heart of the State Machine, a Component that has the Update() method.. So the StateManager is a Unity class and so: It inherits from the MonoBehaviour class It is attached to a GameObject to become a (script) component The following are the three core features that the StateManager script handles: Delegates game control to a State via the Update() method Switch to another state when called to do so Keeping track of the active state

12 NOTE: the game control code that is, as usual, in Update(), is delegated to the StateUpdate() of the activestate object!!! So, when Unity calls the Update() method on the StateManager Component, the StateUpdate() method is called on the currently active State object!!! It is the principle of delegating responsibility from the Update() of the StateManager to the StateUpdate() of the active state!!! This is one of the main principles behind the State Machine behaviour! Delegation from Update() to StateUpdate()

13 No matter how many states you may use to control your game, always remember the following: Every State has the StateUpdate() method The code inside it will be different from every state depending on what it has to accomplish Only one state is active at any time Delegation from Update() to StateUpdate()

14 The following are the three core features that the StateManager script handles: Delegates game control to a State via the Update() method Switch to another state when called to do so Keeping track of the active state It is the code in the active state that will trigger the switch to another state The SwitchState() method on the StateManager script is called to accomplish this Switching to another state

15 When the active state has been identified it is time to switch to another state. The SwitchState() method on the StateManager script is called SwitchState() takes an argument of the new State that has to become active next Switching to another state

16 The newly created State is then assigned to the activestate variable So, when Unity calls Update() on the StateManager script, control is passed to the newly created State whose reference is stored in activestate by calling its StateUpdate() method Keeping track of the active State

17 A Summary of the previous steps!

18 Now let s start practicing with some code! so open Unity in your laptop!

19 We are going to create the StateManager script and the BeginState class Then add some code so we can instantiate BeginState to make an instance of it

20 From 4 9 only in Monodeveloper or VS

21 Or you can do it in Unity from step 1..3

22 We will use the following colour code to indicate the different scripts: BeginState StateManager

23 A Namespace declares where a class file is located in the Unity Project folder structure BeginState

24 A Namespace declares where a class file is located in the Unity Project folder structure In StateManager instead the class location is used!!! StateManager

25 ??? We are instantiating the BeginState class and storing a reference of this new BeginState object in the variable activestate StateManager

26

27 Run outcome!

28 1 1. Start was called by Unity 2. The activestate variable was filled with the BeginState constructor that simply writes the string Constructing BeginState StateManager is called by Unity!

29 Notice that the constructor method name is the same as the class name! Constructor basics

30 2 1. output the string This object : Assets.Code.States.BeginState that is an istance object of the BeginState class! StateManager

31 An interface in general is what it is possible to interact with: For a TV the interface is the TV remote For a smartphone the interface is the touch For a class the interface are the public variables and methods!!! All the possible states have to implement proper variables and methods to be managed by the StateManager and so the interface must be guaranteed (no bugs) and standard! Focus on interfaces!!!

32 Ok, C# does have a way to guarantee that the required methods are implemented in each State by using the C# interface! Now we are going to create a C# interface and implement it in BeginState, but incorrectly to see some errors Focus on interfaces!!!

33 Focus on interfaces!!!

34 In monodeveloper (or Visual Studio for windows) the file structure will look like: the sss directory shall be named States (my mistake at the beginning of the project!) Focus on interfaces!!!

35 Now modify the IStateBase file as follows: Focus on interfaces!!!

36 Focus on interfaces - BeginState

37 save and verify that on the Unity console we have the following two errors: this is exactly what the IStateBase interface was designed for: to inform us that the methods StateUpdate() and ShowIt()!!! Focus on interfaces!!!

38 Focus on interfaces: implement the required methods and verify in Unity that all now is ok!

39