Modeling the behaviour of interactive 3D scenes and applications

Transcription

1 Modeling the behaviour of interactive 3D scenes and applications Romuald Deshayes Software Engineering Lab Tom Mens Deshayes Romuald UMONS 1 / 29

2 Table des matières 1 Presentation context 2 Master thesis : problem statement 3 Proof-of-concept application 4 Modeling interactive behaviour 5 Statechart models 6 The application framework 7 About statecharts 8 PhD thesis proposal Deshayes Romuald UMONS 2 / 29

3 Presentation context Outline : Presentation context Presentation context Deshayes Romuald UMONS 3 / 29

4 Presentation context Presentation context Presentation in 2 parts Knowledge background (from Master thesis) : Statechart modelling of interactive gesture based applications PhD thesis proposal : Behaviour modeling and recognition of virtual 3D objects Deshayes Romuald UMONS 4 / 29

5 Master thesis : problem statement Outline : Master thesis : problem statement Master thesis : problem statement Deshayes Romuald UMONS 5 / 29

6 Master thesis : problem statement Problem Statement Challenge Develop intuitive interactive applications that are easy to maintain Why is it a challenge? Complex behaviour of user interaction Nondeterministic user actions Same events lead to different actions Contributions Gestural user interface for 3D object manipulation Modeling executable behaviour using a visual formalism Validation with an application framework Deshayes Romuald UMONS 6 / 29

7 Proof-of-concept application Outline : Proof-of-concept application Proof-of-concept application Deshayes Romuald UMONS 7 / 29

8 Proof-of-concept application Proof-of-concept application 3D visual drawing tool Uses gestures to create and manipulate 3D objects OpenGL graphical library Microsoft Kinect + NITE Deshayes Romuald UMONS 8 / 29

9 Proof-of-concept application Small video Deshayes Romuald UMONS 9 / 29

10 Modeling interactive behaviour Outline : Modeling interactive behaviour Modeling interactive behaviour Deshayes Romuald UMONS 10 / 29

11 Modeling interactive behaviour Modeling interactive behaviour Context Highly reactive event-driven systems Gesture-based interaction Proposed solutions Visual modeling language Statecharts Petri nets Labelled Transition Systems Amenable to formal analysis Easier to evolve Reduced complexity Deshayes Romuald UMONS 11 / 29

12 Statechart models Outline : Statechart models Statechart models Deshayes Romuald UMONS 12 / 29

13 Statechart models Statechart models - Hand Deshayes Romuald UMONS 13 / 29

14 Statechart models Statechart models - Component Deshayes Romuald UMONS 14 / 29

15 Statechart models Statechart models - Scene Deshayes Romuald UMONS 15 / 29

16 The application framework Outline : The application framework The application framework Deshayes Romuald UMONS 16 / 29

17 The application framework The application framework Framework for developing interactive applications Executable behaviour specified using statecharts Client-server architecture Deshayes Romuald UMONS 17 / 29

18 The application framework The application framework II Features Generic Abstract classes Observer design pattern Communication between statecharts Uses Java SwingStates library Deshayes Romuald UMONS 18 / 29

19 The application framework The application framework III Server extends Thread -serversocket : DatagramSocket Framework <<library>> SwingStates StateMachine Controller 1..* -machine : StateMachine <<Interface>> +addcomponent(c : Component) BodyListener extends +addpointer(p : Pointer) EventListener +addscene(s : Scene) +handmovedx(e : BodyEvent) +handmovedy(e : BodyEvent) +handmovedz(e : BodyEvent) 1 +handclosed(e : BodyEvent) +handopened(e : BodyEvent) 1..* * Component BodyAdapter -boundingbox : Rectangle +addaslistenerof(s : Server) -machine : StateMachine left right 0..1 Pointer -open : boolean machine : StateMachine 1..* Scene -machine : StateMachine selectedmodel 0..1 Model -selected : boolean 1..* Main Application MainScene 3DModel models BodyAdapters Hand -machine : StateMachine GLButton +changeactivetexture() : void +drawbutton(gl : GL) : void +startcounter() : void OpenGLWidgets ExtendableMenu +extend() : void +retract() : void +drawmenu(gl : GL) : void SpawningMenu +addbutton(button : GLButton) : void +addtext(text : String) : void +drawmenu(gl : GL) : void +spawn() : void +reduce() : void ButtonMenu +addbutton(button : GLBUtton) : void +drawmenu(gl : GL) : void ImageMenu +drawmenu(gl : GL) : void +onmodel() : void Deshayes Romuald UMONS 19 / 29

20 About statecharts Outline : About statecharts About statecharts Deshayes Romuald UMONS 20 / 29

21 About statecharts Statecharts for modeling executable gestural behaviour Statecharts Appropriate and scalable formalism for modeling the interactive gesture-based behaviour? Future work Externalise/decouple statechart representation from framework Compare statecharts with Petri nets & LTS Deshayes Romuald UMONS 21 / 29

22 PhD thesis proposal Outline : PhD thesis proposal PhD thesis proposal Deshayes Romuald UMONS 22 / 29

23 PhD thesis proposal Goal of the project Two main scientific contributions : Generic solution to specify the behaviour of the interaction between 3d virtual objects Improvement of the robustness of the 3D recognition algorithms, using 3D sensors Combining those two ideas will allow various applications in many domains such as augmented reality, video games or home automation (domotics) Deshayes Romuald UMONS 23 / 29

24 PhD thesis proposal Research Domain Interface modeling Many research focus on HMI modeling, but (to our knowledge) less are focused on modeling the behaviour of the interaction with virtual 3D objects Potential formalisms to model behaviour of the interaction Statecharts [ABL08] Petri nets [NPLB09] Labelled Transition System [CP09] Each one has its advantages and limitations Study and compare existing formalisms to see if they are suitable to solve our problem Deshayes Romuald UMONS 24 / 29

25 PhD thesis proposal Development of a Domain Specific Modeling Language Two approaches to develop a new DSML : 1 visual notation 1 formalism = studied formalisms (partial) 1 visual notation 3 distinct formalisms Deshayes Romuald UMONS 25 / 29

26 PhD thesis proposal Research Domain Computer Vision Make use of geometric information to recognize more accurately 3D objects in a complex scene Using 3D sensors Partial 3D reconstruction of objects [Des11] Need new recognition algorithms [LBRF11] Interactive system A training phase A real-time recognition phase Deshayes Romuald UMONS 26 / 29

27 PhD thesis proposal Thank you Thank you for your attention! Questions, suggestions? Deshayes Romuald UMONS 27 / 29

28 PhD thesis proposal Bibliography I C. Appert and M. Beaudouin-Lafon. Swingstates : adding state machines to java and the swing toolkit. Softw. Pract. Exper., 38 : , September Sébastien Combéfis and Charles Pecheur. A bisimulation-based approach to the analysis of human-computer interaction. In EICS, pages , R. Deshayes. Reconstruction algorithmique d objets 3d combinée à l interaction homme-machine. Master s thesis, UMONS, Mons, Deshayes Romuald UMONS 28 / 29

29 PhD thesis proposal Bibliography II Kevin Lai, Liefeng Bo, Xiaofeng Ren, and Dieter Fox. A large-scale hierarchical multi-view rgb-d object dataset. In ICRA, pages , David Navarre, Philippe A. Palanque, Jean-François Ladry, and Eric Barboni. Icos : A model-based user interface description technique dedicated to interactive systems addressing usability, reliability and scalability. ACM Trans. Comput.-Hum. Interact., 16(4) :56, Deshayes Romuald UMONS 29 / 29