Wearable Gestural Interface Matthias Schwaller matthias.schwaller@unifr.ch 16.12.2009
Outline Idea of the project State of the art Demonstration application Gestures Hardware Software architecture Integration into InterFace Developed applications Conclusion and future works Demo 16.12.2009 Matthias Schwaller 2
Idea of the project - 1 Develop a wearable gestural interface for smart living room & smart meeting room Device should be able to communicate with other devices The design includes 2 parts: hardware & application [7] 16.12.2009 Matthias Schwaller 3
Idea of the project - 2 Goal of the project: Wearable tool -> small, pervasive Hand gestures for the communication -> simple to use Augment physical world with information -> no need to open a notebook Communication with other devices -> multi-user applications 16.12.2009 Matthias Schwaller 4
Idea of the project - 3 Application Gesture recognition can be divided in the following 3 parts: Image acquisition and preprocessing Tracking Gesture recognition Integration into other applications 16.12.2009 Matthias Schwaller 5
State of the art - 1 [1-3] Augmented Reality Augmented the environment of the user by additional information (mostly visual) Wearable Computing Worn by the user Challenges: Power use Heat dissipation 16.12.2009 Matthias Schwaller 6
State of the art - 2 Gesture Recognition Preprocessing Color segmentation Tracking Condensation Gesture recognition Hidden Markov Model 16.12.2009 Matthias Schwaller 7
State of the art - 3 [4-6] Gesture Recognition Projects sixthsense (WUW Wear Ur World) MIT Inspiration Inter-Face EIA-FR Integration of WGI in Inter-Face 16.12.2009 Matthias Schwaller 8
Demonstration application Tasks of the Application Open PowerPoint Files Switch to presentation mode Navigate into the slides Close presentation mode Send file to another tool 16.12.2009 Matthias Schwaller 9
Gestures - 1 Gesture for clicking and scrolling 16.12.2009 Matthias Schwaller 10
Gestures - 3 Gesture for opening presentation in full screen (F5) 16.12.2009 Matthias Schwaller 11
Gestures - 4 Gesture to close the presentation (ESC) 16.12.2009 Matthias Schwaller 12
Gestures - 5 Gesture for the next slide Left hand Right hand next 16.12.2009 Matthias Schwaller 13
Gestures - 6 Gesture for the previous slide Left hand previous Right hand 16.12.2009 Matthias Schwaller 14
Gestures - 7 Gesture to send a file to another person Left hand Right hand The hand is at the beginning in the middle or on the left side and go further out of the camera range on the right side 16.12.2009 Matthias Schwaller 15
Hardware Comparison has been done Following products will be used. Beamer 3M MPro 110 [8] Camera QuickCam Pro for Notebooks [9] Ultra mobile PC Sony Vaio UX micro PC [10] 16.12.2009 Matthias Schwaller 16
Hardware Fixation - 1 Beamer Fixation at the belt Arm can be moved in almost every direction 16.12.2009 Matthias Schwaller 17
Hardware Fixation - 2 Camera Fixed on a plastic panel Band around the neck 16.12.2009 Matthias Schwaller 18
Hardware Fixation - 3 UMPC Fixed at the belt (cable goes in the pocket because of the length of the cable) 16.12.2009 Matthias Schwaller 19
Hardware Fixation - 4 Finger markers Foxglove with parts of air balloons 16.12.2009 Matthias Schwaller 20
Software architecture (WGI) - 1 16.12.2009 Matthias Schwaller 21
Software architecture (WGI) - 2 GesureRecognition Controller for the gesture recognition Contains a timer Stocks the common objects for the whole gesture recognition Uses the library Emgu CV 16.12.2009 Matthias Schwaller 22
Software architecture (WGI) 3 Preprocessing Input: image (webcam) Image segmented in different colors (finger markers) Cleaning image (morphology operator: opening and closing) Rectangle color creation (one per color with center) Output: coordinates of the color rectangles Uses the library Emgu CV 16.12.2009 Matthias Schwaller 23
Software architecture (WGI) 4 Tracking Input: coordinates of the color rectangles Permits to track the different color rectangles and prediction of the locations Output: prediction of the coordinates Emgu CV don t support the condensation algorithm Tracking made in C++ with OpenCV Wrapping from C++ to C# Library: OpenCV 16.12.2009 Matthias Schwaller 24
Software architecture (WGI) 5 GestureIdentification Input: coordinates of the color rectangles Permits to train different gestures Detect gestures Output: gesture Uses HTK library (voice recogntion) Uses HTK extension GART (mouse gesture recognition) 16.12.2009 Matthias Schwaller 25
Software architecture (WGI) 6 GestureExecution Permits to inform the interesting components Contains the interface WGIListener 16.12.2009 Matthias Schwaller 26
Integration of WGI Necessary steps to include WGI into another applications: Include the DLL WGI Create a class which includes the interface WGIListener Create an object of GestureRecognition Add the class with the interface WGIListener to the eventdistributor of GestureRecognition 16.12.2009 Matthias Schwaller 27
Some additional explications WGI has two modes Touch mode: like two mice Gesture mode: permits to detect the gestures To switch between the two modes, the voice is used To train the gestures video files has to be placed in the folder for the gestures and the newtraining boolean has to be changed to true The raised gestures or mice activities are send to all WGIListeners 16.12.2009 Matthias Schwaller 28
Integration into InterFace 16.12.2009 Matthias Schwaller 29
Developed applications - 1 16.12.2009 Matthias Schwaller 30
Developed applications - 2 16.12.2009 Matthias Schwaller 31
Developed applications - 3 16.12.2009 Matthias Schwaller 32
Optimization and current problems Cutting the movies that every training movie has the same length 92 % of success rate Problems with differencing F5 ESC Next Prev 16.12.2009 Matthias Schwaller 33
Conclusion and future work - 1 Occurred problems HMM of OpenCV In the auxiliary package No documentation aside from the method descriptions Bad error output Some gestures not possible with GART GART beta Sometimes training was not possible 16.12.2009 Matthias Schwaller 34
Conclusion and future work - 2 Improvements and Future work Application for the training More extensive training Brighter portable beamer Extend to 3 dimensional gestures Put camera nearer to the eyes of the user [11] 16.12.2009 Matthias Schwaller 35
Conclusion and future work - 3 Improvements and Future work Use of a IPhone or Android phone instead of a UMPC Beamer integrated in the phone [12] 16.12.2009 Matthias Schwaller 36
Conclusion and future work - 4 Personal impressions Complex project Big challenge Very interesting Enriching project 16.12.2009 Matthias Schwaller 37
Demo
References 1 1. German Wikipedia website for Augmented Reality 2. Cours Interfaces Multimodales at University of Fribourg (Elena Mugellini, Denis Lalanne, Jacques Bapst, Omar Abou Khaled. 2008/2009) 3. A Survey of Augmented Reality (Ronald T. Azuma. 1995) 4. WUW - Wear Ur World - A Wearable Gestural Interface (Pranav Mistry, Pattie Maes, Liyan Chang. 2009) 5. sixthsense website (http://www.pranavmistry.com/projects/sixthsense/) 6. Generic Framework for Transforming Everyday Objects into Interactive Surfaces (Elena Mugellini, Omar Abou Khaled, Stéphane Pierroz, Stefano Carrino,Houda Chabbi Drissi. 2009) Images: 7. http://creativetech.inn.leedsmet.ac.uk/staff/rb/rbwearable.jpg 8. http://www.gizmag.com/pictures/gallery/10358_13110812813.jpg 9. http://www.logitech.com/repository/506/jpg/4229.1.0.jpg 10. http://www.ounae.com/wp-content/uploads//2007/09/sony-vaio-umpc.jpg 11. http://www.geekalerts.com/u/spy-sunglasses.jpg 12. http://www.journaldugeek.com/files/2009/12/77880-lg-expo-telephonepicoprojecteurvideo-projecteur-287x540.jpg 16.12.2009 Matthias Schwaller 39