Steerable Interfaces for Pervasive Computing Spaces

Transcription

1 Steerable Interfaces for Pervasive Computing Spaces Gopal Pingali Claudio Pinhanez,, Anthony Levas,, Rick Kjeldsen,, Mark Podlaseck,, Han Chen, Noi Sukaviriya IBM T.J. Watson Research Center Hawthorne, New York USA

2 Talk Overview! Motivation/definition of steerable interfaces! Overview of key technologies! Demonstration of prototype smart retail environment! Conclusions and discussion

3 Motivation Can a display appear wherever the user needs it rather than the user being enslaved to display devices (monitors, televisions etc.)?

4 Steerable Interfaces for Pervasive Computing Environments Steerable Interface: An interface to computing that can move around a physical environment to appear on ordinary objects or surfaces or even in empty space. Characteristics:! Movable output interface! Movable input interface! Adaptation to user context! Adaptation to environmental context! Device-free interaction! Natural interaction Applications:! Information access in public spaces -- retail, transportation! Instruction and information delivery in military training centers, ships, aircraft! Deviceless interaction in hazardous environments! Augmented reality in hospitals! Interactive real world games

5 Steerable Interfaces -- Key Technologies Environment modeling Calibration Steerable projection WORLD MODEL (3D geometry, user location,display zones, calibration parameters, system state) Steerable vision Steerable microphones A P I User Localization Geometric Reasoning Steerable audio

6 Steerable Projection: The Everywhere Display A device that creates an interactive projected display on any surface in an environment. Prototype bright LCD projector rotating mirror pan/tilt/zoom camera

7 Oblique Projection Distortion display image

8 Correction of Oblique Projection Distortion display image Goal: projected image is free of distortion when viewed by an orthogonal viewer

9 Projector/camera Calibration! computation of homography H/Hv Hv given location of corners on a display surface! need expected hand size in image and direction of entry for interaction! manual specification of focus, pan, tilt, zoom

10 Integration into Office-of-the-Future: BlueSpace at IBM T.J. Watson Task Lights OfficeFront Display Ceiling Lights Wireless Pressure Sensor Primary Display Status Lights Everywhere Display Personal Air & InfoPanel Heating Display Environment Sensors

11 Scenarios in Bluespace Team work support Personalized wall display notification on desk Video conferencing anywhere

12 Steerable Gesture Recognition for Input Projected light changes appearance of hand!! camera steered to projected display! motion detected in camera image stream! motion data matched to finger tip template

13 Surface Independent Interface Specification application space map to surface A map to surface B! Independent specification of what and where! Dynamic configurations of widgets;! Map configuration to surface at run time; image space

14 Widget Architecture Widgets: Touch Button Tracking Area Image Component types: Image Processing Motion Analysis System Functions Motion Detection Change Detection Fingertip Tracking Image Region Definition (IRD) Event Generation (EG) Surface Transformation (ST) Touch Motion Detector In-Region Detection IRD EG ST Return Value Transformation IRD EG Instance Data Good high level abstraction - Efficient widget combination - Flexible widget definition ST

15 First Prototype of Steerable Interfaces (Siggraph( Siggraph,, Aug 2001) Completed picture Interactive cans with M&Ms Color selection Invitation board at entrance Picture in progress: placement, finger painting

16 User Experience at Siggraph 2001 SIGGRAPH01.avi

17 Interactive Applications Anywhere Various interactive applications can be dynamically moved to different surfaces e.g.. MediaPlayer, PowerPoint presentations, Web Browser applications etc.

18 User-following Display Goal: Automatically steer display to appropriate surface at the appropriate position, size and orientation based on user location. Enabling Technologies: 1. 3D model of the environment 2. 3D calibration of projector(s), camera(s) 3. Dynamic localization of the user(s). 4. Geometric reasoning engine for: virtual projection virtual imaging virtual steering occlusion detection computation of warping function

19 1. 3D Environment Modeling VRML model of environment:! Based on real world measurements

20 Modeling toolkit! Basic 3D modeling software supporting simple geometries (planes, cubes) and import of more complex geometries! Provides annotations for all objects -- semantics can be attached to objects! Includes projectors, cameras, and PJsurfaces! Model stored in XML format (objects are tags and annotations are attributes)

21 2. 3D Calibration: The Link Between the Virtual and the Real 3D Camera Calibration! Use correspondence between points in 3D model (known points in environment) and points in camera image plane! Error in image to world mapping within 25 mm (maximum) 3D Projector Calibration! Determine effective location and orientation of projector! Error within 5 mm (coplanar) and 15 mm (non-coplanar) in image to world mapping

22 3. User Localization Real-time tracking of user position! Use image/background differencing with morphological filtering to detect motion regions! Extract bounding contours of resulting regions! Check for presence of a head by analyzing extrema of curvature on the contour and circularity! Track detected head! Triangulate detections from multiple cameras to determine 3D position

23 4. Geometric Reasoning Input: : 3D model, user position, display zones, ED calibration parameters Output: : Information on every display zone:! physical size! user proximity! orientation of zone w.r.t user! percentage of occlusion of a display zone! occlusion mask (a bitmap indicating occlusion)

24 Tying it all together: User- Following Display! Pre-define a set of display zones in relation to environment model! Calibrate projector in 3D for each display zone! Locate user by tracking with pre-calibrated camera.! Map user position to 3D environment.! Select display zone based on user location -- after occlusion check.

25 Video: User-Following Display

26

27 Demo: Everywhere Sales Associate Technical Goal: Adaptation of interface to user and environmental context.! Interactive help anywhere in a store! Dynamic placement of display close to the user! Adaptation of display content/interaction mode! Detection and correction for occlusion of display Examples:! Ubiquitous Product Finder!Adaptive Signage! Interactive Product Bins

28 Video: Product Finder - Table

29 Video: Product Finder Wall(1)

30 Video: Product Finder: Wall (2)

31 Video: Interactive Signage

32 Video: Clothing Bins (distance)

33 Video: Clothing Bins (interactions)

34 Conclusions and Future Directions! Introduced and demonstrated feasibility of new class of steerable interfaces! Adds a new dimension to the vision of computing woven into space without wiring the user/objects Future Directions: Greater multimodality,, adaptation, context awareness! Steerable Audio-visual interface! Underlying distributed architecture for context awareness and adaptation! Adaptation to surface characteristics! Multi-user/collaborative situations! Advanced calibration/model capture! Natural interaction!