Interaction Using Nearby-and-Far Projection Surfaces with a Body-Worn ProCam System

Transcription

1 Interaction Using Nearby-and-Far Projection Surfaces with a Body-Worn ProCam System Takeshi Kurata 1 Nobuchika Sakata 13 Masakatsu Kourogi 1 Takashi Okuma 1 Yuichi Ohta 2 1 AIST, Japan 2 University of Tsukuba, Japan 3 Osaka University, Japan

2 Remote Collaboration and Wearable Visual Interfaces The need for human-computer interface for supporting real-world tasks Heightened ever in the context of social problems; Baby boomer retirement, safety compliance, etc. Wearable visual interfaces Promising technology to support workers who want to Move among different workplaces Interact with real objects Couple virtual worlds with the real world meaningfully Connect person to person in remote places

3 Previous study: WACL vs HMD/HMC Typical real-world tasks during collaboration primarily consist of three phases; Object identification Laser pointer as a visual cue: enough Procedural instruction WACL: Imposed more burdens on the remote experts when they needed to send detailed instructions -> Poor visual cue Comprehension monitoring In ISWC2004 WACL: Wearable Active Camera/Laser pointer HMD: Head-Mounted Display HMC: Head-Mounted Camera

4 BOWL ProCam (BOdy-Worn Laser Projector-Camera) Project rich visual cues directly onto the real world Controller Laser Projector MAP Content Attitude Sensors Fisheye HD Cam Laser projector Project rich visual annotation directly onto the real world Long focal depth, Wide scanning angle Compact structure Low-power consumption Fish-eye HD camera As a ProCam System 30mm 29mm 29mm 31mm 31mm

5 Laser projector Metal-based microscanner Piezoelectric layer: Directly deposited on the stainless-steel frame beside the mirror by means of the Aerosol Deposition (AD) method Wider scanning angle than conventional MEMS scanners

6 BOWL ProCam (BOdy-Worn Laser Projector-Camera) Project rich visual cues directly onto the real world Controller Laser Projector MAP Content Attitude Sensors Fisheye HD Cam Laser projector Fish-eye HD camera Easy to comprehend real workplace conditions regardless of the user s posture change Easy to stabilize projected visual cues and maintain a visual log (Keep observing the same target object) As a ProCam System 30mm 29mm 29mm 31mm 31mm

7 BOWL ProCam (BOdy-Worn Laser Projector-Camera) Project rich visual cues directly onto the real world Controller Laser Projector MAP Content Attitude Sensors Fisheye HD Cam Laser projector Fish-eye HD camera As a ProCam System Obtain 3-D structure of the real world with combination of active-stereo-vision and shape-from-motion technologies Provide intuitive interaction techniques while comprehending the situation of the users and the real environment 30mm 29mm 29mm 31mm 31mm

8 Fixing-point evaluation Fixing point of a wearable camera: Often decided empirically or experimentally Mayol s MATLAB simulator: Articulated humanbody model (1800 polygons, 16 body segments) Modified it for ProCams to support tasks involving physical operation

9 Field of View Weighted mean Stability: Looking around Suitable for wearing a ProCam View of the handling space Stability: Walking

10 Interaction using nearby-and-far projection surfaces An interaction technique utilizing long focal depth and wide scanning angle: Effectively employing both of Nearby projection surface (user s hands) Far projection surface (tabletop and wall) Need to understand where the surfaces are Detecting projection surfaces <- important

11 Proof-of-concept system with a conventional DMD projector Since the laser projector is still under development So, Fixed the distance from the projector to a palm at 35 to 55 cm, and the distance to a far surface at around 65 cm Controller Laser Projector HP mp2225 Attitude Sensors Fisheye HD Cam 70mm 70mm Scorpion & InertiaCube3

12 Nearby-surface detection Active-stereo-based technique with a projected VUI (Visual User Interface) image taken from the camera and the original VUI image 1. Fish-eye image -> Perspective corrected image 2. Rectification of the projected VUI image 3. Feature-point extraction in the original VUI image 4. Normalized-correlation-based stereo matching 5. Depth estimation of the nearby surface

13 Image transformation Equidistance projection -> Perspective projection Projector Radial distance l = f wθ l = f w tan( θ ) Fish-eye cam Rectification of the projected VUI image

14 Stereo matching Rectified image: the mixture of a VUI image and texture of the projection surface Extract only salient points from the original VUI image (the Harris operator) Normalized-correlation-based stereo matching Narrow 2D search (e.g. 3-pixel width) To cover for imperfect rectification Assumptions Lots of feature points in each VUI image Projection planes that not have much obtrusive texture

15 Results Nearby-surface detection: No false positive/negative with 14 images Projected-content selection: the mean distance [42 pixels], the SD [24 pixels] with 10 images (Rectified image: 512x384 pixels, window: 19x19 pixels)

16 Snapshots from the user s field of view

17 Depth difference between the nearby-and-far surfaces Mean disparity (pixel)

18 Hand-posture classification Base-posture acquisition Rectified camera image VUI image Normarized SSD (8x8 pixels) Binarization Morphological Operations ( ) { } = y x y x y x y y x x I y x T y y x x I y x T y x R, 2, 2, 2 ), ( ), ( ), ( ), (, To increase the number of interaction techniques

19 Extracting portions different from the base posture Frame f-n Frame f Thumb-up Subtraction Base posture Clenched fist (Rock) Frame f

20 Results of hand-posture classification Binarization/ Morphological Operations Normarized SSD Rectified cam image Clenched fist (Rock) Thumb-up Subtraction

21 System diagram

22 Occluded projection and shadow Occluded projection for Cam Observed shadow for Cam Occluded projection for User Good for projecting code patterns!?

23 Structured light projected in the hidden region for the user Useful to recover 3-D structure/motion of the far surface User s view Cam s view

24 Conclusions Proposed the concept of a novel wearable AR interface; the BOWL ProCam Hands-free, Eye-free, and Head-free The upper chest area: Selected as the wearing position Proposed a novel interaction technique employing both nearby-and-far projection surfaces Active-stereo and hand-posture-classification techniques Applicable to realize such interaction by using the proof-of-concept system