HUMAN COMPUTER INTERACTION

Transcription

1 HUMAN COMPUTER INTERACTION 7. UI DEVICES AND SYSTEMS I-Chen Lin, National Chiao Tung University, Taiwan

2 Objectives Unique or interesting control/input devices and technologies. Display interfaces in addition to PC monitors. Devices conforming to users perceptual abilities. The applications of advanced display and virtual environment. Several slides are from the following references: D.A. Bowman, E. Kruijff, J.J. LaViola, I. Poupyrev, 3D User Interfaces: Theory and Practice, Addison Wesley Professional, Course notes, 3D User Interfaces, CS, Columbia Univ.. Course notes, Introduction to Human-Computer Interaction Design, CS, Stanford Univ..

3 Dimensions of Performance Continuous vs. discrete Resolution and accuracy Sampling rate and latency Noise, aliasing and nonlinearity Direct vs. Indirect Absolute vs. relative Control-to-display ratio Physical property sensed Position, motion, force Degrees of freedom

4 Symbolic Input Task of entering Text Numbers Symbols Desktop symbolic input Mobile symbolic input Standing, walking, communicating 3D UI symbolic input Tracking, gesture, etc.

5 Symbolic Input (cont.) Conditions for 3D environment Mobile users Not seated: standing, crouching, About to move Actively moving No dedicated desk surface Hands busy or full Eyes busy, occluded, or in low light

6 Keyboards

7 Some Ergonomic Issues Hand position - Freedom of hand for positioning device Touch typing vs. hunt and peck Repetitive stress / fatigue One handed use Need for support

8 Mobile (Chord, Multi-press, )

9 Specific Symbolic Input J. Mankoff & G. Abowd, UIST 98 Samsung Scurry I. Poupyrev, N. Tomokazu, S. Weghorst, VRAIS 98

10 Specific Symbolic Input Data gloves Key poses or continuous sign recognition. HMM-based recognition Learnability

11 Projected Keyboards Virtual Keyboard

12 Pointing Devices Target acquisition Steering / positioning Tracking Gesture Freehand drawing Drawing lines Tracing and digitizing Clicking, Double-clicking, dragging

13 Trackball, Trackpad, Trackpoint What is Sensed Motion (e.g., mouse) Position (e.g., trackpad) Force (e.g., trackpoint)

14 Tracking Pointers ASL EYE-TRAC 7 eye tracking Brain-Computer Interface, MMSPG, EPFL Tobii X1 Light Eye Tracker

15 Fitts Law MT = a + b log 2 (A/W + 1) a : the start/stop time of the device b : the inherent speed of the device A : the distance from the starting point to the center of the target W: the width of the target (along the axis of motion) Target acquisition time is proportional to the log of the ratio of the Distance to the Width of the target. Applies to position control devices Same for direct and indirect

16 Fitts Law (cont.) At each time step, a user travels a proportion of the remaining distance, 1 r for some 0 < r < 1, towards the target. The remaining distance after N the sub-movement is r N A. A target is reached when the remaining distance is less than half the width of the target: r N A < W/2. r N W A 2 N log r W 2A W log2 2A log r 2 1 log The distance left to the target's centre is a function that decays exponentially over time. 2 r 2A log2 W MT = a + b log 2 (A/W + 1)

17 Pen/Touch Input Good for pointing and drawing Natural for gestures Possibility of multiple pointers Not an efficient way for text entry Handwriting Problems of interpretation Special characters/gestures

18 Touch Screen HTC One M8 ipad2, Fig. from en.wikipedia.org/wiki/ipad Surface Pro 3, Microsoft

19 Multi-Touch Screen J.Y. Han, Low-Cost Multi-Touch Sensing through Frustrated Total Internal Reflection, Proc. UIST 05.

20 Multi-Touch Screen (cont.) Simple image processing operations Rectification, background subtraction, noise removal, and connected components analysis Low cost video capturing Inherent drawbacks of vision-based system Requires a significant amount of space Skin reflectance, pixel resolution, etc. contaminated surface

21 Tangible or Mixing Interaction S. Klemmer et al. Designers Outpost: A Tangible Interface for Collaborative Web Site Design, Proc. UIST 01. Designers Outpost

22 Designers Outpost ADD LINK REMOVE INK MOVE MENU SAVE

23 Designers Outpost Touch sensitive SMART board augmented with two digital cameras

24 Tangible or Mixing Interaction MetaDesk

25 Tangible UI (Illuminating Clay) B. Piper, C. Ratti, Y. Wang, B. Zhu, S. Getzoyan, & H. Ishii, Proc. CHI Allow users to interact with real freeform surfaces Users Molds surface of clay System Determines height of surface Projects imagery corresponding to tasks

26 Illuminating Clay (cont.)

27 Pointing in 3D User points with a virtual ray Visualized as a line segment attached to hand Immersive: Ray attached to 6DOF-tracked hand Desktop: Ray controlled by mouse Disambiguate by selecting closest object Hand/tracker jitter amplified by distance Makes it difficult to pick distant objects with precision M. Mine, ISAAC, 1997

28 Pointing in 3D Pointing: Fishing Reel Use additional device to specify length of pointing Vector Allows user to reel object in and out, changing its position along the ray Bowman, 1997

29 3D and Haptic DataGlove Phantom GyroMouse CyberGrasp

30 Wii and Wii remote

31 Wii Remote Controller Accurate positions IR camera + sensor bar Fixed distance between LEDs. Estimate relative positions from sensor views.

32 Wii Remote Controller (cont.) 3-axis linear accelerometer : at least +/- 3g with 10% sensitivity (Recently, accelerometers start being used in laptops or cell phones for damage protection or entertainment purposes.) MotionPlus (add-on) Gyroscope for angular velocities.

33 Pointing in 3D Vibration (Micro Vibrator) Sensor(Touch Sensor / 6-axis (acceleration angular speed) motion sensor) Continuous standby time:approx. 18 days Charge time:approx. 3 hours

34 Active Depth Sensors Can handle the matching problem at textureless regions. Instead of using multiple cameras, active vision systems project laser, IR, or visible light stripes or patterns. Xtion Pro Live, ASUS Fig. from Depth image of Xtion Pro Live, ASUS Fig. from Kinect, Microsoft Fig. from

35 35 Motion Capture Utilizing multiple infrared cameras for stable tracking. Protruding markers help with shape analysis but may obstruct facial motion. The commercial motion capture devices are very accurate but also expensive.

36 Motion Capture (cont.) Visualeyez, Phoenix Technologies Inc.

37 Motion Capture (cont.) Active markers or passive ones? Optotrak, NDI vicon8

38 Marker Tracking Markers in the 1 st frame Potential 3D candidates A B C D t = 1 t = 2 t = 3 t = 4

39 The Tracking problem Without missing markers : A graph problem : minimum cost network flow (MCNF) O(N3logNC). With missing markers: For optimal solutions, time-consuming dynamic programming etc. are required. For near real-time tracking, Kalman filters are insufficient.

40 False Tracking and Conflict Markers in the 1 st frame Potential 3D candidates Missing markers A B C D t = 1 t = 2 t = 3 t = 4

41 False Tracking and Conflict The effects of false tracking are accumulative. t =20 t =50 t =100 t =300 t =500 I.-C. Lin et al., Visual Comp 05

42 Detection and rectification Markers in the 1 st frame Potential 3D candidates Conjectural markers A B C D t = 1 t = 2 t = 3 t = 4

43 Motion Tracking (Body Surface) Data collection & cleaning Merging disconnected trajectories Hole filling Skin animation Capture session Resulting animation Surface model in rest pose S. Park, J.K. Hodgins, Capturing and Animating Skin Deformation in Human Motion, Proc. SIGGRAPH 06

44 Data Collection and Cleaning Occlusions happen in a large region for a long time

45 Data Collection and Cleaning Merging disconnected trajectories Hole filling Reference pose PCA

46 Capture and Synthesis

47 Walking Natural Provides vestibular cues Must track entire VE Scaling is possible Tracker weight, cables, environmental obstacles Outdoor, as well as indoor Fatigue UNC Ceiling Tracker Columbia Univ. Touring Machine

48 Walking Simulators Early attempts used a conventional treadmill No natural way to change direction Track user s head/feet and rotate entire treadmill Omnidirectional treadmills Two sets of orthogonal rollers/belts allow arbitrary yaw, but slowly GaitMaster (H. Iwata et al., U. Tsukuba) Tracked user walks on two platforms that can be continuously moved in 3D Torus Treadmill Typically slow and dangerous! GaitMaster

49 Walking Simulators CirculaFloor (H. Iwata et al., SIGGRAPH 04 E-Tech, IEEE CG&A 05) Four tracked moving tiles automatically reconfigure to meet feet of (slow) tracked user Different configurations correspond to tracked user direction

50 Displays Displays, aka output devices, present information to human senses. For a single user or multiple users. Scene representations are rendered or composed into a form that can be presented by a display. Computer graphics and image processing tech. Properties Sampling Resolution

51 Visual Perception Range 400~700 nm

52 Visual Perception (cont.) Temporal resolution ~ Hz (and higher): flicker fusion frequency Video frame rates are not necessarily equal to flicker rates Dependent on brightness, location on the retina (it is higher for a brighter light source) Field of view Both eyes: ~ 200 horizontal, 120 stereo overlap One eye: ~ 60 toward nose, 90 toward side, 50 up, 70 down

53 Geometry of Large display Surface geometry Planar Curved Hemispheric Multiple surfaces

54 Interactive Fog Screen I. Rakkolainen et al., The Interactive FogScreen (SIGGRAPH Emerging Tech.)

55 Visual Cues E.g. depth cues, helping us understand the depth of objects Monocular, static (pictorial) Monocular, dynamic Binocular

56 Monocular, Static Cues Relative size Occlusion Perspective Linear Aerial

57 Monocular, Static Cues Lighting Shadow Texture gradients

58 Monocular, Dynamic Cues Motion parallax

59 Binocular Cues wickelgren/psyc110/stereopsis.jpg

60 Optical See-Through Displays Augmented info. optics: mirror beam splitter = combiner Real world is full resolution/fov, but virtual world cannot block real world virtual/real worlds at distances Saab AddVisor 150 Google glass Fig from en.wikipedia.org/wiki/google_glass

61 Video See-Through Displays Videomixing Real world is low resolution/fov, but virtual world can block real world virtual/real worlds at same distance problems with camera mounting/optics Camera Synthetic graphics Mixed Reality Systems Lab COASTAR

62 Stereoscopic Viewing Spatial multiplexing Stereoscope Presents each eye with its own view Viewing in virtual environment Changing views according to head poses. users.telenet.be/thomasweynants /stereoscope.html

63 Stereoscopic Viewing Temporal multiplexing Liquid crystal shutter eyewear Right/left lenses alternatively opacify in synchrony with display of left/right views Synchronized to infrared emitter

64 Stereoscopic Viewing Spectral multiplexing Red blue anaglyph stereo Display overlapped left/right images in red/blue. Can process only edges to retain color in object interior mars.jpl.nasa.gov/mpf/mpf/anaglyph-arc.htm

65 Stereoscopic Viewing Polarization multiplexing Linear (horizontal/vertical): limited head tilting Circular (lefthanded/right-handed) Screen must preserve polarization (metallized)

66 Autostereoscopic Viewing Spatial multiplexing Parallax barrier methods Parallax stereogram by interleaving the columns from two images

67 Autostereoscopic Viewing Spatial multiplexing Parallax barrier methods Parallax panoramagram Can use physically moving mechanical or virtually moving liquid crystal barrier to create active display for a tracked user

68 Autostereoscopic Viewing Spatial multiplexing Lenticular approach Array of cylindrical lenses Brighter than barrier

69 Multi-layer 3D Display PureDepth Multi-Layer Display 3D Display

70 CAVE (Cave Automatic Virtual Environment)

71 Advanced Virtual Environment M. Gross et al, blue-c: A Spatially Immersive Display and 3D Video Portal for Telepresence,, Proc. SIGGRAPH 03.

72 Blue-C

73 Blue-C (cont.)

74 Consumer-level Virtual Environment Recently, the prices of virtual environment produces have reached the consumer level. VR may become more common and closer to our daily lives. Oculus Rift Virtuix Omni