Interaction Error based Viewpoint Estimation for Continuous Parallax Error Correction on Interactive Screens

Transcription

1 Interaction Error based Viewpoint Estimation for Continuous Parallax Error Correction on Interactive Screens Bastian Migge (ETH Zurich), Andreas Kunz (ETH Zurich), Tim Schmidt (PARC)

2 Content Introduction/Motivation: Interactive surfaces Problem: Parallax error Calibration Techniques Model based viewpoint estimation Contribution: Empirical observation model for MPC User study Discrete Observation Model Conclusion and outlook 2

3 Interactive Surfaces Ticketing machine (New York Underground) n Pros n Digital Desk (Crion table, ETH Zurich) Single display groupware (Collaboard, ETH Zurich) Intuitive operation, easy to learn Innate interaction (input and output combined) Requisition Good alignment between image plane and tracking system is essential 3

4 Parallax Error ax Vx Image plane Vz Interaction plane 2 z az y User B n n n x User A Vz : Distance between image and interaction plane Vx : Resulting parallax distortion in x-dimension ax,z: Distance from user to interaction point in x,z Vz ax Vx = az Parallax Error (analogous for y) 1 Interaction plane and image pane with offset1 Source: Migge,Kunz: User Model for Predictive Calibration Control on Interactive Screens, IEEE CW

5 Static Calibration Initially to correct geometric distortion A-priori setup Is biased by user characteristics (height, arm length) Depends on a single viewpoint Static Calibration Process Can not deal with user s motion Can not handle multiple users Can not overcome the parallax error stemming from changing viewpoints (VP) 5

6 Continuous Calibration with Viewpoint Estimation Method 1 : Remove sensors - Estimate users viewpoint Add filter model the users position and movement Set the correction for the next (predicted position) interaction Consider effect of correction actions (user disturbance) à Model based Predictive Parallax Error Correction under Uncertainty (POMDP) Pro No additional hardware Cons Models needed (a priori) Interaction on screen needed (at runtime) 1 [Migge, Kunz, Schmidt: POMDP Models for Continuous Calibration of Interactive Surfaces, AAAI SS 2010] 6

7 Model based Predictive Parallax Error Correction under Uncertainty V x Image plane Target V z Interaction plane Touch point User's viewpoint Parallax correction (-V x ): shift pointing device information Information sources Target can be assumed to be next to Touch point Viewpoint is not directly observable VP can be inferred from GUI interactions (ß V x, V z ) Benefits Correction for all touch points Increases the pointing accuracy z x 7

8 Contribution: Viewpoint estimation from interaction error Contribution: A model of the user s interaction behavior to estimate the viewpoint of the user Static Characteristics: Display offset Iz Dynamic Values: Relative User position VPx Relative Interaction Error Ix VPx ~ Ix Observation Model: Pr(Ix = err VPx = vp) Target Image plane Iz Interaction Interaction plane Ix z VPz y x VPx User Dependency between relative viewpoint and relative pointing error 8

9 User Study Measurement Setup Participants: 13(4) (fe)male, avg. age 31.05, avg. height 179mm Interactive System 50 interactive surface (plasma screen with SMART tracking) Pixel pitch x mm, Resolution 1280 x 768 px On-screen target: 15 x 15 px (13 x 12 mm) Parallax offset 13 mm à Measures the Interaction error (in 2D display coordinates) Display body 10 Pen 785 mm 1215 mm Glas plane Touch sensitive film [mm] (a) Schematic measurement setup with test person (b) Assembly of the tracking system on top of the display 9

10 User Study - Task Random clicker User must click a single button of a full screen application Button moves after each interaction Python, QT, X11, Linux system à Provides global interaction position on display Test application 10

11 User Study Tracking Setup Tracking System: Qualisys Motion Tracking System 4 Oqus 300 Camera IR based out-side in system Measures Position of reflective marker (passive) 1 50 Hz; σ = 0.87 mm à Measures the 3D Viewpoint and transforms 2D display coordinates to global 3D coordinate system (a) Head tracking module (b) Passive marker on interactive screen

12 Vertical measurements User Study Results 1/2 Pointing Error 12 Pointing error Deviation from assumed hit point (haptic error) Boxplots Horizontal measurements Error [mm] Interactions [%] Pointing Error Deviation from assumed hit Error [mm] Interactions [%] Pointing Error Deviation from assumed hit Normalized deviation Pointing error Deviation from assumed hit point Image plane Interaction plane 4 User z x y target actual hit point assumed viewpoint assumed hit point Deviation from assumed hit Pointing error and Haptic error

13 User Study Results 2/2 Correlation 13 Normalized viewpoint interaction error 5500 interactions Horizontal measurements Vertical measurements Pearsons correlation coefficient ( ) Viewpoint Interaction error Least square fit Lowess fit Pearsons correlation coefficient ( ) Viewpoint Interaction error Least square fit Lowess fit

14 Data interpretation: Discrete Observation Model Measurement Data: Correlation between Interaction error and Viewpoint Allows the controller to infer the viewpoint from the interaction error Set up the Discrete Model Observation space O State space S P(O S=s) as probability distribution target center applied correction o left o hit o right image plane interaction plane actual viewpoint inferred viewpoint Observation (O) model for a given viewpoint and target (S) as discrete probability distribution P(O S) 14

15 Discrete Observation Model Concrete Example Continuous observation space, Discrete states Click error significantly differs for different viewpoints Discrete observation space 15 Horizontal click error for 5 discrete viewpoint states [ 1, 0.55 ) [ 0.15, 0.15 ) [ 0.55, 1 ) Viewpoint (State) Clickerror (Observation) [ 0.15, 0.15 ) mean var Density P(O s= left ) [ 0.15, 0.55 ) mean var Density P(O s= right )

16 Conclusion and Outlook Focus was to model the correlation between viewpoint position and interaction error on the screen Viewpoint can be distinguished model the user s behavior exists Complete empirical model The correlation between pointing accuracy and target size Effect of correction actions onto the user Develop prototype Prove of concept Compare controller to classical correction methods and tracking based correction 16

17 Bastian Migge 17

18 Continuous Calibration with Viewpoint Tracking Methods: Visual marker tracking or Video image feature extraction Pro High quality viewpoint position Cons Marker at tracked object not applicable Additional camera hardware Initial calibration pose needed Camera lens opening angle critical 3D Skelton Tracking with Microsoft Kinect and OpenNI

19 Model based Predictive Parallax Error Correction under Uncertainty in detail Control: Set the interaction correction parameter V x, V y Uncertainty: Interaction error on screen (Observations) does indicate the viewpoint under uncertainty Predictive: Model the User s behavior predicting his movement 1 Method: Partial Observable Markow Decision Process (POMDP) User s behavior modeled as time discrete Markow Chain (state space) Add control à Markow Decision Process (MDP) Uncertain measurements indicate states (POMDP) Needs a model of the user s interaction behavior 1 [Migge, Kunz :User Model for Predictive Calibration Control on Interactive Screens, IEEE CW 2010] 19