Robust Articulated ICP for Real-Time Hand Tracking

Size: px
Start display at page:

Download "Robust Articulated ICP for Real-Time Hand Tracking"

Transcription

1 Robust Articulated-ICP for Real-Time Hand Tracking Andrea Tagliasacchi* Sofien Bouaziz Matthias Schröder* Mario Botsch Anastasia Tkach Mark Pauly * equal contribution 1/36

2 Real-Time Tracking Setup Data from (single) RGBD Sensors PrimeSense (Carmine) SoftKinetic Intel RealSense low SNR along in depth (z-axis) completely discards small portions of geometry 2

3 Motivation? Augmented Reality Intel Perceptual SDK Oculus Research (VR) MagicLeap (AR) Microsoft HoloLens - PR Video (hololens.com) HoloLens (AR) 3

4 Previous Work Appearance-based (guess solely based on current frame) Model-based (registers model of previous frame) 4

5 Previous Work Appearance vision Model geometry [Keskin ICCV 12] [Qian CVPR 14] [Oikono. CVPR 14] [Tompson SIG 14] [Melax I3D 13] [Tang CVPR 14] [Sridhar 3DV 14] [Schroder ICRA 14] 5

6 Contributions combined 2D/3D registration (within ICP) occlusion-aware correspondences (ICP) regularization with statistical pose-space prior extensible and unified real-time solver (>60fps) revamping ICP for articulated tracking 6

7 ICP: Iterative Closest Point Step 1: optimizing correspondences Step 2: optimizing transformations update correspondences update transformation update correspondences update transformation for more details please refer to Sparse-ICP [Bouaziz, Tagliasacchi, Pauly SGP 13] 7

8 Robust Articulated ICP [Wei et al. SIGA 12] our tracking process successfully tracks the entire motion sequence while ICP fails to track most of frames. This is because ICP is often sensitive to initial poses and prone to local minimum, particularly involving tracking high-dimensional human body poses from noisy depth data. [Zhang et al. SIGA 14] The accompanying video clearly shows that our tracking process is much more robust than the ICP algorithm [ ] our tracking process successfully tracks the entire motion sequence while ICP fails to track most of frames. [Qian et al. CVPR 14] It uses alternate and gradient based optimization, converges fast, and is suitable for realtime applications. However, it can be easily trapped in poor local optima and cannot handle non-rigid objects well. Yet, it is still insufficient for high-dimensional articulated hands, especially under free viewpoints. 8

9 System Overview color image silhouette posed model lost tracking? no yes converged? no yes reinitialize 2D correspondences linear 3D correspondences data fitting solve temporal depth image distance trans. point cloud bounds pose space collision min E points + E silh. +E wrist {z } Fitting terms + E pose + E kin. + E temporal {z } Prior terms 9

10 Preprocessing S s -sensorsilhouette color to identify the Region-of-Interest demo: assumption on picking +y for PCA but all this could be learned!! [Tompson et al. TOG 14] 10

11 Data Fitting Energies color image silhouette posed model lost tracking? no yes converged? no yes reinitialize 2D correspondences linear 3D correspondences data fitting solve temporal depth image distance trans. point cloud bounds pose space collision min E points + E silh. +E wrist {z } Fitting terms + E pose + E kin. + E temporal {z } Prior terms 11

12 3D Registration (w/ occlusions) X s -sensorpointcloud E points =! 1 X x2x s kx M (x, )k 1 2 x M ICP (case #1) ICP (case #2) our method (case #2) M -cylinderhandmodel 3D Registration 12

13 Lesson #1: insufficient to render X s -sensorpointcloud Correspondences of [Wei et al. SIGA 12] (renders the hand model into a point cloud) c 1 c 2 c 1 c 2 Correspondences of [Our Method] (computes correspondences in close form) Ground Truth Motion (finger 2 comes out of occlusion) c 1 c 2 M -cylinderhandmodel 3D Registration 13

14 2D/3D Registration X s -sensorpointcloud M -cylinderhandmodel 3D Registration 14

15 2D/3D Registration X s -sensorpointcloud S s -sensorsilhouette M -cylinderhandmodel 3D Registration S r -renderedsilhouette 2D Registration 15

16 2D Registration S s -sensorsilhouette E silhouette =! 2 X p2s r kp Ss (p, )k 2 2 x Ss S r -renderedsilhouette 2D Registration 16

17 Model Prior Energies color image silhouette posed model lost tracking? no yes converged? no yes reinitialize 2D correspondences linear 3D correspondences data fitting solve temporal depth image distance trans. point cloud bounds pose space collision min E points + E silh. +E wrist {z } Fitting terms + E pose + E kin. + E temporal {z } Prior terms 17

18 Joint Bounds Energy 18

19 Collision Energy 19

20 Temporal Coherence Energy 20

21 Statistical Pose Prior encodes correlation across joints 21

22 Statistical Pose Prior Recorded by VICON tracking system [Schroder ICRA 14] (they are accurate for the person that have been recorded for) 22

23 Statistical Pose Prior (Soft) E pose =! 4 k (µ + P )k 2 2 +! 5 k k 2 2 we optimize the current pose So that it is similar to a reconstructed pose from the low dimensional subspace but when DOF are unconstrained we would like to restore the neutral (i.e. mean) pose. µ 23

24 Statistical Pose Prior (Hard) E pose =! 4 k (µ + P )k 2 2 +! 5 k k 2 2! 4 = 1 only optimize in the subspace (i.e. variable replacement) even with a high number of bases the animation remains stiff!!! 24

25 CPU/GPU Optimization min X i Ē silh. =! 2 X kf i + J i k 2 2 = X i J T i J i 1 X i J T i f i p2s r (n T (J persp (x)j skel (x) +(p Ss (p, ))) 2 S r 20k!!!! J silh 20k 26 J T silhj silh =

26 Results and Limitations 26

27 Motion Transfer to Rig 27

28 Tracking with Fast Motion 28

29 Tracking of Interacting Hands 29

30 Limitation: Calibration 30

31 Limitations: Fist Rotation 31

32 State of the Art - Evaluations Dexter-1 Dataset (MPI) mm [Shroder ICRA 14] Subspace ICP [Tang et al. 2014] [Sridhar et al. 2013] [Sridhar et al. 2014] [ours] [ours] + re-init. [Sridhar ICCV 13] EM Tracker [Tang CVPR 14] Forest Classifiers adbadd flexex1 pinch count tigergrasp wave random [Melax 14] Intel Perceptual SDK (re-initialization helps because Dexter-1 is a low frame-rate dataset only 30Hz) [Tompson TOG 14] ConvNets [Qian CVPR 14] ICP/PSO Hybrid Qualitative Quantitative 32

33 Qualitative Comparisons Convex Dynamics (Intel SDK) Subspace ICP 33

34 Conclusions combined 2D/3D registration (within ICP) occlusion-aware correspondences (ICP) regularization with statistical pose-space prior extensible and unified real-time solver (>60fps) fully open source! 34

35 Shameless Advertisement Who? Prof. Andrea Tagliasacchi and Brian Wyvill What? MSc (PhD) Where? Who pays? Language? English 35

36 Thank you!!! Live demo at SGP 15!!! Don t be shy!! Sofien Matthias Mark Anastasia 36

37 Extra Slides 37

38 Statistical Prior: Side Effects E pose =! 4 k (µ + P )k 2 2 +! 5 k k as this prior correlates joint angles the convergence speed increases by about 3x 38

39 Reinitialization from Failure Determine tracking failure [Melax 13] Detection by ~[Qian et al. CVPR 14] 1 = X x2x s kx M (x, )k 2, 2 = S r \ S s S r [ S s xy-fingers Logistic Regressor (optimize alpha s.t.): 2 1+e T [ 1, 2 ]! tracking ok! z-fingers Exploration of corresp. energy 39

40 Particle Swarm 40

Accurate 3D Face and Body Modeling from a Single Fixed Kinect

Accurate 3D Face and Body Modeling from a Single Fixed Kinect Accurate 3D Face and Body Modeling from a Single Fixed Kinect Ruizhe Wang*, Matthias Hernandez*, Jongmoo Choi, Gérard Medioni Computer Vision Lab, IRIS University of Southern California Abstract In this

More information

Intrinsic3D: High-Quality 3D Reconstruction by Joint Appearance and Geometry Optimization with Spatially-Varying Lighting

Intrinsic3D: High-Quality 3D Reconstruction by Joint Appearance and Geometry Optimization with Spatially-Varying Lighting Intrinsic3D: High-Quality 3D Reconstruction by Joint Appearance and Geometry Optimization with Spatially-Varying Lighting R. Maier 1,2, K. Kim 1, D. Cremers 2, J. Kautz 1, M. Nießner 2,3 Fusion Ours 1

More information

Deep Learning for Virtual Shopping. Dr. Jürgen Sturm Group Leader RGB-D

Deep Learning for Virtual Shopping. Dr. Jürgen Sturm Group Leader RGB-D Deep Learning for Virtual Shopping Dr. Jürgen Sturm Group Leader RGB-D metaio GmbH Augmented Reality with the Metaio SDK: IKEA Catalogue App Metaio: Augmented Reality Metaio SDK for ios, Android and Windows

More information

Dynamic Geometry Processing

Dynamic Geometry Processing Dynamic Geometry Processing EG 2012 Tutorial Will Chang, Hao Li, Niloy Mitra, Mark Pauly, Michael Wand Tutorial: Dynamic Geometry Processing 1 Articulated Global Registration Introduction and Overview

More information

Dynamic 2D/3D Registration for the Kinect

Dynamic 2D/3D Registration for the Kinect Dynamic 2D/3D Registration for the Kinect Sofien Bouaziz Mark Pauly École Polytechnique Fédérale de Lausanne Abstract Image and geometry registration algorithms are an essential component of many computer

More information

A novel approach to motion tracking with wearable sensors based on Probabilistic Graphical Models

A novel approach to motion tracking with wearable sensors based on Probabilistic Graphical Models A novel approach to motion tracking with wearable sensors based on Probabilistic Graphical Models Emanuele Ruffaldi Lorenzo Peppoloni Alessandro Filippeschi Carlo Alberto Avizzano 2014 IEEE International

More information

Back To RGB: Deep Articulated Hand Pose Estimation From a Single Camera Image

Back To RGB: Deep Articulated Hand Pose Estimation From a Single Camera Image Back To RGB: Deep Articulated Hand Pose Estimation From a Single Camera Image Wan-Duo Kurt Ma, J.P. Lewis, Marcus Frean and David Balduzzi School of Engineering and Computer Science, Victoria University

More information

3D Computer Vision. Structured Light II. Prof. Didier Stricker. Kaiserlautern University.

3D Computer Vision. Structured Light II. Prof. Didier Stricker. Kaiserlautern University. 3D Computer Vision Structured Light II Prof. Didier Stricker Kaiserlautern University http://ags.cs.uni-kl.de/ DFKI Deutsches Forschungszentrum für Künstliche Intelligenz http://av.dfki.de 1 Introduction

More information

Single-shot Extrinsic Calibration of a Generically Configured RGB-D Camera Rig from Scene Constraints

Single-shot Extrinsic Calibration of a Generically Configured RGB-D Camera Rig from Scene Constraints Single-shot Extrinsic Calibration of a Generically Configured RGB-D Camera Rig from Scene Constraints Jiaolong Yang*, Yuchao Dai^, Hongdong Li^, Henry Gardner^ and Yunde Jia* *Beijing Institute of Technology

More information

Structured Light II. Thanks to Ronen Gvili, Szymon Rusinkiewicz and Maks Ovsjanikov

Structured Light II. Thanks to Ronen Gvili, Szymon Rusinkiewicz and Maks Ovsjanikov Structured Light II Johannes Köhler Johannes.koehler@dfki.de Thanks to Ronen Gvili, Szymon Rusinkiewicz and Maks Ovsjanikov Introduction Previous lecture: Structured Light I Active Scanning Camera/emitter

More information

Lecture 19: Depth Cameras. Visual Computing Systems CMU , Fall 2013

Lecture 19: Depth Cameras. Visual Computing Systems CMU , Fall 2013 Lecture 19: Depth Cameras Visual Computing Systems Continuing theme: computational photography Cameras capture light, then extensive processing produces the desired image Today: - Capturing scene depth

More information

Chaplin, Modern Times, 1936

Chaplin, Modern Times, 1936 Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection set] Multi-view geometry problems Structure: Given projections

More information

Multiframe Scene Flow with Piecewise Rigid Motion. Vladislav Golyanik,, Kihwan Kim, Robert Maier, Mathias Nießner, Didier Stricker and Jan Kautz

Multiframe Scene Flow with Piecewise Rigid Motion. Vladislav Golyanik,, Kihwan Kim, Robert Maier, Mathias Nießner, Didier Stricker and Jan Kautz Multiframe Scene Flow with Piecewise Rigid Motion Vladislav Golyanik,, Kihwan Kim, Robert Maier, Mathias Nießner, Didier Stricker and Jan Kautz Scene Flow. 2 Scene Flow. 3 Scene Flow. Scene Flow Estimation:

More information

Nonrigid Surface Modelling. and Fast Recovery. Department of Computer Science and Engineering. Committee: Prof. Leo J. Jia and Prof. K. H.

Nonrigid Surface Modelling. and Fast Recovery. Department of Computer Science and Engineering. Committee: Prof. Leo J. Jia and Prof. K. H. Nonrigid Surface Modelling and Fast Recovery Zhu Jianke Supervisor: Prof. Michael R. Lyu Committee: Prof. Leo J. Jia and Prof. K. H. Wong Department of Computer Science and Engineering May 11, 2007 1 2

More information

Registration of Dynamic Range Images

Registration of Dynamic Range Images Registration of Dynamic Range Images Tan-Chi Ho 1,2 Jung-Hong Chuang 1 Wen-Wei Lin 2 Song-Sun Lin 2 1 Department of Computer Science National Chiao-Tung University 2 Department of Applied Mathematics National

More information

Colored Point Cloud Registration Revisited Supplementary Material

Colored Point Cloud Registration Revisited Supplementary Material Colored Point Cloud Registration Revisited Supplementary Material Jaesik Park Qian-Yi Zhou Vladlen Koltun Intel Labs A. RGB-D Image Alignment Section introduced a joint photometric and geometric objective

More information

Real-time Image-based Reconstruction of Pipes Using Omnidirectional Cameras

Real-time Image-based Reconstruction of Pipes Using Omnidirectional Cameras Real-time Image-based Reconstruction of Pipes Using Omnidirectional Cameras Dipl. Inf. Sandro Esquivel Prof. Dr.-Ing. Reinhard Koch Multimedia Information Processing Christian-Albrechts-University of Kiel

More information

Dynamic Time Warping for Binocular Hand Tracking and Reconstruction

Dynamic Time Warping for Binocular Hand Tracking and Reconstruction Dynamic Time Warping for Binocular Hand Tracking and Reconstruction Javier Romero, Danica Kragic Ville Kyrki Antonis Argyros CAS-CVAP-CSC Dept. of Information Technology Institute of Computer Science KTH,

More information

User-assisted Segmentation and 3D Motion Tracking. Michael Fleder Sudeep Pillai Jeremy Scott

User-assisted Segmentation and 3D Motion Tracking. Michael Fleder Sudeep Pillai Jeremy Scott User-assisted Segmentation and 3D Motion Tracking Michael Fleder Sudeep Pillai Jeremy Scott 3D Object Tracking Virtual reality and animation Imitation in robotics Autonomous driving Augmented reality Motivation

More information

Learning 6D Object Pose Estimation and Tracking

Learning 6D Object Pose Estimation and Tracking Learning 6D Object Pose Estimation and Tracking Carsten Rother presented by: Alexander Krull 21/12/2015 6D Pose Estimation Input: RGBD-image Known 3D model Output: 6D rigid body transform of object 21/12/2015

More information

International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Vol. XXXIV-5/W10

International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Vol. XXXIV-5/W10 BUNDLE ADJUSTMENT FOR MARKERLESS BODY TRACKING IN MONOCULAR VIDEO SEQUENCES Ali Shahrokni, Vincent Lepetit, Pascal Fua Computer Vision Lab, Swiss Federal Institute of Technology (EPFL) ali.shahrokni,vincent.lepetit,pascal.fua@epfl.ch

More information

Monocular Tracking and Reconstruction in Non-Rigid Environments

Monocular Tracking and Reconstruction in Non-Rigid Environments Monocular Tracking and Reconstruction in Non-Rigid Environments Kick-Off Presentation, M.Sc. Thesis Supervisors: Federico Tombari, Ph.D; Benjamin Busam, M.Sc. Patrick Ruhkamp 13.01.2017 Introduction Motivation:

More information

TEXTURE OVERLAY ONTO NON-RIGID SURFACE USING COMMODITY DEPTH CAMERA

TEXTURE OVERLAY ONTO NON-RIGID SURFACE USING COMMODITY DEPTH CAMERA TEXTURE OVERLAY ONTO NON-RIGID SURFACE USING COMMODITY DEPTH CAMERA Tomoki Hayashi 1, Francois de Sorbier 1 and Hideo Saito 1 1 Graduate School of Science and Technology, Keio University, 3-14-1 Hiyoshi,

More information

SurfNet: Generating 3D shape surfaces using deep residual networks-supplementary Material

SurfNet: Generating 3D shape surfaces using deep residual networks-supplementary Material SurfNet: Generating 3D shape surfaces using deep residual networks-supplementary Material Ayan Sinha MIT Asim Unmesh IIT Kanpur Qixing Huang UT Austin Karthik Ramani Purdue sinhayan@mit.edu a.unmesh@gmail.com

More information

Accurate, Robust, and Flexible Real-time Hand Tracking

Accurate, Robust, and Flexible Real-time Hand Tracking Accurate, Robust, and Flexible Real-time Hand Tracking Toby Sharp Cem Keskin Duncan Robertson Jonathan Taylor Jamie Shotton David Kim Christoph Rhemann Ido Leichter Alon Vinnikov Yichen Wei Daniel Freedman

More information

arxiv: v3 [cs.cv] 10 Jan 2018

arxiv: v3 [cs.cv] 10 Jan 2018 HAND SEGMENTATION FOR HAND-OBJECT INTERACTION FROM DEPTH MAP Byeongkeun Kang Kar-Han Tan Nan Jiang Hung-Shuo Tai Daniel Tretter Truong Nguyen Department of Electrical and Computer Engineering, UC San Diego,

More information

Modeling 3D Human Poses from Uncalibrated Monocular Images

Modeling 3D Human Poses from Uncalibrated Monocular Images Modeling 3D Human Poses from Uncalibrated Monocular Images Xiaolin K. Wei Texas A&M University xwei@cse.tamu.edu Jinxiang Chai Texas A&M University jchai@cse.tamu.edu Abstract This paper introduces an

More information

Dense 3D Reconstruction from Autonomous Quadrocopters

Dense 3D Reconstruction from Autonomous Quadrocopters Dense 3D Reconstruction from Autonomous Quadrocopters Computer Science & Mathematics TU Munich Martin Oswald, Jakob Engel, Christian Kerl, Frank Steinbrücker, Jan Stühmer & Jürgen Sturm Autonomous Quadrocopters

More information

3D Reconstruction with Tango. Ivan Dryanovski, Google Inc.

3D Reconstruction with Tango. Ivan Dryanovski, Google Inc. 3D Reconstruction with Tango Ivan Dryanovski, Google Inc. Contents Problem statement and motivation The Tango SDK 3D reconstruction - data structures & algorithms Applications Developer tools Problem formulation

More information

High-Fidelity Augmented Reality Interactions Hrvoje Benko Researcher, MSR Redmond

High-Fidelity Augmented Reality Interactions Hrvoje Benko Researcher, MSR Redmond High-Fidelity Augmented Reality Interactions Hrvoje Benko Researcher, MSR Redmond New generation of interfaces Instead of interacting through indirect input devices (mice and keyboard), the user is interacting

More information

TEXTURE OVERLAY ONTO NON-RIGID SURFACE USING COMMODITY DEPTH CAMERA

TEXTURE OVERLAY ONTO NON-RIGID SURFACE USING COMMODITY DEPTH CAMERA TEXTURE OVERLAY ONTO NON-RIGID SURFACE USING COMMODITY DEPTH CAMERA Tomoki Hayashi, Francois de Sorbier and Hideo Saito Graduate School of Science and Technology, Keio University, 3-14-1 Hiyoshi, Kohoku-ku,

More information

High-Fidelity Facial and Speech Animation for VR HMDs

High-Fidelity Facial and Speech Animation for VR HMDs High-Fidelity Facial and Speech Animation for VR HMDs Institute of Computer Graphics and Algorithms Vienna University of Technology Forecast facial recognition with Head-Mounted Display (HMD) two small

More information

4422 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 27, NO. 9, SEPTEMBER 2018

4422 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 27, NO. 9, SEPTEMBER 2018 4422 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 27, NO. 9, SEPTEMBER 2018 Robust 3D Hand Pose Estimation From Single Depth Images Using Multi-View CNNs Liuhao Ge, Hui Liang, Member, IEEE, Junsong Yuan,

More information

Augmenting Reality, Naturally:

Augmenting Reality, Naturally: Augmenting Reality, Naturally: Scene Modelling, Recognition and Tracking with Invariant Image Features by Iryna Gordon in collaboration with David G. Lowe Laboratory for Computational Intelligence Department

More information

Planning, Execution and Learning Application: Examples of Planning in Perception

Planning, Execution and Learning Application: Examples of Planning in Perception 15-887 Planning, Execution and Learning Application: Examples of Planning in Perception Maxim Likhachev Robotics Institute Carnegie Mellon University Two Examples Graph search for perception Planning for

More information

Overview. Related Work Tensor Voting in 2-D Tensor Voting in 3-D Tensor Voting in N-D Application to Vision Problems Stereo Visual Motion

Overview. Related Work Tensor Voting in 2-D Tensor Voting in 3-D Tensor Voting in N-D Application to Vision Problems Stereo Visual Motion Overview Related Work Tensor Voting in 2-D Tensor Voting in 3-D Tensor Voting in N-D Application to Vision Problems Stereo Visual Motion Binary-Space-Partitioned Images 3-D Surface Extraction from Medical

More information

Tri-modal Human Body Segmentation

Tri-modal Human Body Segmentation Tri-modal Human Body Segmentation Master of Science Thesis Cristina Palmero Cantariño Advisor: Sergio Escalera Guerrero February 6, 2014 Outline 1 Introduction 2 Tri-modal dataset 3 Proposed baseline 4

More information

3D object recognition used by team robotto

3D object recognition used by team robotto 3D object recognition used by team robotto Workshop Juliane Hoebel February 1, 2016 Faculty of Computer Science, Otto-von-Guericke University Magdeburg Content 1. Introduction 2. Depth sensor 3. 3D object

More information

Segmentation and Tracking of Partial Planar Templates

Segmentation and Tracking of Partial Planar Templates Segmentation and Tracking of Partial Planar Templates Abdelsalam Masoud William Hoff Colorado School of Mines Colorado School of Mines Golden, CO 800 Golden, CO 800 amasoud@mines.edu whoff@mines.edu Abstract

More information

ACQUIRING 3D models of deforming objects in real-life is

ACQUIRING 3D models of deforming objects in real-life is IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 1 Robust Non-rigid Motion Tracking and Surface Reconstruction Using L 0 Regularization Kaiwen Guo, Feng Xu, Yangang Wang, Yebin Liu, Member, IEEE

More information

Perceptual Grouping from Motion Cues Using Tensor Voting

Perceptual Grouping from Motion Cues Using Tensor Voting Perceptual Grouping from Motion Cues Using Tensor Voting 1. Research Team Project Leader: Graduate Students: Prof. Gérard Medioni, Computer Science Mircea Nicolescu, Changki Min 2. Statement of Project

More information

Compressive Sensing: Opportunities and Perils for Computer Vision

Compressive Sensing: Opportunities and Perils for Computer Vision Compressive Sensing: Opportunities and Perils for Computer Vision Rama Chellappa And Volkan Cevher (Rice) Joint work with Aswin C. Sankaranarayanan Dikpal Reddy Dr. Ashok Veeraraghavan (MERL) Prof. Rich

More information

3D Human Motion Analysis and Manifolds

3D Human Motion Analysis and Manifolds D E P A R T M E N T O F C O M P U T E R S C I E N C E U N I V E R S I T Y O F C O P E N H A G E N 3D Human Motion Analysis and Manifolds Kim Steenstrup Pedersen DIKU Image group and E-Science center Motivation

More information

Outline. 1 Why we re interested in Real-Time tracking and mapping. 3 Kinect Fusion System Overview. 4 Real-time Surface Mapping

Outline. 1 Why we re interested in Real-Time tracking and mapping. 3 Kinect Fusion System Overview. 4 Real-time Surface Mapping Outline CSE 576 KinectFusion: Real-Time Dense Surface Mapping and Tracking PhD. work from Imperial College, London Microsoft Research, Cambridge May 6, 2013 1 Why we re interested in Real-Time tracking

More information

3D Face and Hand Tracking for American Sign Language Recognition

3D Face and Hand Tracking for American Sign Language Recognition 3D Face and Hand Tracking for American Sign Language Recognition NSF-ITR (2004-2008) D. Metaxas, A. Elgammal, V. Pavlovic (Rutgers Univ.) C. Neidle (Boston Univ.) C. Vogler (Gallaudet) The need for automated

More information

Virtual Reality in Assembly Simulation

Virtual Reality in Assembly Simulation Virtual Reality in Assembly Simulation Collision Detection, Simulation Algorithms, and Interaction Techniques Contents 1. Motivation 2. Simulation and description of virtual environments 3. Interaction

More information

CRF Based Point Cloud Segmentation Jonathan Nation

CRF Based Point Cloud Segmentation Jonathan Nation CRF Based Point Cloud Segmentation Jonathan Nation jsnation@stanford.edu 1. INTRODUCTION The goal of the project is to use the recently proposed fully connected conditional random field (CRF) model to

More information

Robust Model-Free Tracking of Non-Rigid Shape. Abstract

Robust Model-Free Tracking of Non-Rigid Shape. Abstract Robust Model-Free Tracking of Non-Rigid Shape Lorenzo Torresani Stanford University ltorresa@cs.stanford.edu Christoph Bregler New York University chris.bregler@nyu.edu New York University CS TR2003-840

More information

3D Reconstruction of Dynamic Textures with Crowd Sourced Data. Dinghuang Ji, Enrique Dunn and Jan-Michael Frahm

3D Reconstruction of Dynamic Textures with Crowd Sourced Data. Dinghuang Ji, Enrique Dunn and Jan-Michael Frahm 3D Reconstruction of Dynamic Textures with Crowd Sourced Data Dinghuang Ji, Enrique Dunn and Jan-Michael Frahm 1 Background Large scale scene reconstruction Internet imagery 3D point cloud Dense geometry

More information

ECE 6554:Advanced Computer Vision Pose Estimation

ECE 6554:Advanced Computer Vision Pose Estimation ECE 6554:Advanced Computer Vision Pose Estimation Sujay Yadawadkar, Virginia Tech, Agenda: Pose Estimation: Part Based Models for Pose Estimation Pose Estimation with Convolutional Neural Networks (Deep

More information

Robotics Programming Laboratory

Robotics Programming Laboratory Chair of Software Engineering Robotics Programming Laboratory Bertrand Meyer Jiwon Shin Lecture 8: Robot Perception Perception http://pascallin.ecs.soton.ac.uk/challenges/voc/databases.html#caltech car

More information

3D Perception. CS 4495 Computer Vision K. Hawkins. CS 4495 Computer Vision. 3D Perception. Kelsey Hawkins Robotics

3D Perception. CS 4495 Computer Vision K. Hawkins. CS 4495 Computer Vision. 3D Perception. Kelsey Hawkins Robotics CS 4495 Computer Vision Kelsey Hawkins Robotics Motivation What do animals, people, and robots want to do with vision? Detect and recognize objects/landmarks Find location of objects with respect to themselves

More information

Computer Vision II Lecture 14

Computer Vision II Lecture 14 Computer Vision II Lecture 14 Articulated Tracking I 08.07.2014 Bastian Leibe RWTH Aachen http://www.vision.rwth-aachen.de leibe@vision.rwth-aachen.de Outline of This Lecture Single-Object Tracking Bayesian

More information

Personal Navigation and Indoor Mapping: Performance Characterization of Kinect Sensor-based Trajectory Recovery

Personal Navigation and Indoor Mapping: Performance Characterization of Kinect Sensor-based Trajectory Recovery Personal Navigation and Indoor Mapping: Performance Characterization of Kinect Sensor-based Trajectory Recovery 1 Charles TOTH, 1 Dorota BRZEZINSKA, USA 2 Allison KEALY, Australia, 3 Guenther RETSCHER,

More information

3D Photography: Active Ranging, Structured Light, ICP

3D Photography: Active Ranging, Structured Light, ICP 3D Photography: Active Ranging, Structured Light, ICP Kalin Kolev, Marc Pollefeys Spring 2013 http://cvg.ethz.ch/teaching/2013spring/3dphoto/ Schedule (tentative) Feb 18 Feb 25 Mar 4 Mar 11 Mar 18 Mar

More information

Fast and Robust Hand Tracking Using Detection-Guided Optimization

Fast and Robust Hand Tracking Using Detection-Guided Optimization Fast and Robust Hand Tracking Using Detection-Guided Optimization Srinath Sridhar 1, Franziska Mueller 1,2, Antti Oulasvirta 3, Christian Theobalt 1 1 Max Planck Institute for Informatics, 2 Saarland University,

More information

Task analysis based on observing hands and objects by vision

Task analysis based on observing hands and objects by vision Task analysis based on observing hands and objects by vision Yoshihiro SATO Keni Bernardin Hiroshi KIMURA Katsushi IKEUCHI Univ. of Electro-Communications Univ. of Karlsruhe Univ. of Tokyo Abstract In

More information

This document is downloaded from DR-NTU, Nanyang Technological University Library, Singapore.

This document is downloaded from DR-NTU, Nanyang Technological University Library, Singapore. This document is downloaded from DR-NTU, Nanyang Technological University Library, Singapore. Title Egocentric hand pose estimation and distance recovery in a single RGB image Author(s) Liang, Hui; Yuan,

More information

Learning Articulated Skeletons From Motion

Learning Articulated Skeletons From Motion Learning Articulated Skeletons From Motion Danny Tarlow University of Toronto, Machine Learning with David Ross and Richard Zemel (and Brendan Frey) August 6, 2007 Point Light Displays It's easy for humans

More information

Model-Based Reinforcement of Kinect Depth Data for Human Motion Capture Applications

Model-Based Reinforcement of Kinect Depth Data for Human Motion Capture Applications Sensors 2013, 13, 8835-8855; doi:10.3390/s130708835 OPEN ACCESS sensors ISSN 1424-8220 www.mdpi.com/journal/sensors Article Model-Based Reinforcement of Kinect Depth Data for Human Motion Capture Applications

More information

What have we leaned so far?

What have we leaned so far? What have we leaned so far? Camera structure Eye structure Project 1: High Dynamic Range Imaging What have we learned so far? Image Filtering Image Warping Camera Projection Model Project 2: Panoramic

More information

Dynamic Shape Tracking via Region Matching

Dynamic Shape Tracking via Region Matching Dynamic Shape Tracking via Region Matching Ganesh Sundaramoorthi Asst. Professor of EE and AMCS KAUST (Joint work with Yanchao Yang) The Problem: Shape Tracking Given: exact object segmentation in frame1

More information

Dense 3D Modelling and Monocular Reconstruction of Deformable Objects

Dense 3D Modelling and Monocular Reconstruction of Deformable Objects Dense 3D Modelling and Monocular Reconstruction of Deformable Objects Anastasios (Tassos) Roussos Lecturer in Computer Science, University of Exeter Research Associate, Imperial College London Overview

More information

LUMS Mine Detector Project

LUMS Mine Detector Project LUMS Mine Detector Project Using visual information to control a robot (Hutchinson et al. 1996). Vision may or may not be used in the feedback loop. Visual (image based) features such as points, lines

More information

Shape from Selfies : Human Body Shape Estimation using CCA Regression Forests

Shape from Selfies : Human Body Shape Estimation using CCA Regression Forests Shape from Selfies : Human Body Shape Estimation using CCA Regression Forests Endri Dibra 1 Cengiz Öztireli1 Remo Ziegler 2 Markus Gross 1 1 Department of Computer Science, ETH Zürich 2 Vizrt {edibra,cengizo,grossm}@inf.ethz.ch

More information

Surface Registration. Gianpaolo Palma

Surface Registration. Gianpaolo Palma Surface Registration Gianpaolo Palma The problem 3D scanning generates multiple range images Each contain 3D points for different parts of the model in the local coordinates of the scanner Find a rigid

More information

Iterative Estimation of 3D Transformations for Object Alignment

Iterative Estimation of 3D Transformations for Object Alignment Iterative Estimation of 3D Transformations for Object Alignment Tao Wang and Anup Basu Department of Computing Science, Univ. of Alberta, Edmonton, AB T6G 2E8, Canada Abstract. An Iterative Estimation

More information

The Future of Natural User Interaction

The Future of Natural User Interaction The Future of Natural User Interaction NUI: a great new discipline in the making Baining Guo Microsoft Research Asia Kinect from Insiders Perspective An interview with Kinect contributors Kinect from

More information

Augmented Reality VU. Computer Vision 3D Registration (2) Prof. Vincent Lepetit

Augmented Reality VU. Computer Vision 3D Registration (2) Prof. Vincent Lepetit Augmented Reality VU Computer Vision 3D Registration (2) Prof. Vincent Lepetit Feature Point-Based 3D Tracking Feature Points for 3D Tracking Much less ambiguous than edges; Point-to-point reprojection

More information

calibrated coordinates Linear transformation pixel coordinates

calibrated coordinates Linear transformation pixel coordinates 1 calibrated coordinates Linear transformation pixel coordinates 2 Calibration with a rig Uncalibrated epipolar geometry Ambiguities in image formation Stratified reconstruction Autocalibration with partial

More information

Localization, Mapping and Exploration with Multiple Robots. Dr. Daisy Tang

Localization, Mapping and Exploration with Multiple Robots. Dr. Daisy Tang Localization, Mapping and Exploration with Multiple Robots Dr. Daisy Tang Two Presentations A real-time algorithm for mobile robot mapping with applications to multi-robot and 3D mapping, by Thrun, Burgard

More information

arxiv: v1 [cs.cv] 28 Sep 2018

arxiv: v1 [cs.cv] 28 Sep 2018 Camera Pose Estimation from Sequence of Calibrated Images arxiv:1809.11066v1 [cs.cv] 28 Sep 2018 Jacek Komorowski 1 and Przemyslaw Rokita 2 1 Maria Curie-Sklodowska University, Institute of Computer Science,

More information

Grasp Recognition using a 3D Articulated Model and Infrared Images

Grasp Recognition using a 3D Articulated Model and Infrared Images Grasp Recognition using a 3D Articulated Model and Infrared Images Koichi Ogawara Institute of Industrial Science, Univ. of Tokyo, Tokyo, Japan Jun Takamatsu Institute of Industrial Science, Univ. of Tokyo,

More information

Recovering 6D Object Pose and Predicting Next-Best-View in the Crowd Supplementary Material

Recovering 6D Object Pose and Predicting Next-Best-View in the Crowd Supplementary Material Recovering 6D Object Pose and Predicting Next-Best-View in the Crowd Supplementary Material Andreas Doumanoglou 1,2, Rigas Kouskouridas 1, Sotiris Malassiotis 2, Tae-Kyun Kim 1 1 Imperial College London

More information

Multiple View Geometry

Multiple View Geometry Multiple View Geometry Martin Quinn with a lot of slides stolen from Steve Seitz and Jianbo Shi 15-463: Computational Photography Alexei Efros, CMU, Fall 2007 Our Goal The Plenoptic Function P(θ,φ,λ,t,V

More information

CS 775: Advanced Computer Graphics. Lecture 17 : Motion Capture

CS 775: Advanced Computer Graphics. Lecture 17 : Motion Capture CS 775: Advanced Computer Graphics Lecture 17 : History Study of human motion Leonardo da Vinci (1452 1519) History Study of human motion Edward J. Muybridge, 1830 1904 http://en.wikipedia.org/wiki/eadweard_muybridge

More information

Human Upper Body Pose Estimation in Static Images

Human Upper Body Pose Estimation in Static Images 1. Research Team Human Upper Body Pose Estimation in Static Images Project Leader: Graduate Students: Prof. Isaac Cohen, Computer Science Mun Wai Lee 2. Statement of Project Goals This goal of this project

More information

KinectFusion: Real-Time Dense Surface Mapping and Tracking

KinectFusion: Real-Time Dense Surface Mapping and Tracking KinectFusion: Real-Time Dense Surface Mapping and Tracking Gabriele Bleser Thanks to Richard Newcombe for providing the ISMAR slides Overview General: scientific papers (structure, category) KinectFusion:

More information

Computer Vision Lecture 17

Computer Vision Lecture 17 Computer Vision Lecture 17 Epipolar Geometry & Stereo Basics 13.01.2015 Bastian Leibe RWTH Aachen http://www.vision.rwth-aachen.de leibe@vision.rwth-aachen.de Announcements Seminar in the summer semester

More information

ECCV Presented by: Boris Ivanovic and Yolanda Wang CS 331B - November 16, 2016

ECCV Presented by: Boris Ivanovic and Yolanda Wang CS 331B - November 16, 2016 ECCV 2016 Presented by: Boris Ivanovic and Yolanda Wang CS 331B - November 16, 2016 Fundamental Question What is a good vector representation of an object? Something that can be easily predicted from 2D

More information

Does Dimensionality Reduction Improve the Quality of Motion Interpolation?

Does Dimensionality Reduction Improve the Quality of Motion Interpolation? Does Dimensionality Reduction Improve the Quality of Motion Interpolation? Sebastian Bitzer, Stefan Klanke and Sethu Vijayakumar School of Informatics - University of Edinburgh Informatics Forum, 10 Crichton

More information

Computer Vision Lecture 17

Computer Vision Lecture 17 Announcements Computer Vision Lecture 17 Epipolar Geometry & Stereo Basics Seminar in the summer semester Current Topics in Computer Vision and Machine Learning Block seminar, presentations in 1 st week

More information

Correspondence. CS 468 Geometry Processing Algorithms. Maks Ovsjanikov

Correspondence. CS 468 Geometry Processing Algorithms. Maks Ovsjanikov Shape Matching & Correspondence CS 468 Geometry Processing Algorithms Maks Ovsjanikov Wednesday, October 27 th 2010 Overall Goal Given two shapes, find correspondences between them. Overall Goal Given

More information

Single Camera Calibration

Single Camera Calibration Single Camera Calibration using Partially Visible Calibration Objects Based on Random Dots Marker Tracking Algorithm *Yuji Oyamada1,2, Pascal Fallavollita2, and Nassir Navab2 1. Keio University, Japan

More information

a real convergence computer vision finally becoming useful

a real convergence computer vision finally becoming useful Introduce the field a real convergence computer vision finally becoming useful Pikachu Nintendo - http://www.freelargeimages.com/wpcontent/uploads/2014/12/pikachu_04.png HoloLens Microsoft - http://www.glassappsource.com/wpcontent/uploads/2016/01/hololens.jpg

More information

Combining PGMs and Discriminative Models for Upper Body Pose Detection

Combining PGMs and Discriminative Models for Upper Body Pose Detection Combining PGMs and Discriminative Models for Upper Body Pose Detection Gedas Bertasius May 30, 2014 1 Introduction In this project, I utilized probabilistic graphical models together with discriminative

More information

PROBLEM FORMULATION AND RESEARCH METHODOLOGY

PROBLEM FORMULATION AND RESEARCH METHODOLOGY PROBLEM FORMULATION AND RESEARCH METHODOLOGY ON THE SOFT COMPUTING BASED APPROACHES FOR OBJECT DETECTION AND TRACKING IN VIDEOS CHAPTER 3 PROBLEM FORMULATION AND RESEARCH METHODOLOGY The foregoing chapter

More information

On-line and Off-line 3D Reconstruction for Crisis Management Applications

On-line and Off-line 3D Reconstruction for Crisis Management Applications On-line and Off-line 3D Reconstruction for Crisis Management Applications Geert De Cubber Royal Military Academy, Department of Mechanical Engineering (MSTA) Av. de la Renaissance 30, 1000 Brussels geert.de.cubber@rma.ac.be

More information

Structure-aware 3D Hand Pose Regression from a Single Depth Image

Structure-aware 3D Hand Pose Regression from a Single Depth Image Structure-aware 3D Hand Pose Regression from a Single Depth Image Jameel Malik 1,2, Ahmed Elhayek 1, and Didier Stricker 1 1 Department Augmented Vision, DFKI Kaiserslautern, Germany 2 NUST-SEECS, Pakistan

More information

Virtualized Reality Using Depth Camera Point Clouds

Virtualized Reality Using Depth Camera Point Clouds Virtualized Reality Using Depth Camera Point Clouds Jordan Cazamias Stanford University jaycaz@stanford.edu Abhilash Sunder Raj Stanford University abhisr@stanford.edu Abstract We explored various ways

More information

A Model-based Approach to Rapid Estimation of Body Shape and Postures Using Low-Cost Depth Cameras

A Model-based Approach to Rapid Estimation of Body Shape and Postures Using Low-Cost Depth Cameras A Model-based Approach to Rapid Estimation of Body Shape and Postures Using Low-Cost Depth Cameras Abstract Byoung-Keon D. PARK*, Matthew P. REED University of Michigan, Transportation Research Institute,

More information

Leveraging Depth Cameras and Wearable Pressure Sensors for Full-body Kinematics and Dynamics Capture

Leveraging Depth Cameras and Wearable Pressure Sensors for Full-body Kinematics and Dynamics Capture Leveraging Depth Cameras and Wearable Pressure Sensors for Full-body Kinematics and Dynamics Capture Peizhao Zhang 1 Kristin Siu 2 Jianjie Zhang 1 C. Karen Liu 2 Jinxiang Chai 1 1 Texas 2 Georgia A&M University

More information

3D Geometric Computer Vision. Martin Jagersand Univ. Alberta Edmonton, Alberta, Canada

3D Geometric Computer Vision. Martin Jagersand Univ. Alberta Edmonton, Alberta, Canada 3D Geometric Computer Vision Martin Jagersand Univ. Alberta Edmonton, Alberta, Canada Multi-view geometry Build 3D models from images Carry out manipulation tasks http://webdocs.cs.ualberta.ca/~vis/ibmr/

More information

Tracking. Hao Guan( 管皓 ) School of Computer Science Fudan University

Tracking. Hao Guan( 管皓 ) School of Computer Science Fudan University Tracking Hao Guan( 管皓 ) School of Computer Science Fudan University 2014-09-29 Multimedia Video Audio Use your eyes Video Tracking Use your ears Audio Tracking Tracking Video Tracking Definition Given

More information

Direct Plane Tracking in Stereo Images for Mobile Navigation

Direct Plane Tracking in Stereo Images for Mobile Navigation Direct Plane Tracking in Stereo Images for Mobile Navigation Jason Corso, Darius Burschka,Greg Hager Computational Interaction and Robotics Lab 1 Input: The Problem Stream of rectified stereo images, known

More information

Robust Human Body Shape and Pose Tracking

Robust Human Body Shape and Pose Tracking Robust Human Body Shape and Pose Tracking Chun-Hao Huang 1 Edmond Boyer 2 Slobodan Ilic 1 1 Technische Universität München 2 INRIA Grenoble Rhône-Alpes Marker-based motion capture (mocap.) Adventages:

More information

Motion Tracking and Event Understanding in Video Sequences

Motion Tracking and Event Understanding in Video Sequences Motion Tracking and Event Understanding in Video Sequences Isaac Cohen Elaine Kang, Jinman Kang Institute for Robotics and Intelligent Systems University of Southern California Los Angeles, CA Objectives!

More information

Stereo and Epipolar geometry

Stereo and Epipolar geometry Previously Image Primitives (feature points, lines, contours) Today: Stereo and Epipolar geometry How to match primitives between two (multiple) views) Goals: 3D reconstruction, recognition Jana Kosecka

More information

A consumer level 3D object scanning device using Kinect for web-based C2C business

A consumer level 3D object scanning device using Kinect for web-based C2C business A consumer level 3D object scanning device using Kinect for web-based C2C business Geoffrey Poon, Yu Yin Yeung and Wai-Man Pang Caritas Institute of Higher Education Introduction Internet shopping is popular

More information

Pose estimation using a variety of techniques

Pose estimation using a variety of techniques Pose estimation using a variety of techniques Keegan Go Stanford University keegango@stanford.edu Abstract Vision is an integral part robotic systems a component that is needed for robots to interact robustly

More information