Deep Models for 3D Reconstruction

Size: px
Start display at page:

Download "Deep Models for 3D Reconstruction"

Transcription

1 Deep Models for 3D Reconstruction Andreas Geiger Autonomous Vision Group, MPI for Intelligent Systems, Tübingen Computer Vision and Geometry Group, ETH Zürich October 12, 2017 Max Planck Institute for Intelligent Systems Autonomous Vision Group

2 3D Reconstruction [Furukawa & Hernandez: Multi-View Stereo: A Tutorial] Task: Given a set of 2D images Reconstruct 3D shape of object/scene 2

3 3D Reconstruction Pipeline Input Images 3

4 3D Reconstruction Pipeline Input Images Camera Poses 3

5 3D Reconstruction Pipeline Input Images Camera Poses Dense Correspondences 3

6 3D Reconstruction Pipeline Input Images Camera Poses Dense Correspondences Depth Maps 3

7 3D Reconstruction Pipeline Input Images Camera Poses Dense Correspondences Depth Map Fusion Depth Maps 3

8 3D Reconstruction Pipeline Input Images Camera Poses Dense Correspondences 3D Reconstruction Depth Map Fusion Depth Maps 3

9 3D Reconstruction Pipeline Input Images Camera Poses Dense Correspondences 3D Reconstruction Depth Map Fusion Depth Maps 3

10 Large 3D Datasets and Repositories [Newcombe et al., 2011] [Choi et al., 2011] [Dai et al., 2017] [Wu et al., 2015] [Chang et al., 2015] [Chang et al., 2017] 4

11 Can we learn 3D Reconstruction from Data?

12 OctNet: Learning Deep 3D Representations at High Resolutions [Riegler, Ulusoy, & Geiger, CVPR 2017]

13 Deep Learning in 2D [LeCun, 1998] 7

14 Deep Learning in 3D 8

15 Deep Learning in 3D Existing 3D networks limited to 32 3 voxels 8

16 3D Data is often Sparse [Geiger et al., 2012] 9

17 3D Data is often Sparse [Li et al., 2016] 9

18 3D Data is often Sparse [Li et al., 2016] Can we exploit sparsity for efficient deep learning? 9

19 Network Activations Layer 1: 32 3 Layer 2: 16 3 Layer 3:

20 Network Activations Layer 1: 32 3 Layer 2: 16 3 Layer 3:

21 Network Activations Layer 1: 32 3 Layer 2: 16 3 Layer 3: 8 3 Idea: Partition space adaptively based on sparse input 10

22 Convolution 11

23 Convolution

24 Convolution

25 Convolution

26 Convolution

27 Convolution

28 Convolution

29 Convolution

30 Convolution

31 Convolution

32 Convolution

33 Convolution

34 Convolution

35 Convolution

36 Convolution

37 Convolution

38 Convolution 11

39 Convolution 11

40 Convolution Differentiable allows for end-to-end learning 11

41 Efficient Convolution This operation can be implemented very efficiently: 4 different cases First case requires only 1 evaluation! 12

42 Pooling 13

43 Pooling 13

44 Pooling 13

45 Pooling Unpooling operation defined similarly 13

46 Results: 3D Shape Classification Airplane Convolution and Pooling Convolution and Pooling Fully Conn. Fully Conn. 14

47 Results: 3D Shape Classification Memory [GB] OctNet DenseNet Input Resolution 15

48 Results: 3D Shape Classification Runtime [s] OctNet DenseNet Input Resolution 15

49 Results: 3D Shape Classification Accuracy OctNet 0.70 DenseNet Input Resolution Input: voxelized meshes from ModelNet 16

50 Results: 3D Shape Classification Accuracy OctNet 1 OctNet 2 OctNet Input Resolution Input: voxelized meshes from ModelNet 16

51 Results: 3D Shape Classification 17

52 Results: 3D Semantic Labeling Input I Prediction Dataset: RueMonge

53 Results: 3D Semantic Labeling Skip Skip Convolution and Pooling Convolution and Pooling Unpooling and Conv. Unpooling and Conv. Decoder octree structure copied from encoder 19

54 Results: 3D Semantic Labeling IoU [Riemenschneider et al., 2014] 42.3 [Martinovic et al., 2015] 52.2 [Gadde et al., 2016] 54.4 OctNet OctNet OctNet

55 OctNetFusion: Learning Depth Fusion from Data [Riegler, Ulusoy, Bischof & Geiger, 3DV 2017]

56 Volumetric Fusion d i+1 (p) = w i(p)d i (p) + ŵ(p) ˆd(p) w i (p) + ŵ(p) w i+1 (p) = w i (p) + ŵ(p) p R 3 : voxel location d: distance, w: weight [Curless and Levoy, SIGGRAPH 1996] 22

57 Volumetric Fusion Pros: Simple, fast, easy to implement Defacto gold standard (KinectFusion, Voxel Hashing,...) Ground Truth Volumetric Fusion 23

58 Volumetric Fusion Pros: Simple, fast, easy to implement Defacto gold standard (KinectFusion, Voxel Hashing,...) Cons: Requires many redundant views to reduce noise Can t handle outliers / complete missing surfaces Ground Truth Volumetric Fusion 23

59 TV-L1 Fusion Pros: Prior on surface area Noise reduction Ground Truth Volumetric Fusion TV-L1 Fusion 23

60 TV-L1 Fusion Pros: Prior on surface area Noise reduction Cons: Simplistic local prior (penalizes surface area, shrinking bias) Can t complete missing surfaces Ground Truth Volumetric Fusion TV-L1 Fusion 23

61 Learned Fusion Pros: Learn noise suppression from data Learn surface completion from data Ground Truth Volumetric Fusion TV-L1 Fusion OctNetFusion 23

62 Learned Fusion Pros: Learn noise suppression from data Learn surface completion from data Cons: Requires large 3D datasets for training How to scale to high resolutions? Ground Truth Volumetric Fusion TV-L1 Fusion OctNetFusion 23

63 Learning 3D Fusion Skip Skip Convolution and Pooling Convolution and Pooling Unpooling and Conv. Unpooling and Conv. Input Representation: TSDF Higher-order statistics Output Representation: Occupancy TSDF 24

64 Learning 3D Fusion Skip Skip Convolution and Pooling Convolution and Pooling Unpooling and Conv. Unpooling and Conv. What is the problem? 24

65 Learning 3D Fusion Skip Skip Convolution and Pooling Convolution and Pooling Unpooling and Conv. Unpooling and Conv. What is the problem? Octree structure unknown needs to be inferred as well! 24

66 OctNetFusion Architecture 64³ 64³ 64 Input Output Octree Structure 128³ Features 128³ 128 Input Output Octree Structure Features 256³ 256³ 256 Input Output 25

67 Results: Surface Reconstruction VolFus TV-L1 Ours Ground Truth

68 Results: Volumetric Completion [Firman, 2016] Ours Ground Truth 27

69 Thank you!

Geometric and Semantic 3D Reconstruction: Part 4A: Volumetric Semantic 3D Reconstruction. CVPR 2017 Tutorial Christian Häne UC Berkeley

Geometric and Semantic 3D Reconstruction: Part 4A: Volumetric Semantic 3D Reconstruction. CVPR 2017 Tutorial Christian Häne UC Berkeley Geometric and Semantic 3D Reconstruction: Part 4A: Volumetric Semantic 3D Reconstruction CVPR 2017 Tutorial Christian Häne UC Berkeley Dense Multi-View Reconstruction Goal: 3D Model from Images (Depth

More information

arxiv: v3 [cs.cv] 31 Oct 2017

arxiv: v3 [cs.cv] 31 Oct 2017 OctNetFusion: Learning Depth Fusion from Data Gernot Riegler 1 Ali Osman Ulusoy 2 Horst Bischof 1 Andreas Geiger 2,3 1 Institute for Computer Graphics and Vision, Graz University of Technology 2 Autonomous

More information

Learning from 3D Data

Learning from 3D Data Learning from 3D Data Thomas Funkhouser Princeton University* * On sabbatical at Stanford and Google Disclaimer: I am talking about the work of these people Shuran Song Andy Zeng Fisher Yu Yinda Zhang

More information

Intrinsic3D: High-Quality 3D Reconstruction by Joint Appearance and Geometry Optimization with Spatially-Varying Lighting

Intrinsic3D: High-Quality 3D Reconstruction by Joint Appearance and Geometry Optimization with Spatially-Varying Lighting Intrinsic3D: High-Quality 3D Reconstruction by Joint Appearance and Geometry Optimization with Spatially-Varying Lighting R. Maier 1,2, K. Kim 1, D. Cremers 2, J. Kautz 1, M. Nießner 2,3 Fusion Ours 1

More information

Geometric Reconstruction Dense reconstruction of scene geometry

Geometric Reconstruction Dense reconstruction of scene geometry Lecture 5. Dense Reconstruction and Tracking with Real-Time Applications Part 2: Geometric Reconstruction Dr Richard Newcombe and Dr Steven Lovegrove Slide content developed from: [Newcombe, Dense Visual

More information

CS468: 3D Deep Learning on Point Cloud Data. class label part label. Hao Su. image. May 10, 2017

CS468: 3D Deep Learning on Point Cloud Data. class label part label. Hao Su. image. May 10, 2017 CS468: 3D Deep Learning on Point Cloud Data class label part label Hao Su image. May 10, 2017 Agenda Point cloud generation Point cloud analysis CVPR 17, Point Set Generation Pipeline render CVPR 17, Point

More information

Learning to generate 3D shapes

Learning to generate 3D shapes Learning to generate 3D shapes Subhransu Maji College of Information and Computer Sciences University of Massachusetts, Amherst http://people.cs.umass.edu/smaji August 10, 2018 @ Caltech Creating 3D shapes

More information

Perceiving the 3D World from Images and Videos. Yu Xiang Postdoctoral Researcher University of Washington

Perceiving the 3D World from Images and Videos. Yu Xiang Postdoctoral Researcher University of Washington Perceiving the 3D World from Images and Videos Yu Xiang Postdoctoral Researcher University of Washington 1 2 Act in the 3D World Sensing & Understanding Acting Intelligent System 3D World 3 Understand

More information

Volumetric and Multi-View CNNs for Object Classification on 3D Data Supplementary Material

Volumetric and Multi-View CNNs for Object Classification on 3D Data Supplementary Material Volumetric and Multi-View CNNs for Object Classification on 3D Data Supplementary Material Charles R. Qi Hao Su Matthias Nießner Angela Dai Mengyuan Yan Leonidas J. Guibas Stanford University 1. Details

More information

Project Updates Short lecture Volumetric Modeling +2 papers

Project Updates Short lecture Volumetric Modeling +2 papers Volumetric Modeling Schedule (tentative) Feb 20 Feb 27 Mar 5 Introduction Lecture: Geometry, Camera Model, Calibration Lecture: Features, Tracking/Matching Mar 12 Mar 19 Mar 26 Apr 2 Apr 9 Apr 16 Apr 23

More information

3D Object Recognition and Scene Understanding from RGB-D Videos. Yu Xiang Postdoctoral Researcher University of Washington

3D Object Recognition and Scene Understanding from RGB-D Videos. Yu Xiang Postdoctoral Researcher University of Washington 3D Object Recognition and Scene Understanding from RGB-D Videos Yu Xiang Postdoctoral Researcher University of Washington 1 2 Act in the 3D World Sensing & Understanding Acting Intelligent System 3D World

More information

Deep Incremental Scene Understanding. Federico Tombari & Christian Rupprecht Technical University of Munich, Germany

Deep Incremental Scene Understanding. Federico Tombari & Christian Rupprecht Technical University of Munich, Germany Deep Incremental Scene Understanding Federico Tombari & Christian Rupprecht Technical University of Munich, Germany C. Couprie et al. "Toward Real-time Indoor Semantic Segmentation Using Depth Information"

More information

Image Based Reconstruction II

Image Based Reconstruction II Image Based Reconstruction II Qixing Huang Feb. 2 th 2017 Slide Credit: Yasutaka Furukawa Image-Based Geometry Reconstruction Pipeline Last Lecture: Multi-View SFM Multi-View SFM This Lecture: Multi-View

More information

Semantic 3D Reconstruction of Heads Supplementary Material

Semantic 3D Reconstruction of Heads Supplementary Material Semantic 3D Reconstruction of Heads Supplementary Material Fabio Maninchedda1, Christian Ha ne2,?, Bastien Jacquet3,?, Amae l Delaunoy?, Marc Pollefeys1,4 1 ETH Zurich 2 UC Berkeley 3 Kitware SAS 4 Microsoft

More information

Outline. 1 Why we re interested in Real-Time tracking and mapping. 3 Kinect Fusion System Overview. 4 Real-time Surface Mapping

Outline. 1 Why we re interested in Real-Time tracking and mapping. 3 Kinect Fusion System Overview. 4 Real-time Surface Mapping Outline CSE 576 KinectFusion: Real-Time Dense Surface Mapping and Tracking PhD. work from Imperial College, London Microsoft Research, Cambridge May 6, 2013 1 Why we re interested in Real-Time tracking

More information

Dense 3D Reconstruction from Autonomous Quadrocopters

Dense 3D Reconstruction from Autonomous Quadrocopters Dense 3D Reconstruction from Autonomous Quadrocopters Computer Science & Mathematics TU Munich Martin Oswald, Jakob Engel, Christian Kerl, Frank Steinbrücker, Jan Stühmer & Jürgen Sturm Autonomous Quadrocopters

More information

Lecture 10 Dense 3D Reconstruction

Lecture 10 Dense 3D Reconstruction Institute of Informatics Institute of Neuroinformatics Lecture 10 Dense 3D Reconstruction Davide Scaramuzza 1 REMODE: Probabilistic, Monocular Dense Reconstruction in Real Time M. Pizzoli, C. Forster,

More information

Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputs Supplementary Material

Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputs Supplementary Material Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputs Supplementary Material Peak memory usage, GB 10 1 0.1 0.01 OGN Quadratic Dense Cubic Iteration time, s 10

More information

Computer Vision: Making machines see

Computer Vision: Making machines see Computer Vision: Making machines see Roberto Cipolla Department of Engineering http://www.eng.cam.ac.uk/~cipolla/people.html http://www.toshiba.eu/eu/cambridge-research- Laboratory/ Vision: what is where

More information

Multi-view stereo. Many slides adapted from S. Seitz

Multi-view stereo. Many slides adapted from S. Seitz Multi-view stereo Many slides adapted from S. Seitz Beyond two-view stereo The third eye can be used for verification Multiple-baseline stereo Pick a reference image, and slide the corresponding window

More information

Multi-View 3D Object Detection Network for Autonomous Driving

Multi-View 3D Object Detection Network for Autonomous Driving Multi-View 3D Object Detection Network for Autonomous Driving Xiaozhi Chen, Huimin Ma, Ji Wan, Bo Li, Tian Xia CVPR 2017 (Spotlight) Presented By: Jason Ku Overview Motivation Dataset Network Architecture

More information

CVPR 2014 Visual SLAM Tutorial Kintinuous

CVPR 2014 Visual SLAM Tutorial Kintinuous CVPR 2014 Visual SLAM Tutorial Kintinuous kaess@cmu.edu The Robotics Institute Carnegie Mellon University Recap: KinectFusion [Newcombe et al., ISMAR 2011] RGB-D camera GPU 3D/color model RGB TSDF (volumetric

More information

Depth from Stereo. Dominic Cheng February 7, 2018

Depth from Stereo. Dominic Cheng February 7, 2018 Depth from Stereo Dominic Cheng February 7, 2018 Agenda 1. Introduction to stereo 2. Efficient Deep Learning for Stereo Matching (W. Luo, A. Schwing, and R. Urtasun. In CVPR 2016.) 3. Cascade Residual

More information

08 An Introduction to Dense Continuous Robotic Mapping

08 An Introduction to Dense Continuous Robotic Mapping NAVARCH/EECS 568, ROB 530 - Winter 2018 08 An Introduction to Dense Continuous Robotic Mapping Maani Ghaffari March 14, 2018 Previously: Occupancy Grid Maps Pose SLAM graph and its associated dense occupancy

More information

3D Object Classification via Spherical Projections

3D Object Classification via Spherical Projections 3D Object Classification via Spherical Projections Zhangjie Cao 1,QixingHuang 2,andRamaniKarthik 3 1 School of Software Tsinghua University, China 2 Department of Computer Science University of Texas at

More information

Deconvolution Networks

Deconvolution Networks Deconvolution Networks Johan Brynolfsson Mathematical Statistics Centre for Mathematical Sciences Lund University December 6th 2016 1 / 27 Deconvolution Neural Networks 2 / 27 Image Deconvolution True

More information

Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture David Eigen, Rob Fergus

Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture David Eigen, Rob Fergus Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture David Eigen, Rob Fergus Presented by: Rex Ying and Charles Qi Input: A Single RGB Image Estimate

More information

3D Shape Segmentation with Projective Convolutional Networks

3D Shape Segmentation with Projective Convolutional Networks 3D Shape Segmentation with Projective Convolutional Networks Evangelos Kalogerakis 1 Melinos Averkiou 2 Subhransu Maji 1 Siddhartha Chaudhuri 3 1 University of Massachusetts Amherst 2 University of Cyprus

More information

Object Localization, Segmentation, Classification, and Pose Estimation in 3D Images using Deep Learning

Object Localization, Segmentation, Classification, and Pose Estimation in 3D Images using Deep Learning Allan Zelener Dissertation Proposal December 12 th 2016 Object Localization, Segmentation, Classification, and Pose Estimation in 3D Images using Deep Learning Overview 1. Introduction to 3D Object Identification

More information

Matryoshka Networks: Predicting 3D Geometry via Nested Shape Layers

Matryoshka Networks: Predicting 3D Geometry via Nested Shape Layers Matryoshka Networks: Predicting 3D Geometry via Nested Shape Layers Stephan R. Richter 1,2 Stefan Roth 1 1 TU Darmstadt 2 Intel Labs Figure 1. Exemplary shape reconstructions from a single by our Matryoshka

More information

3D Shape Analysis with Multi-view Convolutional Networks. Evangelos Kalogerakis

3D Shape Analysis with Multi-view Convolutional Networks. Evangelos Kalogerakis 3D Shape Analysis with Multi-view Convolutional Networks Evangelos Kalogerakis 3D model repositories [3D Warehouse - video] 3D geometry acquisition [KinectFusion - video] 3D shapes come in various flavors

More information

Lecture 10 Multi-view Stereo (3D Dense Reconstruction) Davide Scaramuzza

Lecture 10 Multi-view Stereo (3D Dense Reconstruction) Davide Scaramuzza Lecture 10 Multi-view Stereo (3D Dense Reconstruction) Davide Scaramuzza REMODE: Probabilistic, Monocular Dense Reconstruction in Real Time, ICRA 14, by Pizzoli, Forster, Scaramuzza [M. Pizzoli, C. Forster,

More information

3D model classification using convolutional neural network

3D model classification using convolutional neural network 3D model classification using convolutional neural network JunYoung Gwak Stanford jgwak@cs.stanford.edu Abstract Our goal is to classify 3D models directly using convolutional neural network. Most of existing

More information

Super-Resolution Keyframe Fusion for 3D Modeling with High-Quality Textures

Super-Resolution Keyframe Fusion for 3D Modeling with High-Quality Textures Super-Resolution Keyframe Fusion for 3D Modeling with High-Quality Textures Robert Maier, Jörg Stückler, Daniel Cremers International Conference on 3D Vision (3DV) October 2015, Lyon, France Motivation

More information

arxiv: v2 [cs.cv] 16 Nov 2016

arxiv: v2 [cs.cv] 16 Nov 2016 OctNet: Learning Deep 3D Representations at High Resolutions Gernot Riegler 1 Ali Osman Ulusoy 2 Andreas Geiger 2,3 1 Institute for Computer Graphics and Vision, Graz University of Technology 2 Autonomous

More information

Lecture 7: Semantic Segmentation

Lecture 7: Semantic Segmentation Semantic Segmentation CSED703R: Deep Learning for Visual Recognition (207F) Segmenting images based on its semantic notion Lecture 7: Semantic Segmentation Bohyung Han Computer Vision Lab. bhhanpostech.ac.kr

More information

PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space

PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space Sikai Zhong February 14, 2018 COMPUTER SCIENCE Table of contents 1. PointNet 2. PointNet++ 3. Experiments 1 PointNet Property

More information

MULTI-LEVEL 3D CONVOLUTIONAL NEURAL NETWORK FOR OBJECT RECOGNITION SAMBIT GHADAI XIAN LEE ADITYA BALU SOUMIK SARKAR ADARSH KRISHNAMURTHY

MULTI-LEVEL 3D CONVOLUTIONAL NEURAL NETWORK FOR OBJECT RECOGNITION SAMBIT GHADAI XIAN LEE ADITYA BALU SOUMIK SARKAR ADARSH KRISHNAMURTHY MULTI-LEVEL 3D CONVOLUTIONAL NEURAL NETWORK FOR OBJECT RECOGNITION SAMBIT GHADAI XIAN LEE ADITYA BALU SOUMIK SARKAR ADARSH KRISHNAMURTHY Outline Object Recognition Multi-Level Volumetric Representations

More information

Mesh from Depth Images Using GR 2 T

Mesh from Depth Images Using GR 2 T Mesh from Depth Images Using GR 2 T Mairead Grogan & Rozenn Dahyot School of Computer Science and Statistics Trinity College Dublin Dublin, Ireland mgrogan@tcd.ie, Rozenn.Dahyot@tcd.ie www.scss.tcd.ie/

More information

Unsupervised Learning of Spatiotemporally Coherent Metrics

Unsupervised Learning of Spatiotemporally Coherent Metrics Unsupervised Learning of Spatiotemporally Coherent Metrics Ross Goroshin, Joan Bruna, Jonathan Tompson, David Eigen, Yann LeCun arxiv 2015. Presented by Jackie Chu Contributions Insight between slow feature

More information

CAPNet: Continuous Approximation Projection For 3D Point Cloud Reconstruction Using 2D Supervision

CAPNet: Continuous Approximation Projection For 3D Point Cloud Reconstruction Using 2D Supervision CAPNet: Continuous Approximation Projection For 3D Point Cloud Reconstruction Using 2D Supervision Navaneet K L *, Priyanka Mandikal * and R. Venkatesh Babu Video Analytics Lab, IISc, Bangalore, India

More information

CAPNet: Continuous Approximation Projection For 3D Point Cloud Reconstruction Using 2D Supervision

CAPNet: Continuous Approximation Projection For 3D Point Cloud Reconstruction Using 2D Supervision CAPNet: Continuous Approximation Projection For 3D Point Cloud Reconstruction Using 2D Supervision Navaneet K L *, Priyanka Mandikal *, Mayank Agarwal, and R. Venkatesh Babu Video Analytics Lab, CDS, Indian

More information

Dense Tracking and Mapping for Autonomous Quadrocopters. Jürgen Sturm

Dense Tracking and Mapping for Autonomous Quadrocopters. Jürgen Sturm Computer Vision Group Prof. Daniel Cremers Dense Tracking and Mapping for Autonomous Quadrocopters Jürgen Sturm Joint work with Frank Steinbrücker, Jakob Engel, Christian Kerl, Erik Bylow, and Daniel Cremers

More information

Deep Marching Cubes: Learning Explicit Surface Representations

Deep Marching Cubes: Learning Explicit Surface Representations Deep Marching Cubes: Learning Explicit Surface Representations Yiyi Liao 1,2 Simon Donné 1,3 Andreas Geiger 1,4 1 Autonomous Vision Group, MPI for Intelligent Systems and University of Tübingen 2 Institute

More information

An Empirical Study of Generative Adversarial Networks for Computer Vision Tasks

An Empirical Study of Generative Adversarial Networks for Computer Vision Tasks An Empirical Study of Generative Adversarial Networks for Computer Vision Tasks Report for Undergraduate Project - CS396A Vinayak Tantia (Roll No: 14805) Guide: Prof Gaurav Sharma CSE, IIT Kanpur, India

More information

ECCV Presented by: Boris Ivanovic and Yolanda Wang CS 331B - November 16, 2016

ECCV Presented by: Boris Ivanovic and Yolanda Wang CS 331B - November 16, 2016 ECCV 2016 Presented by: Boris Ivanovic and Yolanda Wang CS 331B - November 16, 2016 Fundamental Question What is a good vector representation of an object? Something that can be easily predicted from 2D

More information

Using Faster-RCNN to Improve Shape Detection in LIDAR

Using Faster-RCNN to Improve Shape Detection in LIDAR Using Faster-RCNN to Improve Shape Detection in LIDAR TJ Melanson Stanford University Stanford, CA 94305 melanson@stanford.edu Abstract In this paper, I propose a method for extracting objects from unordered

More information

POINT CLOUD DEEP LEARNING

POINT CLOUD DEEP LEARNING POINT CLOUD DEEP LEARNING Innfarn Yoo, 3/29/28 / 57 Introduction AGENDA Previous Work Method Result Conclusion 2 / 57 INTRODUCTION 3 / 57 2D OBJECT CLASSIFICATION Deep Learning for 2D Object Classification

More information

Prof. Trevor Darrell Lecture 18: Multiview and Photometric Stereo

Prof. Trevor Darrell Lecture 18: Multiview and Photometric Stereo C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 18: Multiview and Photometric Stereo Today Multiview stereo revisited Shape from large image collections Voxel Coloring Digital

More information

Fast scene understanding and prediction for autonomous platforms. Bert De Brabandere, KU Leuven, October 2017

Fast scene understanding and prediction for autonomous platforms. Bert De Brabandere, KU Leuven, October 2017 Fast scene understanding and prediction for autonomous platforms Bert De Brabandere, KU Leuven, October 2017 Who am I? MSc in Electrical Engineering at KU Leuven, Belgium Last year PhD student with Luc

More information

Learning visual odometry with a convolutional network

Learning visual odometry with a convolutional network Learning visual odometry with a convolutional network Kishore Konda 1, Roland Memisevic 2 1 Goethe University Frankfurt 2 University of Montreal konda.kishorereddy@gmail.com, roland.memisevic@gmail.com

More information

Large-Scale Point Cloud Classification Benchmark

Large-Scale Point Cloud Classification Benchmark Large-Scale Point Cloud Classification Benchmark www.semantic3d.net IGP & CVG, ETH Zürich www.semantic3d.net, info@semantic3d.net 7/6/2016 1 Timo Hackel Nikolay Savinov Ľubor Ladický Jan Dirk Wegner Konrad

More information

Live Metric 3D Reconstruction on Mobile Phones ICCV 2013

Live Metric 3D Reconstruction on Mobile Phones ICCV 2013 Live Metric 3D Reconstruction on Mobile Phones ICCV 2013 Main Contents 1. Target & Related Work 2. Main Features of This System 3. System Overview & Workflow 4. Detail of This System 5. Experiments 6.

More information

Deep Supervision with Shape Concepts for Occlusion-Aware 3D Object Parsing

Deep Supervision with Shape Concepts for Occlusion-Aware 3D Object Parsing Deep Supervision with Shape Concepts for Occlusion-Aware 3D Object Parsing Supplementary Material Introduction In this supplementary material, Section 2 details the 3D annotation for CAD models and real

More information

Local Readjustment for High-Resolution 3D Reconstruction: Supplementary Material

Local Readjustment for High-Resolution 3D Reconstruction: Supplementary Material Local Readjustment for High-Resolution 3D Reconstruction: Supplementary Material Siyu Zhu 1, Tian Fang 2, Jianxiong Xiao 3, and Long Quan 4 1,2,4 The Hong Kong University of Science and Technology 3 Princeton

More information

OctNet: Learning Deep 3D Representations at High Resolutions

OctNet: Learning Deep 3D Representations at High Resolutions OctNet: Learning Deep 3D Representations at High Resolutions Gernot Riegler 1 Ali Osman Ulusoy 2 Andreas Geiger 2,3 1 Institute for Computer Graphics and Vision, Graz University of Technology 2 Autonomous

More information

3D Computer Vision. Depth Cameras. Prof. Didier Stricker. Oliver Wasenmüller

3D Computer Vision. Depth Cameras. Prof. Didier Stricker. Oliver Wasenmüller 3D Computer Vision Depth Cameras Prof. Didier Stricker Oliver Wasenmüller Kaiserlautern University http://ags.cs.uni-kl.de/ DFKI Deutsches Forschungszentrum für Künstliche Intelligenz http://av.dfki.de

More information

3D Reconstruction with Tango. Ivan Dryanovski, Google Inc.

3D Reconstruction with Tango. Ivan Dryanovski, Google Inc. 3D Reconstruction with Tango Ivan Dryanovski, Google Inc. Contents Problem statement and motivation The Tango SDK 3D reconstruction - data structures & algorithms Applications Developer tools Problem formulation

More information

RayNet: Learning Volumetric 3D Reconstruction with Ray Potentials

RayNet: Learning Volumetric 3D Reconstruction with Ray Potentials RayNet: Learning Volumetric 3D Reconstruction with Ray Potentials Despoina Paschalidou 1,5 Ali Osman Ulusoy 2 Carolin Schmitt 1 Luc van Gool 3 Andreas Geiger 1,4 1 Autonomous Vision Group, MPI for Intelligent

More information

Point2Sequence: Learning the Shape Representation of 3D Point Clouds with an Attention-based Sequence to Sequence Network

Point2Sequence: Learning the Shape Representation of 3D Point Clouds with an Attention-based Sequence to Sequence Network Point2Sequence: Learning the Shape Representation of 3D Point Clouds with an Attention-based Sequence to Sequence Network Xinhai Liu, Zhizhong Han,2, Yu-Shen Liu, Matthias Zwicker 2 School of Software,

More information

Multi-view Stereo. Ivo Boyadzhiev CS7670: September 13, 2011

Multi-view Stereo. Ivo Boyadzhiev CS7670: September 13, 2011 Multi-view Stereo Ivo Boyadzhiev CS7670: September 13, 2011 What is stereo vision? Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape

More information

Supplementary: Cross-modal Deep Variational Hand Pose Estimation

Supplementary: Cross-modal Deep Variational Hand Pose Estimation Supplementary: Cross-modal Deep Variational Hand Pose Estimation Adrian Spurr, Jie Song, Seonwook Park, Otmar Hilliges ETH Zurich {spurra,jsong,spark,otmarh}@inf.ethz.ch Encoder/Decoder Linear(512) Table

More information

Deep learning for dense per-pixel prediction. Chunhua Shen The University of Adelaide, Australia

Deep learning for dense per-pixel prediction. Chunhua Shen The University of Adelaide, Australia Deep learning for dense per-pixel prediction Chunhua Shen The University of Adelaide, Australia Image understanding Classification error Convolution Neural Networks 0.3 0.2 0.1 Image Classification [Krizhevsky

More information

3D Object Representations. COS 526, Fall 2016 Princeton University

3D Object Representations. COS 526, Fall 2016 Princeton University 3D Object Representations COS 526, Fall 2016 Princeton University 3D Object Representations How do we... Represent 3D objects in a computer? Acquire computer representations of 3D objects? Manipulate computer

More information

Deep Supervision with Shape Concepts for Occlusion-Aware 3D Object Parsing Supplementary Material

Deep Supervision with Shape Concepts for Occlusion-Aware 3D Object Parsing Supplementary Material Deep Supervision with Shape Concepts for Occlusion-Aware 3D Object Parsing Supplementary Material Chi Li, M. Zeeshan Zia 2, Quoc-Huy Tran 2, Xiang Yu 2, Gregory D. Hager, and Manmohan Chandraker 2 Johns

More information

arxiv: v1 [cs.cv] 28 Mar 2018

arxiv: v1 [cs.cv] 28 Mar 2018 arxiv:1803.10409v1 [cs.cv] 28 Mar 2018 3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation Angela Dai 1 Matthias Nießner 2 1 Stanford University 2 Technical University of Munich Fig.

More information

RTSR: Enhancing Real-time H.264 Video Streaming using Deep Learning based Video Super Resolution Spring 2017 CS570 Project Presentation June 8, 2017

RTSR: Enhancing Real-time H.264 Video Streaming using Deep Learning based Video Super Resolution Spring 2017 CS570 Project Presentation June 8, 2017 RTSR: Enhancing Real-time H.264 Video Streaming using Deep Learning based Video Super Resolution Spring 2017 CS570 Project Presentation June 8, 2017 Team 16 Soomin Kim Leslie Tiong Youngki Kwon Insu Jang

More information

Supplementary Material for Sparsity Invariant CNNs

Supplementary Material for Sparsity Invariant CNNs Supplementary Material for Sparsity Invariant CNNs Jonas Uhrig,1,2 Nick Schneider,1,3 Lukas Schneider 1,4 Uwe Franke 1 Thomas Brox 2 Andreas Geiger 4,5 1 Daimler R&D Sindelfingen 2 University of Freiburg

More information

COMP 551 Applied Machine Learning Lecture 16: Deep Learning

COMP 551 Applied Machine Learning Lecture 16: Deep Learning COMP 551 Applied Machine Learning Lecture 16: Deep Learning Instructor: Ryan Lowe (ryan.lowe@cs.mcgill.ca) Slides mostly by: Class web page: www.cs.mcgill.ca/~hvanho2/comp551 Unless otherwise noted, all

More information

arxiv: v1 [cs.cv] 31 Mar 2018

arxiv: v1 [cs.cv] 31 Mar 2018 arxiv:1804.00090v1 [cs.cv] 31 Mar 2018 FloorNet: A Unified Framework for Floorplan Reconstruction from 3D Scans Chen Liu Jiaye Wu Washington University in St. Louis {chenliu,jiaye.wu}@wustl.edu Yasutaka

More information

Learning-based Localization

Learning-based Localization Learning-based Localization Eric Brachmann ECCV 2018 Tutorial on Visual Localization - Feature-based vs. Learned Approaches Torsten Sattler, Eric Brachmann Roadmap Machine Learning Basics [10min] Convolutional

More information

FloorNet: A Unified Framework for Floorplan Reconstruction from 3D Scans

FloorNet: A Unified Framework for Floorplan Reconstruction from 3D Scans FloorNet: A Unified Framework for Floorplan Reconstruction from 3D Scans Chen Liu 1, Jiaye Wu 1, and Yasutaka Furukawa 2 1 Washington University in St. Louis, St. Louis, USA {chenliu,jiaye.wu}@wustl.edu

More information

arxiv: v2 [cs.cv] 27 Nov 2018

arxiv: v2 [cs.cv] 27 Nov 2018 Learning 3D Shape Completion under Weak Supervision David Stutz Andreas Geiger arxiv:185.729v2 [cs.cv] 27 Nov 218 Abstract We address the problem of 3D shape completion from sparse and noisy point clouds,

More information

arxiv: v1 [cs.cv] 18 May 2018 Received: date / Accepted: date

arxiv: v1 [cs.cv] 18 May 2018 Received: date / Accepted: date Noname manuscript No. (will be inserted by the editor) Learning 3D Shape Completion under Weak Supervision David Stutz Andreas Geiger arxiv:185.729v1 [cs.cv] 18 May 218 Received: date / Accepted: date

More information

From Point Clouds to Mesh using Regression

From Point Clouds to Mesh using Regression From Point Clouds to Mesh using Regression L ubor Ladický ETH Zürich lubor.ladicky@inf.ethz.ch Fabio Maninchedda ETH Zürich fabiom@inf.ethz.ch Olivier Saurer ETH Zürich saurero@inf.ethz.ch Marc Pollefeys

More information

Hierarchical Surface Prediction for 3D Object Reconstruction

Hierarchical Surface Prediction for 3D Object Reconstruction Hierarchical Surface Prediction for 3D Object Reconstruction Christian Häne, Shubham Tulsiani, Jitendra Malik University of California, Berkeley {chaene, shubhtuls, malik}@eecs.berkeley.edu Abstract Recently,

More information

Stereo Human Keypoint Estimation

Stereo Human Keypoint Estimation Stereo Human Keypoint Estimation Kyle Brown Stanford University Stanford Intelligent Systems Laboratory kjbrown7@stanford.edu Abstract The goal of this project is to accurately estimate human keypoint

More information

Scanning and Printing Objects in 3D Jürgen Sturm

Scanning and Printing Objects in 3D Jürgen Sturm Scanning and Printing Objects in 3D Jürgen Sturm Metaio (formerly Technical University of Munich) My Research Areas Visual navigation for mobile robots RoboCup Kinematic Learning Articulated Objects Quadrocopters

More information

Geometric Registration for Deformable Shapes 1.1 Introduction

Geometric Registration for Deformable Shapes 1.1 Introduction Geometric Registration for Deformable Shapes 1.1 Introduction Overview Data Sources and Applications Problem Statement Overview Presenters Will Chang University of California at San Diego, USA Hao Li ETH

More information

A Patch Prior for Dense 3D Reconstruction in Man-Made Environments

A Patch Prior for Dense 3D Reconstruction in Man-Made Environments A Patch Prior for Dense 3D Reconstruction in Man-Made Environments Christian Häne 1, Christopher Zach 2, Bernhard Zeisl 1, Marc Pollefeys 1 1 ETH Zürich 2 MSR Cambridge October 14, 2012 A Patch Prior for

More information

Multi-view 3D Models from Single Images with a Convolutional Network

Multi-view 3D Models from Single Images with a Convolutional Network Multi-view 3D Models from Single Images with a Convolutional Network Maxim Tatarchenko University of Freiburg Skoltech - 2nd Christmas Colloquium on Computer Vision Humans have prior knowledge about 3D

More information

VISION FOR AUTOMOTIVE DRIVING

VISION FOR AUTOMOTIVE DRIVING VISION FOR AUTOMOTIVE DRIVING French Japanese Workshop on Deep Learning & AI, Paris, October 25th, 2017 Quoc Cuong PHAM, PhD Vision and Content Engineering Lab AI & MACHINE LEARNING FOR ADAS AND SELF-DRIVING

More information

Discrete Optimization of Ray Potentials for Semantic 3D Reconstruction

Discrete Optimization of Ray Potentials for Semantic 3D Reconstruction Discrete Optimization of Ray Potentials for Semantic 3D Reconstruction Marc Pollefeys Joined work with Nikolay Savinov, Christian Haene, Lubor Ladicky 2 Comparison to Volumetric Fusion Higher-order ray

More information

3D Semantic Segmentation with Submanifold Sparse Convolutional Networks

3D Semantic Segmentation with Submanifold Sparse Convolutional Networks 3D Semantic Segmentation with Submanifold Sparse Convolutional Networks Benjamin Graham Facebook AI Research benjamingraham@fb.com Martin Engelcke University of Oxford martin@robots.ox.ac.uk Laurens van

More information

Learning to Reconstruct High-quality 3D Shapes with Cascaded Fully Convolutional Networks

Learning to Reconstruct High-quality 3D Shapes with Cascaded Fully Convolutional Networks Learning to Reconstruct High-quality 3D Shapes with Cascaded Fully Convolutional Networks * Yan-Pei Cao 1,2, * Zheng-Ning Liu 1, Zheng-Fei Kuang 1, Leif Kobbelt 3, Shi-Min Hu 1 * equal contribution 1 Tsinghua

More information

Computing the Stereo Matching Cost with CNN

Computing the Stereo Matching Cost with CNN University at Austin Figure. The of lefttexas column displays the left input image, while the right column displays the output of our stereo method. Examples are sorted by difficulty, with easy examples

More information

FLaME: Fast Lightweight Mesh Estimation using Variational Smoothing on Delaunay Graphs

FLaME: Fast Lightweight Mesh Estimation using Variational Smoothing on Delaunay Graphs FLaME: Fast Lightweight Mesh Estimation using Variational Smoothing on Delaunay Graphs W. Nicholas Greene Robust Robotics Group, MIT CSAIL LPM Workshop IROS 2017 September 28, 2017 with Nicholas Roy 1

More information

Object Detection. CS698N Final Project Presentation AKSHAT AGARWAL SIDDHARTH TANWAR

Object Detection. CS698N Final Project Presentation AKSHAT AGARWAL SIDDHARTH TANWAR Object Detection CS698N Final Project Presentation AKSHAT AGARWAL SIDDHARTH TANWAR Problem Description Arguably the most important part of perception Long term goals for object recognition: Generalization

More information

From Orientation to Functional Modeling for Terrestrial and UAV Images

From Orientation to Functional Modeling for Terrestrial and UAV Images From Orientation to Functional Modeling for Terrestrial and UAV Images Helmut Mayer 1 Andreas Kuhn 1, Mario Michelini 1, William Nguatem 1, Martin Drauschke 2 and Heiko Hirschmüller 2 1 Visual Computing,

More information

CS5670: Computer Vision

CS5670: Computer Vision CS5670: Computer Vision Noah Snavely Multi-view stereo Announcements Project 3 ( Autostitch ) due Monday 4/17 by 11:59pm Recommended Reading Szeliski Chapter 11.6 Multi-View Stereo: A Tutorial Furukawa

More information

arxiv: v2 [cs.cv] 28 Mar 2018

arxiv: v2 [cs.cv] 28 Mar 2018 ScanComplete: Large-Scale Scene Completion and Semantic Segmentation for 3D Scans Angela Dai 1,3,5 Daniel Ritchie 2 Martin Bokeloh 3 Scott Reed 4 Jürgen Sturm 3 Matthias Nießner 5 1 Stanford University

More information

Encoder-Decoder Networks for Semantic Segmentation. Sachin Mehta

Encoder-Decoder Networks for Semantic Segmentation. Sachin Mehta Encoder-Decoder Networks for Semantic Segmentation Sachin Mehta Outline > Overview of Semantic Segmentation > Encoder-Decoder Networks > Results What is Semantic Segmentation? Input: RGB Image Output:

More information

ScanComplete: Large-Scale Scene Completion and Semantic Segmentation for 3D Scans

ScanComplete: Large-Scale Scene Completion and Semantic Segmentation for 3D Scans ScanComplete: Large-Scale Scene Completion and Semantic Segmentation for 3D Scans Angela Dai 1,3,5 Daniel Ritchie 2 Martin Bokeloh 3 Scott Reed 4 Jürgen Sturm 3 Matthias Nießner 5 1 Stanford University

More information

Point Cloud Completion of Foot Shape from a Single Depth Map for Fit Matching using Deep Learning View Synthesis

Point Cloud Completion of Foot Shape from a Single Depth Map for Fit Matching using Deep Learning View Synthesis Point Cloud Completion of Foot Shape from a Single Depth Map for Fit Matching using Deep Learning View Synthesis Nolan Lunscher University of Waterloo 200 University Ave W. nlunscher@uwaterloo.ca John

More information

DeepIM: Deep Iterative Matching for 6D Pose Estimation - Supplementary Material

DeepIM: Deep Iterative Matching for 6D Pose Estimation - Supplementary Material DeepIM: Deep Iterative Matching for 6D Pose Estimation - Supplementary Material Yi Li 1, Gu Wang 1, Xiangyang Ji 1, Yu Xiang 2, and Dieter Fox 2 1 Tsinghua University, BNRist 2 University of Washington

More information

Flow-Based Video Recognition

Flow-Based Video Recognition Flow-Based Video Recognition Jifeng Dai Visual Computing Group, Microsoft Research Asia Joint work with Xizhou Zhu*, Yuwen Xiong*, Yujie Wang*, Lu Yuan and Yichen Wei (* interns) Talk pipeline Introduction

More information

arxiv: v2 [cs.cv] 6 Nov 2017

arxiv: v2 [cs.cv] 6 Nov 2017 Hierarchical Surface Prediction for 3D Object Reconstruction Christian Häne, Shubham Tulsiani, Jitendra Malik University of California, Berkeley {chaene, shubhtuls, malik}@eecs.berkeley.edu arxiv:1704.00710v2

More information

Computing 3D Geometry Directly From Range Images

Computing 3D Geometry Directly From Range Images Computing 3D Geometry Directly From Range Images Sarah F. Frisken and Ronald N. Perry Mitsubishi Electric Research Laboratories Geometry from Range Data A Classic Approach Subject Range images Range surfaces

More information

S7348: Deep Learning in Ford's Autonomous Vehicles. Bryan Goodman Argo AI 9 May 2017

S7348: Deep Learning in Ford's Autonomous Vehicles. Bryan Goodman Argo AI 9 May 2017 S7348: Deep Learning in Ford's Autonomous Vehicles Bryan Goodman Argo AI 9 May 2017 1 Ford s 12 Year History in Autonomous Driving Today: examples from Stereo image processing Object detection Using RNN

More information

Deep Learning in Image Processing

Deep Learning in Image Processing Deep Learning in Image Processing Roland Memisevic University of Montreal & TwentyBN ICISP 2016 Roland Memisevic Deep Learning in Image Processing ICISP 2016 f 2? cathedral high-rise f 1 It s the features,

More information