Super-Resolution Keyframe Fusion for 3D Modeling with High-Quality Textures

Similar documents
Intrinsic3D: High-Quality 3D Reconstruction by Joint Appearance and Geometry Optimization with Spatially-Varying Lighting

Efficient Online Surface Correction for Real-time Large-Scale 3D Reconstruction

CVPR 2014 Visual SLAM Tutorial Kintinuous

Scanning and Printing Objects in 3D Jürgen Sturm

Dense 3D Reconstruction from Autonomous Quadrocopters

Dense Tracking and Mapping for Autonomous Quadrocopters. Jürgen Sturm

Semi-Dense Direct SLAM

Scanning and Printing Objects in 3D

Deep Incremental Scene Understanding. Federico Tombari & Christian Rupprecht Technical University of Munich, Germany

Direct Methods in Visual Odometry

Project Updates Short lecture Volumetric Modeling +2 papers

Jakob Engel, Thomas Schöps, Daniel Cremers Technical University Munich. LSD-SLAM: Large-Scale Direct Monocular SLAM

Colored Point Cloud Registration Revisited Supplementary Material

RGBD Point Cloud Alignment Using Lucas-Kanade Data Association and Automatic Error Metric Selection

Large-Scale Multi-Resolution Surface Reconstruction from RGB-D Sequences

Large-Scale Multi-Resolution Surface Reconstruction from RGB-D Sequences

Robust Online 3D Reconstruction Combining a Depth Sensor and Sparse Feature Points

Geometric Reconstruction Dense reconstruction of scene geometry

Mobile Point Fusion. Real-time 3d surface reconstruction out of depth images on a mobile platform

FLaME: Fast Lightweight Mesh Estimation using Variational Smoothing on Delaunay Graphs

A Real-Time RGB-D Registration and Mapping Approach by Heuristically Switching Between Photometric And Geometric Information

Simultaneous Localization and Calibration: Self-Calibration of Consumer Depth Cameras

CuFusion2: Accurate and Denoised Volumetric 3D Object Reconstruction Using Depth Cameras

OpenStreetSLAM: Global Vehicle Localization using OpenStreetMaps

DNA-SLAM: Dense Noise Aware SLAM for ToF RGB-D Cameras

arxiv: v1 [cs.ro] 23 Nov 2015

Deep Models for 3D Reconstruction

Computer Vision 2 Lecture 1

Real-Time Vision-Based State Estimation and (Dense) Mapping

A Benchmark for RGB-D Visual Odometry, 3D Reconstruction and SLAM

3D Object Recognition and Scene Understanding from RGB-D Videos. Yu Xiang Postdoctoral Researcher University of Washington

Tracking an RGB-D Camera Using Points and Planes

Oblique Image Processing in SURE - First Experiments and Results

3D Computer Vision. Depth Cameras. Prof. Didier Stricker. Oliver Wasenmüller

3D Object Representations. COS 526, Fall 2016 Princeton University

Photorealistic 3D Mapping of Indoors by RGB-D Scanning Process

Simultaneously Estimation of Super-Resolution Images and Depth Maps from Low Resolution Sensors

Multi-Volume High Resolution RGB-D Mapping with Dynamic Volume Placement. Michael Salvato

ElasticFusion: Dense SLAM Without A Pose Graph

Intrinsic3D: High-Quality 3D Reconstruction by Joint Appearance and Geometry Optimization with Spatially-Varying Lighting

MonoRGBD-SLAM: Simultaneous Localization and Mapping Using Both Monocular and RGBD Cameras

CopyMe3D: Scanning and Printing Persons in 3D

Introduction to Mobile Robotics Techniques for 3D Mapping

Approximate Surface Reconstruction and Registration for RGB-D SLAM

3D Reconstruction with Tango. Ivan Dryanovski, Google Inc.

Integrating Depth and Color Cues for Dense Multi-Resolution Scene Mapping Using RGB-D Cameras

Hybrids Mixed Approaches

CuFusion: Accurate real-time camera tracking and volumetric scene reconstruction with a cuboid

Reconstructing Reflective and Transparent Surfaces from Epipolar Plane Images

Optimized KinectFusion Algorithm for 3D Scanning Applications

Perceiving the 3D World from Images and Videos. Yu Xiang Postdoctoral Researcher University of Washington

arxiv: v1 [cs.cv] 1 Aug 2017

Reconstructing Articulated Rigged Models from RGB-D Videos

Outline. 1 Why we re interested in Real-Time tracking and mapping. 3 Kinect Fusion System Overview. 4 Real-time Surface Mapping

3D Colored Model Generation Based on Multiview Textures and Triangular Mesh

Stereo and Kinect fusion for continuous. 3D reconstruction and visual odometry

Combining Depth Fusion and Photometric Stereo for Fine-Detailed 3D Models

Joint Multiview Segmentation and Localization of RGB-D Images using Depth-Induced Silhouette Consistency

arxiv: v2 [cs.cv] 26 Aug 2018

LEARNING RIGIDITY IN DYNAMIC SCENES FOR SCENE FLOW ESTIMATION

Signed Distance Function Representation, Tracking, and Mapping. Tanner Schmidt

Robustness to Lighting Variations: An RGB-D Indoor Visual Odometry Using Line Segments

ROBUST LOOP CLOSURES FOR SCENE RECONSTRUCTION BY COMBINING ODOMETRY AND VISUAL CORRESPONDENCES

Deformation-based Loop Closure for Large Scale Dense RGB-D SLAM

Lecture 10 Multi-view Stereo (3D Dense Reconstruction) Davide Scaramuzza

Volumetric 3D Mapping in Real-Time on a CPU

KinectFusion: Real-Time Dense Surface Mapping and Tracking

Kintinuous: Spatially Extended KinectFusion Thomas Whelan, Michael Kaess, Maurice Fallon, Hordur Johannsson, John Leonard, and John McDonald

Multiframe Scene Flow with Piecewise Rigid Motion. Vladislav Golyanik,, Kihwan Kim, Robert Maier, Mathias Nießner, Didier Stricker and Jan Kautz

Feature-based RGB-D camera pose optimization for real-time 3D reconstruction

Robust Real-Time Visual Odometry for Dense RGB-D Mapping

o Basic signal processing o Filtering, resampling, warping,... Rendering o Polygon rendering pipeline o Ray tracing Modeling

Vol agile avec des micro-robots volants contrôlés par vision

Lecture 10 Dense 3D Reconstruction

3D Line Segments Extraction from Semi-dense SLAM

3D Imaging from Video and Planar Radiography

Patch-Based Optimization for Image-Based Texture Mapping

Photorealistic Texturing of Human Busts Reconstructions

Local-Level 3D Deep Learning. Andy Zeng

Visual-Inertial Localization and Mapping for Robot Navigation

Autonomous Navigation for Flying Robots

3DLite: Towards Commodity 3D Scanning for Content Creation

PL-SVO: Semi-Direct Monocular Visual Odometry by Combining Points and Line Segments

WHITE PAPER Scale-Awareness of Light Field Camera Based Visual Odometry

Nonrigid Surface Modelling. and Fast Recovery. Department of Computer Science and Engineering. Committee: Prof. Leo J. Jia and Prof. K. H.

Virtualized Reality Using Depth Camera Point Clouds

Visual Perception for Robots

Application questions. Theoretical questions

Live Metric 3D Reconstruction on Mobile Phones ICCV 2013

arxiv: v1 [cs.ro] 24 Nov 2018

arxiv: v2 [cs.cv] 9 Sep 2016

Semi-Direct EKF-based Monocular Visual-Inertial Odometry

Visual Navigation for Flying Robots Exploration, Multi-Robot Coordination and Coverage

Application of LSD-SLAM for Visualization Temperature in Wide-area Environment

L10 Layered Depth Normal Images. Introduction Related Work Structured Point Representation Boolean Operations Conclusion

RGB-D Object Modelling for Object Recognition and Tracking

3D Scene Reconstruction with a Mobile Camera

Ping Tan. Simon Fraser University

Fast Odometry and Scene Flow from RGB-D Cameras based on Geometric Clustering

RGB-D Edge Detection and Edge-based Registration

Transcription:

Super-Resolution Keyframe Fusion for 3D Modeling with High-Quality Textures Robert Maier, Jörg Stückler, Daniel Cremers International Conference on 3D Vision (3DV) October 2015, Lyon, France

Motivation Given: Computer Vision Group Low-resolution RGB-D frames (640 x 480) Accurate geometric reconstruction R. Maier, J. Stückler, D. Cremers: Super-Resolution Keyframe Fusion for 3D Modeling with High-Quality Textures 2

Motivation Given: Computer Vision Group Low-resolution RGB-D frames (640 x 480) Accurate geometric State-of-the-art w.r.t. visual appearance: reconstruction Vertex colors: limited resolution Texture mapping: good-quality results, but slow/impractical R. Maier, J. Stückler, D. Cremers: Super-Resolution Keyframe Fusion for 3D Modeling with High-Quality Textures 2

Motivation Given: Computer Vision Group Low-resolution RGB-D frames (640 x 480) Accurate geometric State-of-the-art w.r.t. visual appearance: reconstruction Vertex colors: limited resolution Texture mapping: good-quality results, but slow/impractical Problem: Gap in research of fast and robust estimation of highquality visual appearance from low-cost RGB-D sensors R. Maier, J. Stückler, D. Cremers: Super-Resolution Keyframe Fusion for 3D Modeling with High-Quality Textures 2

Texture Mapping with Super-Resolution Keyframes Our Approach: R. Maier, J. Stückler, D. Cremers: Super-Resolution Keyframe Fusion for 3D Modeling with High-Quality Textures 3

Texture Mapping with Super-Resolution Keyframes Our Approach: Super-resolution (SR) Keyframe Fusion and Deblurring R. Maier, J. Stückler, D. Cremers: Super-Resolution Keyframe Fusion for 3D Modeling with High-Quality Textures 3

Texture Mapping with Super-Resolution Keyframes Our Approach: Super-resolution (SR) Keyframe Fusion and Deblurring Texture Mapping using SR keyframes (weighted median) R. Maier, J. Stückler, D. Cremers: Super-Resolution Keyframe Fusion for 3D Modeling with High-Quality Textures 3

Texture Mapping with Super-Resolution Keyframes Our Approach: Super-resolution (SR) Keyframe Fusion and Deblurring Texture Mapping using SR keyframes (weighted median) Vertex colors Our approach R. Maier, J. Stückler, D. Cremers: Super-Resolution Keyframe Fusion for 3D Modeling with High-Quality Textures 3

Related Work R. Maier, J. Stückler, D. Cremers: Super-Resolution Keyframe Fusion for 3D Modeling with High-Quality Textures 4

Related Work Vertex colors (weighted average) Newcombe et al., KinectFusion: Real-time dense surface mapping and tracking. ISMAR 2011 Sturm et al., CopyMe3D: Scanning and printing persons in 3D. GCPR 2013 R. Maier, J. Stückler, D. Cremers: Super-Resolution Keyframe Fusion for 3D Modeling with High-Quality Textures 4

Related Work Vertex colors (weighted average) Newcombe et al., KinectFusion: Real-time dense surface mapping and tracking. ISMAR 2011 Sturm et al., CopyMe3D: Scanning and printing persons in 3D. GCPR 2013 Texture Mapping Weighted Median [Coorg and Teller. Automatic extraction of textured vertical facades from pose imagery. 1998] Single input view per face, minimize seams [Gal et al., Seamless montage for texturing models. 2010] Variational super-resolution [Goldlücke et al., A super-resolution framework for high-accuracy multiview reconstruction, IJCV 2014] R. Maier, J. Stückler, D. Cremers: Super-Resolution Keyframe Fusion for 3D Modeling with High-Quality Textures 4

Related Work Vertex colors (weighted average) Newcombe et al., KinectFusion: Real-time dense surface mapping and tracking. ISMAR 2011 Sturm et al., CopyMe3D: Scanning and printing persons in 3D. GCPR 2013 Texture Mapping Weighted Median [Coorg and Teller. Automatic extraction of textured vertical facades from pose imagery. 1998] Single input view per face, minimize seams [Gal et al., Seamless montage for texturing models. 2010] Variational super-resolution [Goldlücke et al., A super-resolution framework for high-accuracy multiview reconstruction, IJCV 2014] RGB-D based 3D Reconstruction Super-resolution RGB-D keyframes [Meilland and Comport. Super-resolution 3D tracking and mapping. ICRA 2013] Optimization of camera poses and non-rigid correction [Zhou and Koltun. Color map optimization for 3D reconstruction with consumer depth cameras. TOG 2014] R. Maier, J. Stückler, D. Cremers: Super-Resolution Keyframe Fusion for 3D Modeling with High-Quality Textures 4

Related Work Vertex colors (weighted average) Newcombe et al., KinectFusion: Real-time dense surface mapping and tracking. ISMAR 2011 Sturm et al., CopyMe3D: Scanning and printing persons in 3D. GCPR 2013 Texture Mapping Weighted Median [Coorg and Teller. Automatic extraction of textured vertical facades from pose imagery. 1998] Single input view per face, minimize seams [Gal et al., Seamless montage for texturing models. 2010] Variational super-resolution [Goldlücke et al., A super-resolution framework for high-accuracy multiview reconstruction, IJCV 2014] RGB-D based 3D Reconstruction Super-resolution RGB-D keyframes [Meilland and Comport. Super-resolution 3D tracking and mapping. ICRA 2013] Optimization of camera poses and non-rigid correction [Zhou and Koltun. Color map optimization for 3D reconstruction with consumer depth cameras. TOG 2014] Contribution: Texture Mapping using Super-resolution Keyframes R. Maier, J. Stückler, D. Cremers: Super-Resolution Keyframe Fusion for 3D Modeling with High-Quality Textures 4

System Overview R. Maier, J. Stückler, D. Cremers: Super-Resolution Keyframe Fusion for 3D Modeling with High-Quality Textures 5

System Overview Low-resolution RGB-D frames R. Maier, J. Stückler, D. Cremers: Super-Resolution Keyframe Fusion for 3D Modeling with High-Quality Textures 5

System Overview Low-resolution RGB-D frames DVO-SLAM TSDF Volume Integration 3D Model R. Maier, J. Stückler, D. Cremers: Super-Resolution Keyframe Fusion for 3D Modeling with High-Quality Textures 5

System Overview Low-resolution RGB-D frames DVO-SLAM TSDF Volume Integration 3D Model Keyframe Fusion Super-resolution Keyframes R. Maier, J. Stückler, D. Cremers: Super-Resolution Keyframe Fusion for 3D Modeling with High-Quality Textures 5

System Overview Low-resolution RGB-D frames DVO-SLAM TSDF Volume Integration 3D Model Parametrization Texture Keyframe Fusion Super-resolution Keyframes Texel Color Computation R. Maier, J. Stückler, D. Cremers: Super-Resolution Keyframe Fusion for 3D Modeling with High-Quality Textures 5

Geometric 3D Reconstruction R. Maier, J. Stückler, D. Cremers: Super-Resolution Keyframe Fusion for 3D Modeling with High-Quality Textures 6

Geometric 3D Reconstruction DVO-SLAM: camera trajectory [Kerl et al., Dense visual slam for RGB-D cameras. IROS 2013] Real-time 3D reconstruction on a CPU Robust Dense Visual Odometry Loop closure detection + pose graph optimization R. Maier, J. Stückler, D. Cremers: Super-Resolution Keyframe Fusion for 3D Modeling with High-Quality Textures 6

Geometric 3D Reconstruction DVO-SLAM: camera trajectory [Kerl et al., Dense visual slam for RGB-D cameras. IROS 2013] Real-time 3D reconstruction on a CPU Robust Dense Visual Odometry Loop closure detection + pose graph optimization Model Fusion (TSDF Volume): 3D mesh R. Maier, J. Stückler, D. Cremers: Super-Resolution Keyframe Fusion for 3D Modeling with High-Quality Textures 6

Vertex Color Computation Mesh vertex v, input views C i (blur measure b i ), camera poses R. Maier, J. Stückler, D. Cremers: Super-Resolution Keyframe Fusion for 3D Modeling with High-Quality Textures 7

Vertex Color Computation Mesh vertex v, input views C i (blur measure b i ), camera poses Compute observations (x i is obs. of v in view i): x 0 x 1 R. Maier, J. Stückler, D. Cremers: Super-Resolution Keyframe Fusion for 3D Modeling with High-Quality Textures 7

Vertex Color Computation Mesh vertex v, input views C i (blur measure b i ), camera poses Compute observations (x i is obs. of v in view i): Discard x i close to depth discontinuities x 0 x 1 R. Maier, J. Stückler, D. Cremers: Super-Resolution Keyframe Fusion for 3D Modeling with High-Quality Textures 7

Vertex Color Computation Mesh vertex v, input views C i (blur measure b i ), camera poses Compute observations (x i is obs. of v in view i): Discard x i close to depth discontinuities Weights of obs. x i : x 0 x 1 R. Maier, J. Stückler, D. Cremers: Super-Resolution Keyframe Fusion for 3D Modeling with High-Quality Textures 7

Vertex Color Computation Mesh vertex v, input views C i (blur measure b i ), camera poses Compute observations (x i is obs. of v in view i): Discard x i close to depth discontinuities Weights of obs. x i : x 0 Compute vertex color x: Weighted mean: x 1 Weighted median: R. Maier, J. Stückler, D. Cremers: Super-Resolution Keyframe Fusion for 3D Modeling with High-Quality Textures 7

Vertex Color Computation Unweighted Mean Weighted Mean Weighted Median R. Maier, J. Stückler, D. Cremers: Super-Resolution Keyframe Fusion for 3D Modeling with High-Quality Textures 8

Texture Mapping using SR Keyframes Per-vertex colors: limited resolution Increase resolution: Texture Approach: SR Keyframe Fusion Texture Mapping from SR keyframes R. Maier, J. Stückler, D. Cremers: Super-Resolution Keyframe Fusion for 3D Modeling with High-Quality Textures 9

Keyframe Fusion Idea: fuse low-resolution (LR) input RGB-D frames into high resolution RGB-D keyframes R. Maier, J. Stückler, D. Cremers: Super-Resolution Keyframe Fusion for 3D Modeling with High-Quality Textures 10

Keyframe Fusion Idea: fuse low-resolution (LR) input RGB-D frames into high resolution RGB-D keyframes Depth fusion Warp LR depth maps into keyframe (using relative poses) Upsample and fuse depth using weighted averaging R. Maier, J. Stückler, D. Cremers: Super-Resolution Keyframe Fusion for 3D Modeling with High-Quality Textures 10

Keyframe Fusion Idea: fuse low-resolution (LR) input RGB-D frames into high resolution RGB-D keyframes Depth fusion Warp LR depth maps into keyframe (using relative poses) Upsample and fuse depth using weighted averaging Color fusion Deconvolution: Wiener Filter on LR input images Warp fused keyframe depth to input images for color lookup Fuse colors using weighted median R. Maier, J. Stückler, D. Cremers: Super-Resolution Keyframe Fusion for 3D Modeling with High-Quality Textures 10

Keyframe Fusion LR input image Fused SR keyframe R. Maier, J. Stückler, D. Cremers: Super-Resolution Keyframe Fusion for 3D Modeling with High-Quality Textures 11

Texture Mapping using SR Keyframes R. Maier, J. Stückler, D. Cremers: Super-Resolution Keyframe Fusion for 3D Modeling with High-Quality Textures 12

Texture Mapping using SR Keyframes Texture Parametrization: One-to-one mapping between 3D mesh and 2D texture R. Maier, J. Stückler, D. Cremers: Super-Resolution Keyframe Fusion for 3D Modeling with High-Quality Textures 12

Texture Mapping using SR Keyframes Texture Parametrization: One-to-one mapping between 3D mesh and 2D texture Texel color computation: Compute 3D vertex for 2D texel (based on enclosing triangle using barycentric coordinates) Compute color from SR keyframes analogous to per-vertex recoloring scheme (weighted median) R. Maier, J. Stückler, D. Cremers: Super-Resolution Keyframe Fusion for 3D Modeling with High-Quality Textures 12

Qualitative Evaluation: Deconvolution Without deconvolution With deconvolution R. Maier, J. Stückler, D. Cremers: Super-Resolution Keyframe Fusion for 3D Modeling with High-Quality Textures 13

Qualitative Evaluation: SR Keyframe resolution Keyframe dimensions 1280 x 960 (scale s = 2) Keyframe dimensions 2560 x 1920 (scale s = 4) R. Maier, J. Stückler, D. Cremers: Super-Resolution Keyframe Fusion for 3D Modeling with High-Quality Textures 14

Qualitative Evaluation: LR input frames vs. SR keyframes With LR input frames With SR keyframes R. Maier, J. Stückler, D. Cremers: Super-Resolution Keyframe Fusion for 3D Modeling with High-Quality Textures 15

Runtime Evaluation Datasets: R. Maier, J. Stückler, D. Cremers: Super-Resolution Keyframe Fusion for 3D Modeling with High-Quality Textures 16

Runtime Evaluation Datasets: Runtimes: (Standard desktop PC with Intel Core i7-2600 CPU with 3.40GHz and 8GB RAM) R. Maier, J. Stückler, D. Cremers: Super-Resolution Keyframe Fusion for 3D Modeling with High-Quality Textures 16

Phone dataset RGB input images Vertex colors Our approach R. Maier, J. Stückler, D. Cremers: Super-Resolution Keyframe Fusion for 3D Modeling with High-Quality Textures 17

Keyboard dataset Vertex colors Our approach RGB input images R. Maier, J. Stückler, D. Cremers: Super-Resolution Keyframe Fusion for 3D Modeling with High-Quality Textures 18

Conclusion Computer Vision Group Robust and efficient method for high-quality texture mapping in RGB-D-based 3D reconstruction R. Maier, J. Stückler, D. Cremers: Super-Resolution Keyframe Fusion for 3D Modeling with High-Quality Textures 19

Conclusion Computer Vision Group Robust and efficient method for high-quality texture mapping in RGB-D-based 3D reconstruction Fuse low-quality color images into SR keyframes R. Maier, J. Stückler, D. Cremers: Super-Resolution Keyframe Fusion for 3D Modeling with High-Quality Textures 19

Conclusion Computer Vision Group Robust and efficient method for high-quality texture mapping in RGB-D-based 3D reconstruction Fuse low-quality color images into SR keyframes Map high-quality keyframes onto 3D model texture using weighted median scheme R. Maier, J. Stückler, D. Cremers: Super-Resolution Keyframe Fusion for 3D Modeling with High-Quality Textures 19

Conclusion Computer Vision Group Robust and efficient method for high-quality texture mapping in RGB-D-based 3D reconstruction Fuse low-quality color images into SR keyframes Map high-quality keyframes onto 3D model texture using weighted median scheme Experimental results: Increased photo-realism of reconstructed 3D models Very efficient and practical post-processing step (runtimes within a few minutes) R. Maier, J. Stückler, D. Cremers: Super-Resolution Keyframe Fusion for 3D Modeling with High-Quality Textures 19

Conclusion Computer Vision Group Robust and efficient method for high-quality texture mapping in RGB-D-based 3D reconstruction Fuse low-quality color images into SR keyframes Map high-quality keyframes onto 3D model texture using weighted median scheme Experimental results: Increased photo-realism of reconstructed 3D models Very efficient and practical post-processing step (runtimes within a few minutes) Thank you! R. Maier, J. Stückler, D. Cremers: Super-Resolution Keyframe Fusion for 3D Modeling with High-Quality Textures 19

2024 © technodocbox.com Privacy Policy | Terms of Service | Feedback