Outline. Immersive Visual Communication Pipeline. Lec 02 Human Vision System
|
|
- Felicia Fisher
- 5 years ago
- Views:
Transcription
1 Outline CS/EE 5590 / (Class Ids: 44873, 44874) Fall 2016, Tue 5-8:15pm@Edu Room 260 Special Topics: Advanced Multimedia Communication Lec 02 Human Vision System Re-Cap of Lec 01 Human Vision System Human Vision Perception Stereo Vision Summary Zhu Li Dept of CSEE, UMKC Office: FH560E, lizhu@umkc.edu, Ph: x Z. Li, Adv. Multimedia Communciation, 2016 Fall p.1 Z. Li, Adv. Multimedia Communciation, 2016 Fall p : What is New? Image Sensors Communication Applications mmwave 5G Immersive/3D Visual Communication FD-MIMO Immersive Visual Communication Pipeline End-to-End pipeline Latency Compression Efficiency: HMD requires very high resolution Human Vision System Characteristics LiDAR Depth Sensor Free View Point Video Device Centric Networking 360 Camera Massive Connectivity & QoS Slicing Precise/Personalized Medicine: Genome Data Compression Image stitching, projection, and mapping encode decode Rendering Hyperspectral p.3 p.4
2 Synthetic Content to VR HMD Playing desktop game on HMD with direct body and gesture interaction Challenges Mobile device GPU is weak, cannot render UHD quality game content Render at the edge and stream to HMD with very low latency Precise body/head motion/gesture tracking Innovation Synthetic content transcoding with full 3D object, texture, motion and lighting info Very low latency/complexity decoding Mixed transcoding and local rendering scalable coding from low-res local rendering HVS enabled pre-filtering of the signal Low latency precise gesture/body motion tracking Point Cloud Capture and Compression Point Cloud Capture Tech Stereo Camera Array ToF depth sensor Structure Light + Stereo Camera Point Cloud Data Compression Static: Octree decomposition Dynamic: no good solutions yet New approach: Graph Signal Processing Z. Li, Adv. Multimedia Communciation, 2016 Fall p.5 Z. Li, Adv. Multimedia Communciation, 2016 Fall p.6 Image/Video Super Resolution Useful for display adaptation and compression TV now UHD, but most content on DVD are just HD (720p) Scalable coding: stream an HD version to phones, extra for TV as super resolved version Light Field Sensorial Data: Light Field Compression Approaches Sparse Coupled Dictionaries: o Image super-resolution via sparse representation, J. Yang, J Wright, TS Huang, Y Ma, IEEE transactions on image processing 19 (11), Deep Learning: o Chao Dong, Chen Change Loy, Kaiming He, Xiaoou Tang: Image Super-Resolution Using Deep Convolutional Networks. IEEE Trans. Pattern Anal. Mach. Intell. 38(2): (2016) Utilizing existing image/video codec Z. Li, Adv. Multimedia Communciation, 2016 Fall p.7 Z. Li, Adv. Multimedia Communciation, 2016 Fall p.8
3 Genome Data Compression Outline Precision Medicine Genome data for personalized diagnosis & treatment Genome data is HUGE Need compression: low delay, high thruput. MPEG Genome Data Compression Group Started the work FastQ: non-aligned data SAM/BAM: aligned data New approach Deep learning to learn a better context model to drive Arithmetic coding efficiency Re-Cap of Lec 01 Human Vision System Human Vision Perception Stereo Vision Summary Z. Li, Adv. Multimedia Communciation, 2016 Fall p.9 Z. Li, Adv. Multimedia Communciation, 2016 Fall p.10 Anatomy of human eye The Human Vision System Pipeline The Human Eye optics: Lens: cornea and aqueous humour Lens control: muscle group called zonula, changes the shape and position of the lens Aperture control: iris is a muscle that change the size of pupil. Human eye sensors: Photon sensors: the back of the eye is called retina, photo sensor cells concentrate around fovea Blind spot: where optical nerve terminates Top down view The signal path: * + Z. Li, Adv. Multimedia Communciation, 2016 Fall p.11 Z. Li, Adv. Multimedia Communciation, 2016 Fall p.12
4 The Retina Circuits Retina photon sensor cells Approx. 120 million rods Approx. 6 million cones Approx less than 1 million optical nerves (ganlion) connecting to brain Visual Functions at Retina Vision function at retina Cones concentrated around the yellow spot, or macular, about 2.5-3mm in diameter In the center of the macular, approx. 0.3mm in diameter, has no rods, called fovea centralis, for high acuity vision. Rods are distributed sparsely away from fovea, and are good for low light vision, and motion detection. Nigh vision 2 nd blind spot: on fovea. Rods for low light vision, cones for normal light high resolution vision Z. Li, Adv. Multimedia Communciation, 2016 Fall p.13 Z. Li, Adv. Multimedia Communciation, 2016 Fall p.14 Lateral Geniculate Nucleus Retina is doing low level luminance processing via rods/cones Approx 1 million optical nerves connect the signal to LGN (Lateral Geniculate Nucleus) : mid level vision LGN has 6 layers More on the contrasts and movements First stage of stereo vision processing Color vision: Paired response for red-green and blue-yellow signals Primary and secondary visual cortex Optical radiations connect to primary visual cortex Primary is then connected to secondary cortex Complete higher level of vision tasks 1000x1000 color Z. Li, Adv. Multimedia Communciation, 2016 Fall p.15 Lateral Inhibition Edge Perceiving Edge info processing at Retina circuits More rods/cones than optical nerves Not all photon reception is feedback to brains, the ganlion cells have this lateral inhibition function to suppress the amount of information fired back to visual cortex No inhibition Inhibition: enhance edge Mach Band: the edge perception with inhibition Z. Li, Adv. Multimedia Communciation, 2016 Fall p.16
5 Human Vision Perception FOV: Field of View (approx. 130 o /120 o ) Overall FoV FoV Mono vision Stereo vision Foveated vision Z. Li, Adv. Multimedia Communciation, 2016 Fall p.17 Z. Li, Adv. Multimedia Communciation, 2016 Fall p.18 Spatial Contrast Sensitive Function Spatial CSF Contrast: (I max I min )/(I max + I min ) Flickering test: Temporal CSF Contrast Temporal Contrast Sensitivity Function Greater than 60hz, no flickering- that is why LED panel need > 60hz refresh rate Contrasts sensitivity is low at both higher and lower freq. Spatial Freq Z. Li, Adv. Multimedia Communciation, 2016 Fall p.19 Z. Li, Adv. Multimedia Communciation, 2016 Fall p.20
6 Spatio-temporal CSF and Application Render error tolerance map Seeing in 3D Depth Perception Pinhole camera and Perspective Projection Perspective Projection Cues Bigger is closer, if have known size, Ames room illusion Yangli Hector Yee, Sumanta N. Pattanaik, Donald P. Greenberg: Spatiotemporal sensitivity and visual attention for efficient rendering of dynamic environments. ACM Trans. Graph. 20(1): (2001) Z. Li, Adv. Multimedia Communciation, 2016 Fall p.21 Z. Li, Adv. Multimedia Communciation, 2016 Fall p.22 Mono View Depth Illusion Julian Beever s work: perspective cue illusion Mono Vision: Lighting & Occlusion Depth cue from Lighting Have a sense of protruding vs receding patterns, prob developed during the human vision evolution Occlusion Z. Li, Adv. Multimedia Communciation, 2016 Fall p.23 Z. Li, Adv. Multimedia Communciation, 2016 Fall p.24
7 Mono Vision Cue for 3D - Focus Depth from Focus Human eyes constantly re-focus to get a sense of depth HMD VR has fixed depth content, and a major cause for fatigue and nauseate Light Field! Stereo Vision Depth Perception For closer views, stereoscopic vision provides more accurate depth perception Nvidia LF HMD From The Art of Photography, Canon Z. Li, Adv. Multimedia Communciation, 2016 Fall p.25 Z. Li, Adv. Multimedia Communciation, 2016 Fall p.26 Binocular stereo Depth from binocular disparity Given a calibrated binocular stereo pair, fuse it to produce a depth image Humans can do it Human Perception of Depth P: converging point C: object nearer projects to the outside of the P, disparity = + Stereograms: Invented by Sir Charles Wheatstone, 1838 Sign and magnitude of disparity F: object farther projects to the inside of the P, disparity = - Z. Li, Adv. Multimedia Communication, 2016 p.27
8 Binocular Optical Range Finder - WWII Depth from disparity Binocular stereo Given a calibrated binocular stereo pair, fuse it to produce a depth image Humans can do it USS Alabama Yamato Z. Li, Adv. Multimedia Communciation, 2016 Fall p.29 Autostereograms: Z. Li, Adv. Multimedia Communication, 2016 p.30 Basic stereo matching algorithm Simplest Case: Parallel images Image planes of cameras are parallel to each other and to the baseline Camera centers are at same height Focal lengths are the same For each pixel in the first image Find corresponding epipolar line in the right image Examine all pixels on the epipolar line and pick the best match Triangulate the matches to get depth information Simplest case: epipolar lines are scanlines When does this happen? Z. Li, Adv. Multimedia Communication, 2016 p.31 Z. Li, Adv. Multimedia Communication, 2016 p.32
9 Simplest Case: Parallel images Image planes of cameras are parallel to each other and to the baseline Camera centers are at same height Focal lengths are the same Then, epipolar lines fall along the horizontal scan lines of the images Stereo Correspondence Determine Pixel Correspondence Pairs of points that correspond to same scene point epipolar line X epipolar plane C 1 C 2 epipolar line Epipolar Constraint Reduces correspondence problem to 1D search along conjugate epipolar lines Java demo: Z. Li, Adv. Multimedia Communication, 2016 p.33 Depth from disparity X Correspondence problem Multiple matching hypotheses satisfy the epipolar constraint, but which one is correct? x x z f f O Baseline O B B f B f disparity : d x x depth : z z d Disparity is inversely proportional to depth! Z. Li, Adv. Multimedia Communication, 2016 p.35 Z. Li, Adv. Multimedia Communciation, 2016 Fall p.36
10 Correspondence problem Let s make some assumptions to simplify the matching problem The baseline is relatively small (compared to the depth of scene points) Then most scene points are visible in both views Also, matching regions are similar in appearance Correspondence problem Let s make some assumptions to simplify the matching problem The baseline is relatively small (compared to the depth of scene points) Then most scene points are visible in both views Also, matching regions are similar in appearance Z. Li, Adv. Multimedia Communciation, 2016 Fall p.37 Z. Li, Adv. Multimedia Communciation, 2016 Fall p.38 Correspondence search with similarity constraint Image registration (revisited) scanline Left Right How do we determine correspondences? block matching or SSD (sum squared differences) d is the disparity (horizontal motion) Matching cost disparity Slide a window along the right scanline and compare contents of that window with the reference window in the left image Matching cost: SSD or normalized correlation What is the proper window size? Z. Li, Adv. Multimedia Communciation, 2016 Fall p.39 Stereo matching 40
11 Effect of window size The similarity constraint Smaller window + More detail More noise Corresponding regions in two images should be similar in appearance and non-corresponding regions should be different Larger window + Smoother disparity maps Less detail Left view Right view When will the similarity constraint fail? W = 3 W = 20 Z. Li, Adv. Multimedia Communciation, 2016 Fall p.41 Z. Li, Adv. Multimedia Communciation, 2016 Fall p.42 Limitations of similarity constraint Where pixel similarity fails Results with window search Windowed search: SSD of windowed pixels Textureless surfaces Occlusions, repetition Depth Ground truth Window-based matching Non-Lambertian surfaces Z. Li, Adv. Multimedia Communciation, 2016 Fall p.43 Z. Li, Adv. Multimedia Communciation, 2016 Fall p.44
12 Improve Window Method:Non-local constraints Uniqueness For any point in one image, there should be at most one matching point in the other image Non-local constraints Uniqueness For any point in one image, there should be at most one matching point in the other image Ordering Corresponding points should be in the same order in both views Occlusion: ordering constraint doesn t hold Z. Li, Adv. Multimedia Communciation, 2016 Fall p.45 Z. Li, Adv. Multimedia Communciation, 2016 Fall p.46 Non-local constraints Uniqueness For any point in one image, there should be at most one matching point in the other image Ordering Corresponding points should be in the same order in both views Smoothness We expect disparity values to change slowly (for the most part) Achieved by various smoothness penalties. Scanline stereo Try to coherently match pixels on the entire scanline Different scanlines are still optimized independently Left image Right image Z. Li, Adv. Multimedia Communciation, 2016 Fall p.47 Z. Li, Adv. Multimedia Communciation, 2016 Fall p.48
13 Left occlusion Shortest paths for scan-line stereo I Left image Right image I Coherent stereo on 2D grid Scanline stereo generates streaking artifacts S left Right occlusion q C corr C occl t s p S right C occl Can be implemented with dynamic programming Ohta & Kanade 85, Cox et al. 96 Slide credit: Y. Boykov Z. Li, Adv. Multimedia Communciation, 2016 Fall p.49 Can t use dynamic programming to find spatially coherent disparities/ correspondences on a 2D grid Z. Li, Adv. Multimedia Communciation, 2016 Fall p.50 Stereo matching as energy minimization Energy functions of this form can be minimized using graph cuts I 1 I 2 D W 1 (i) W 2 (i+d(i)) D(i) Stereo matching as energy minimization I 1 I 2 D W 1 (i) W 2 (i+d(i)) D(i) E data is the energy from pixel matching E smooth penalizes the large displacement with a monotonically increasing function ρ E E I, I, D) E ( D) E W 2 1( i) W2 ( i D( i data )) i data( 1 2 smooth Esmooth D( i) D( j) neighborsi, j Y. Boykov, O. Veksler, and R. Zabih, Fast Approximate Energy Minimization via Graph Cuts, PAMI 2001 Z. Li, Adv. Multimedia Communciation, 2016 Fall p.51 P D I, I ) P( I, I D) P( ) ( D Probabilistic interpretation: we want to find a Maximum A Posteriori (MAP) estimate of disparity image D: log P( D I1, I2) log P( I1, I2 D) log P( D) E Edata ( I1, I2, D) Esmooth ( D) Z. Li, Adv. Multimedia Communciation, 2016 Fall p.52
14 The role of the baseline Small baseline: large depth error that is why 15m range finder Large baseline: difficult search problem due to occlusion Problem for wide baselines: Foreshortening Matching with fixed-size windows will fail! Possible solution: adaptively vary window size Another solution: model-based stereo Small Baseline Large Baseline Source: S. Seitz Z. Li, Adv. Multimedia Communciation, 2016 Fall p.53 Z. Li, Adv. Multimedia Communciation, 2016 Fall p.54 Active stereo with structured light Active stereo with structured light Project structured light patterns onto the object Simplifies the correspondence problem Allows us to use only one camera camera projector Magic Leap L. Zhang, B. Curless, and S. M. Seitz. Rapid Shape Acquisition Using Color Structured Light and Multi-pass Dynamic Programming. 3DPVT 2002 Z. Li, Adv. Multimedia Communciation, 2016 Fall p.55 Z. Li, Adv. Multimedia Communciation, 2016 Fall p.56
15 Laser scanning Laser scanned models Optical triangulation Project a single stripe of laser light Scan it across the surface of the object This is a very precise version of structured light scanning Digital Michelangelo Project Source: S. Seitz Z. Li, Adv. Multimedia Communciation, 2016 Fall p.57 The Digital Michelangelo Project, Levoy et al. Source: S. Seitz Z. Li, Adv. Multimedia Communciation, 2016 Fall p.58 Laser scanned models Details: Laser scanned models 1.0 mm resolution (total 56 million triangles) The Digital Michelangelo Project, Levoy et al. Z. Li, Adv. Multimedia Communciation, 2016 Fall Source: S. Seitz p.59 The Digital Michelangelo Project, Levoy et al. Source: S. Seitz Z. Li, Adv. Multimedia Communciation, 2016 Fall p.60
16 Aligning range images A single range scan is not sufficient to describe a complex surface Need techniques to register multiple range images Check out Point Cloud Library! B. Curless and M. Levoy, A Volumetric Method for Building Complex Models from Range Images, SIGGRAPH 1996 Z. Li, Adv. Multimedia Communciation, 2016 Fall p.61 Summary Human Vision System Retina : serving as lens and photon sensor units, as well as some low level vision processing LGN: 6 layers with mid level vision functions Visual Cortex: higher level vision functions Human Depth perception: Mono vs Stereo vision Depth Perception Computational Approach Stereo matching Structured light assisted stereo matching Next Class: Stereo Depth Estimation Depth Map Compression Potential project: Sunny Optics Structured Light Stereo Depth sensor for gesture recognition. Z. Li, Adv. Multimedia Communciation, 2016 Fall p.62
Stereo vision. Many slides adapted from Steve Seitz
Stereo vision Many slides adapted from Steve Seitz What is stereo vision? Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape What is
More informationStereo. Many slides adapted from Steve Seitz
Stereo Many slides adapted from Steve Seitz Binocular stereo Given a calibrated binocular stereo pair, fuse it to produce a depth image image 1 image 2 Dense depth map Binocular stereo Given a calibrated
More informationBinocular stereo. Given a calibrated binocular stereo pair, fuse it to produce a depth image. Where does the depth information come from?
Binocular Stereo Binocular stereo Given a calibrated binocular stereo pair, fuse it to produce a depth image Where does the depth information come from? Binocular stereo Given a calibrated binocular stereo
More informationStereo. 11/02/2012 CS129, Brown James Hays. Slides by Kristen Grauman
Stereo 11/02/2012 CS129, Brown James Hays Slides by Kristen Grauman Multiple views Multi-view geometry, matching, invariant features, stereo vision Lowe Hartley and Zisserman Why multiple views? Structure
More informationBIL Computer Vision Apr 16, 2014
BIL 719 - Computer Vision Apr 16, 2014 Binocular Stereo (cont d.), Structure from Motion Aykut Erdem Dept. of Computer Engineering Hacettepe University Slide credit: S. Lazebnik Basic stereo matching algorithm
More informationStereo: Disparity and Matching
CS 4495 Computer Vision Aaron Bobick School of Interactive Computing Administrivia PS2 is out. But I was late. So we pushed the due date to Wed Sept 24 th, 11:55pm. There is still *no* grace period. To
More informationCS 4495 Computer Vision A. Bobick. Motion and Optic Flow. Stereo Matching
Stereo Matching Fundamental matrix Let p be a point in left image, p in right image l l Epipolar relation p maps to epipolar line l p maps to epipolar line l p p Epipolar mapping described by a 3x3 matrix
More informationCS 4495 Computer Vision A. Bobick. Motion and Optic Flow. Stereo Matching
Stereo Matching Fundamental matrix Let p be a point in left image, p in right image l l Epipolar relation p maps to epipolar line l p maps to epipolar line l p p Epipolar mapping described by a 3x3 matrix
More informationLecture 9 & 10: Stereo Vision
Lecture 9 & 10: Stereo Vision Professor Fei- Fei Li Stanford Vision Lab 1 What we will learn today? IntroducEon to stereo vision Epipolar geometry: a gentle intro Parallel images Image receficaeon Solving
More informationCS4495/6495 Introduction to Computer Vision. 3B-L3 Stereo correspondence
CS4495/6495 Introduction to Computer Vision 3B-L3 Stereo correspondence For now assume parallel image planes Assume parallel (co-planar) image planes Assume same focal lengths Assume epipolar lines are
More informationChaplin, Modern Times, 1936
Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection set] Multi-view geometry problems Structure: Given projections
More informationEECS 442 Computer vision. Stereo systems. Stereo vision Rectification Correspondence problem Active stereo vision systems
EECS 442 Computer vision Stereo systems Stereo vision Rectification Correspondence problem Active stereo vision systems Reading: [HZ] Chapter: 11 [FP] Chapter: 11 Stereo vision P p p O 1 O 2 Goal: estimate
More informationImage Based Reconstruction II
Image Based Reconstruction II Qixing Huang Feb. 2 th 2017 Slide Credit: Yasutaka Furukawa Image-Based Geometry Reconstruction Pipeline Last Lecture: Multi-View SFM Multi-View SFM This Lecture: Multi-View
More informationLecture 10: Multi view geometry
Lecture 10: Multi view geometry Professor Fei Fei Li Stanford Vision Lab 1 What we will learn today? Stereo vision Correspondence problem (Problem Set 2 (Q3)) Active stereo vision systems Structure from
More informationEpipolar Geometry and Stereo Vision
Epipolar Geometry and Stereo Vision Computer Vision Shiv Ram Dubey, IIIT Sri City Many slides from S. Seitz and D. Hoiem Last class: Image Stitching Two images with rotation/zoom but no translation. X
More informationEpipolar Geometry and Stereo Vision
Epipolar Geometry and Stereo Vision Computer Vision Jia-Bin Huang, Virginia Tech Many slides from S. Seitz and D. Hoiem Last class: Image Stitching Two images with rotation/zoom but no translation. X x
More informationStereo. Outline. Multiple views 3/29/2017. Thurs Mar 30 Kristen Grauman UT Austin. Multi-view geometry, matching, invariant features, stereo vision
Stereo Thurs Mar 30 Kristen Grauman UT Austin Outline Last time: Human stereopsis Epipolar geometry and the epipolar constraint Case example with parallel optical axes General case with calibrated cameras
More informationMultiple View Geometry
Multiple View Geometry Martin Quinn with a lot of slides stolen from Steve Seitz and Jianbo Shi 15-463: Computational Photography Alexei Efros, CMU, Fall 2007 Our Goal The Plenoptic Function P(θ,φ,λ,t,V
More informationIntroduction à la vision artificielle X
Introduction à la vision artificielle X Jean Ponce Email: ponce@di.ens.fr Web: http://www.di.ens.fr/~ponce Planches après les cours sur : http://www.di.ens.fr/~ponce/introvis/lect10.pptx http://www.di.ens.fr/~ponce/introvis/lect10.pdf
More informationFundamental matrix. Let p be a point in left image, p in right image. Epipolar relation. Epipolar mapping described by a 3x3 matrix F
Fundamental matrix Let p be a point in left image, p in right image l l Epipolar relation p maps to epipolar line l p maps to epipolar line l p p Epipolar mapping described by a 3x3 matrix F Fundamental
More informationCS5670: Computer Vision
CS5670: Computer Vision Noah Snavely, Zhengqi Li Stereo Single image stereogram, by Niklas Een Mark Twain at Pool Table", no date, UCR Museum of Photography Stereo Given two images from different viewpoints
More informationWhat have we leaned so far?
What have we leaned so far? Camera structure Eye structure Project 1: High Dynamic Range Imaging What have we learned so far? Image Filtering Image Warping Camera Projection Model Project 2: Panoramic
More informationFinal project bits and pieces
Final project bits and pieces The project is expected to take four weeks of time for up to four people. At 12 hours per week per person that comes out to: ~192 hours of work for a four person team. Capstone:
More informationComputer Vision Lecture 17
Computer Vision Lecture 17 Epipolar Geometry & Stereo Basics 13.01.2015 Bastian Leibe RWTH Aachen http://www.vision.rwth-aachen.de leibe@vision.rwth-aachen.de Announcements Seminar in the summer semester
More informationComputer Vision Lecture 17
Announcements Computer Vision Lecture 17 Epipolar Geometry & Stereo Basics Seminar in the summer semester Current Topics in Computer Vision and Machine Learning Block seminar, presentations in 1 st week
More informationLecture 10: Multi-view geometry
Lecture 10: Multi-view geometry Professor Stanford Vision Lab 1 What we will learn today? Review for stereo vision Correspondence problem (Problem Set 2 (Q3)) Active stereo vision systems Structure from
More informationThere are many cues in monocular vision which suggests that vision in stereo starts very early from two similar 2D images. Lets see a few...
STEREO VISION The slides are from several sources through James Hays (Brown); Srinivasa Narasimhan (CMU); Silvio Savarese (U. of Michigan); Bill Freeman and Antonio Torralba (MIT), including their own
More informationLecture 14: Computer Vision
CS/b: Artificial Intelligence II Prof. Olga Veksler Lecture : Computer Vision D shape from Images Stereo Reconstruction Many Slides are from Steve Seitz (UW), S. Narasimhan Outline Cues for D shape perception
More informationProject 2 due today Project 3 out today. Readings Szeliski, Chapter 10 (through 10.5)
Announcements Stereo Project 2 due today Project 3 out today Single image stereogram, by Niklas Een Readings Szeliski, Chapter 10 (through 10.5) Public Library, Stereoscopic Looking Room, Chicago, by Phillips,
More informationPublic Library, Stereoscopic Looking Room, Chicago, by Phillips, 1923
Public Library, Stereoscopic Looking Room, Chicago, by Phillips, 1923 Teesta suspension bridge-darjeeling, India Mark Twain at Pool Table", no date, UCR Museum of Photography Woman getting eye exam during
More informationMultiple View Geometry
Multiple View Geometry CS 6320, Spring 2013 Guest Lecture Marcel Prastawa adapted from Pollefeys, Shah, and Zisserman Single view computer vision Projective actions of cameras Camera callibration Photometric
More informationStereo II CSE 576. Ali Farhadi. Several slides from Larry Zitnick and Steve Seitz
Stereo II CSE 576 Ali Farhadi Several slides from Larry Zitnick and Steve Seitz Camera parameters A camera is described by several parameters Translation T of the optical center from the origin of world
More informationColorado School of Mines. Computer Vision. Professor William Hoff Dept of Electrical Engineering &Computer Science.
Professor William Hoff Dept of Electrical Engineering &Computer Science http://inside.mines.edu/~whoff/ 1 Stereo Vision 2 Inferring 3D from 2D Model based pose estimation single (calibrated) camera > Can
More informationProject 4 Results. Representation. Data. Learning. Zachary, Hung-I, Paul, Emanuel. SIFT and HoG are popular and successful.
Project 4 Results Representation SIFT and HoG are popular and successful. Data Hugely varying results from hard mining. Learning Non-linear classifier usually better. Zachary, Hung-I, Paul, Emanuel Project
More informationMiniature faking. In close-up photo, the depth of field is limited.
Miniature faking In close-up photo, the depth of field is limited. http://en.wikipedia.org/wiki/file:jodhpur_tilt_shift.jpg Miniature faking Miniature faking http://en.wikipedia.org/wiki/file:oregon_state_beavers_tilt-shift_miniature_greg_keene.jpg
More informationEpipolar Geometry and Stereo Vision
CS 1674: Intro to Computer Vision Epipolar Geometry and Stereo Vision Prof. Adriana Kovashka University of Pittsburgh October 5, 2016 Announcement Please send me three topics you want me to review next
More informationNinio, J. and Stevens, K. A. (2000) Variations on the Hermann grid: an extinction illusion. Perception, 29,
Ninio, J. and Stevens, K. A. (2000) Variations on the Hermann grid: an extinction illusion. Perception, 29, 1209-1217. CS 4495 Computer Vision A. Bobick Sparse to Dense Correspodence Building Rome in
More informationStereo Vision A simple system. Dr. Gerhard Roth Winter 2012
Stereo Vision A simple system Dr. Gerhard Roth Winter 2012 Stereo Stereo Ability to infer information on the 3-D structure and distance of a scene from two or more images taken from different viewpoints
More informationProject 3 code & artifact due Tuesday Final project proposals due noon Wed (by ) Readings Szeliski, Chapter 10 (through 10.5)
Announcements Project 3 code & artifact due Tuesday Final project proposals due noon Wed (by email) One-page writeup (from project web page), specifying:» Your team members» Project goals. Be specific.
More informationRecap from Previous Lecture
Recap from Previous Lecture Tone Mapping Preserve local contrast or detail at the expense of large scale contrast. Changing the brightness within objects or surfaces unequally leads to halos. We are now
More informationRealtime 3D Computer Graphics Virtual Reality
Realtime 3D Computer Graphics Virtual Reality Human Visual Perception The human visual system 2 eyes Optic nerve: 1.5 million fibers per eye (each fiber is the axon from a neuron) 125 million rods (achromatic
More informationRecap: Features and filters. Recap: Grouping & fitting. Now: Multiple views 10/29/2008. Epipolar geometry & stereo vision. Why multiple views?
Recap: Features and filters Epipolar geometry & stereo vision Tuesday, Oct 21 Kristen Grauman UT-Austin Transforming and describing images; textures, colors, edges Recap: Grouping & fitting Now: Multiple
More informationLecture'9'&'10:'' Stereo'Vision'
Lecture'9'&'10:'' Stereo'Vision' Dr.'Juan'Carlos'Niebles' Stanford'AI'Lab' ' Professor'FeiAFei'Li' Stanford'Vision'Lab' 1' Dimensionality'ReducIon'Machine'(3D'to'2D)' 3D world 2D image Point of observation
More informationThink-Pair-Share. What visual or physiological cues help us to perceive 3D shape and depth?
Think-Pair-Share What visual or physiological cues help us to perceive 3D shape and depth? [Figure from Prados & Faugeras 2006] Shading Focus/defocus Images from same point of view, different camera parameters
More informationEE795: Computer Vision and Intelligent Systems
EE795: Computer Vision and Intelligent Systems Spring 2012 TTh 17:30-18:45 FDH 204 Lecture 14 130307 http://www.ee.unlv.edu/~b1morris/ecg795/ 2 Outline Review Stereo Dense Motion Estimation Translational
More informationLecture 19: Depth Cameras. Visual Computing Systems CMU , Fall 2013
Lecture 19: Depth Cameras Visual Computing Systems Continuing theme: computational photography Cameras capture light, then extensive processing produces the desired image Today: - Capturing scene depth
More informationStereo and structured light
Stereo and structured light http://graphics.cs.cmu.edu/courses/15-463 15-463, 15-663, 15-862 Computational Photography Fall 2018, Lecture 20 Course announcements Homework 5 is still ongoing. - Make sure
More informationComplex Sensors: Cameras, Visual Sensing. The Robotics Primer (Ch. 9) ECE 497: Introduction to Mobile Robotics -Visual Sensors
Complex Sensors: Cameras, Visual Sensing The Robotics Primer (Ch. 9) Bring your laptop and robot everyday DO NOT unplug the network cables from the desktop computers or the walls Tuesday s Quiz is on Visual
More informationCameras and Stereo CSE 455. Linda Shapiro
Cameras and Stereo CSE 455 Linda Shapiro 1 Müller-Lyer Illusion http://www.michaelbach.de/ot/sze_muelue/index.html What do you know about perspective projection? Vertical lines? Other lines? 2 Image formation
More information3D Computer Vision. Depth Cameras. Prof. Didier Stricker. Oliver Wasenmüller
3D Computer Vision Depth Cameras Prof. Didier Stricker Oliver Wasenmüller Kaiserlautern University http://ags.cs.uni-kl.de/ DFKI Deutsches Forschungszentrum für Künstliche Intelligenz http://av.dfki.de
More informationKinect Device. How the Kinect Works. Kinect Device. What the Kinect does 4/27/16. Subhransu Maji Slides credit: Derek Hoiem, University of Illinois
4/27/16 Kinect Device How the Kinect Works T2 Subhransu Maji Slides credit: Derek Hoiem, University of Illinois Photo frame-grabbed from: http://www.blisteredthumbs.net/2010/11/dance-central-angry-review
More informationHuman Body Recognition and Tracking: How the Kinect Works. Kinect RGB-D Camera. What the Kinect Does. How Kinect Works: Overview
Human Body Recognition and Tracking: How the Kinect Works Kinect RGB-D Camera Microsoft Kinect (Nov. 2010) Color video camera + laser-projected IR dot pattern + IR camera $120 (April 2012) Kinect 1.5 due
More informationMulti-view stereo. Many slides adapted from S. Seitz
Multi-view stereo Many slides adapted from S. Seitz Beyond two-view stereo The third eye can be used for verification Multiple-baseline stereo Pick a reference image, and slide the corresponding window
More informationBasic distinctions. Definitions. Epstein (1965) familiar size experiment. Distance, depth, and 3D shape cues. Distance, depth, and 3D shape cues
Distance, depth, and 3D shape cues Pictorial depth cues: familiar size, relative size, brightness, occlusion, shading and shadows, aerial/ atmospheric perspective, linear perspective, height within image,
More informationPERCEIVING DEPTH AND SIZE
PERCEIVING DEPTH AND SIZE DEPTH Cue Approach Identifies information on the retina Correlates it with the depth of the scene Different cues Previous knowledge Slide 3 Depth Cues Oculomotor Monocular Binocular
More informationCEng Computational Vision
CEng 583 - Computational Vision 2011-2012 Spring Week 4 18 th of March, 2011 Today 3D Vision Binocular (Multi-view) cues: Stereopsis Motion Monocular cues Shading Texture Familiar size etc. "God must
More informationStereo CSE 576. Ali Farhadi. Several slides from Larry Zitnick and Steve Seitz
Stereo CSE 576 Ali Farhadi Several slides from Larry Zitnick and Steve Seitz Why do we perceive depth? What do humans use as depth cues? Motion Convergence When watching an object close to us, our eyes
More informationFundamentals of Stereo Vision Michael Bleyer LVA Stereo Vision
Fundamentals of Stereo Vision Michael Bleyer LVA Stereo Vision What Happened Last Time? Human 3D perception (3D cinema) Computational stereo Intuitive explanation of what is meant by disparity Stereo matching
More informationLecture 6 Stereo Systems Multi-view geometry
Lecture 6 Stereo Systems Multi-view geometry Professor Silvio Savarese Computational Vision and Geometry Lab Silvio Savarese Lecture 6-5-Feb-4 Lecture 6 Stereo Systems Multi-view geometry Stereo systems
More informationCS 4495/7495 Computer Vision Frank Dellaert, Fall 07. Dense Stereo Some Slides by Forsyth & Ponce, Jim Rehg, Sing Bing Kang
CS 4495/7495 Computer Vision Frank Dellaert, Fall 07 Dense Stereo Some Slides by Forsyth & Ponce, Jim Rehg, Sing Bing Kang Etymology Stereo comes from the Greek word for solid (στερεο), and the term can
More informationEpipolar Geometry and Stereo Vision
CS 1699: Intro to Computer Vision Epipolar Geometry and Stereo Vision Prof. Adriana Kovashka University of Pittsburgh October 8, 2015 Today Review Projective transforms Image stitching (homography) Epipolar
More informationLecture 14: Basic Multi-View Geometry
Lecture 14: Basic Multi-View Geometry Stereo If I needed to find out how far point is away from me, I could use triangulation and two views scene point image plane optical center (Graphic from Khurram
More informationCS 2770: Intro to Computer Vision. Multiple Views. Prof. Adriana Kovashka University of Pittsburgh March 14, 2017
CS 277: Intro to Computer Vision Multiple Views Prof. Adriana Kovashka Universit of Pittsburgh March 4, 27 Plan for toda Affine and projective image transformations Homographies and image mosaics Stereo
More informationMosaics wrapup & Stereo
Mosaics wrapup & Stereo Tues Oct 20 Last time: How to stitch a panorama? Basic Procedure Take a sequence of images from the same position Rotate the camera about its optical center Compute transformation
More informationStereo. Shadows: Occlusions: 3D (Depth) from 2D. Depth Cues. Viewing Stereo Stereograms Autostereograms Depth from Stereo
Stereo Viewing Stereo Stereograms Autostereograms Depth from Stereo 3D (Depth) from 2D 3D information is lost by projection. How do we recover 3D information? Image 3D Model Depth Cues Shadows: Occlusions:
More informationStereovision. Binocular disparity
Stereovision Binocular disparity Retinal correspondence Uncrossed disparity Horoptor Crossed disparity Horoptor, crossed and uncrossed disparity Wheatsteone stereoscope (c. 1838) Red-green anaglyph How
More informationStereo and Epipolar geometry
Previously Image Primitives (feature points, lines, contours) Today: Stereo and Epipolar geometry How to match primitives between two (multiple) views) Goals: 3D reconstruction, recognition Jana Kosecka
More informationA virtual tour of free viewpoint rendering
A virtual tour of free viewpoint rendering Cédric Verleysen ICTEAM institute, Université catholique de Louvain, Belgium cedric.verleysen@uclouvain.be Organization of the presentation Context Acquisition
More informationLecture 10 Dense 3D Reconstruction
Institute of Informatics Institute of Neuroinformatics Lecture 10 Dense 3D Reconstruction Davide Scaramuzza 1 REMODE: Probabilistic, Monocular Dense Reconstruction in Real Time M. Pizzoli, C. Forster,
More informationDense 3D Reconstruction. Christiano Gava
Dense 3D Reconstruction Christiano Gava christiano.gava@dfki.de Outline Previous lecture: structure and motion II Structure and motion loop Triangulation Wide baseline matching (SIFT) Today: dense 3D reconstruction
More informationMulti-stable Perception. Necker Cube
Multi-stable Perception Necker Cube Spinning dancer illusion, Nobuyuki Kayahara Multiple view geometry Stereo vision Epipolar geometry Lowe Hartley and Zisserman Depth map extraction Essential matrix
More informationDense 3D Reconstruction. Christiano Gava
Dense 3D Reconstruction Christiano Gava christiano.gava@dfki.de Outline Previous lecture: structure and motion II Structure and motion loop Triangulation Today: dense 3D reconstruction The matching problem
More informationPerception II: Pinhole camera and Stereo Vision
Perception II: Pinhole camera and Stereo Vision Davide Scaramuzza Margarita Chli, Paul Furgale, Marco Hutter, Roland Siegwart 1 Mobile Robot Control Scheme knowledge, data base mission commands Localization
More informationVisual Pathways to the Brain
Visual Pathways to the Brain 1 Left half of visual field which is imaged on the right half of each retina is transmitted to right half of brain. Vice versa for right half of visual field. From each eye
More informationFinally: Motion and tracking. Motion 4/20/2011. CS 376 Lecture 24 Motion 1. Video. Uses of motion. Motion parallax. Motion field
Finally: Motion and tracking Tracking objects, video analysis, low level motion Motion Wed, April 20 Kristen Grauman UT-Austin Many slides adapted from S. Seitz, R. Szeliski, M. Pollefeys, and S. Lazebnik
More informationImage Formation. CS418 Computer Graphics Eric Shaffer.
Image Formation CS418 Computer Graphics Eric Shaffer http://graphics.cs.illinois.edu/cs418/fa14 Some stuff about the class Grades probably on usual scale: 97 to 93: A 93 to 90: A- 90 to 87: B+ 87 to 83:
More informationCapturing, Modeling, Rendering 3D Structures
Computer Vision Approach Capturing, Modeling, Rendering 3D Structures Calculate pixel correspondences and extract geometry Not robust Difficult to acquire illumination effects, e.g. specular highlights
More information12/3/2009. What is Computer Vision? Applications. Application: Assisted driving Pedestrian and car detection. Application: Improving online search
Introduction to Artificial Intelligence V22.0472-001 Fall 2009 Lecture 26: Computer Vision Rob Fergus Dept of Computer Science, Courant Institute, NYU Slides from Andrew Zisserman What is Computer Vision?
More informationCS 563 Advanced Topics in Computer Graphics Stereoscopy. by Sam Song
CS 563 Advanced Topics in Computer Graphics Stereoscopy by Sam Song Stereoscopy Introduction Parallax Camera Displaying and Viewing Results Stereoscopy What is it? seeing in three dimensions creates the
More informationLecture 10 Multi-view Stereo (3D Dense Reconstruction) Davide Scaramuzza
Lecture 10 Multi-view Stereo (3D Dense Reconstruction) Davide Scaramuzza REMODE: Probabilistic, Monocular Dense Reconstruction in Real Time, ICRA 14, by Pizzoli, Forster, Scaramuzza [M. Pizzoli, C. Forster,
More informationStructured light , , Computational Photography Fall 2017, Lecture 27
Structured light http://graphics.cs.cmu.edu/courses/15-463 15-463, 15-663, 15-862 Computational Photography Fall 2017, Lecture 27 Course announcements Homework 5 has been graded. - Mean: 129. - Median:
More informationRobert Collins CSE486, Penn State Lecture 08: Introduction to Stereo
Lecture 08: Introduction to Stereo Reading: T&V Section 7.1 Stereo Vision Inferring depth from images taken at the same time by two or more cameras. Basic Perspective Projection Scene Point Perspective
More informationThe Human Visual System!
! The Human Visual System! Gordon Wetzstein! Stanford University! EE 267 Virtual Reality! Lecture 5! stanford.edu/class/ee267/!! nautilus eye, wikipedia! Dawkins, Climbing Mount Improbable,! Norton & Company,
More informationToday. Stereo (two view) reconstruction. Multiview geometry. Today. Multiview geometry. Computational Photography
Computational Photography Matthias Zwicker University of Bern Fall 2009 Today From 2D to 3D using multiple views Introduction Geometry of two views Stereo matching Other applications Multiview geometry
More informationMulti-View Geometry (Ch7 New book. Ch 10/11 old book)
Multi-View Geometry (Ch7 New book. Ch 10/11 old book) Guido Gerig CS-GY 6643, Spring 2016 gerig@nyu.edu Credits: M. Shah, UCF CAP5415, lecture 23 http://www.cs.ucf.edu/courses/cap6411/cap5415/, Trevor
More informationMahdi Amiri. May Sharif University of Technology
Course Presentation Multimedia Systems 3D Technologies Mahdi Amiri May 2014 Sharif University of Technology Binocular Vision (Two Eyes) Advantages A spare eye in case one is damaged. A wider field of view
More informationProf. Feng Liu. Spring /27/2014
Prof. Feng Liu Spring 2014 http://www.cs.pdx.edu/~fliu/courses/cs510/ 05/27/2014 Last Time Video Stabilization 2 Today Stereoscopic 3D Human depth perception 3D displays 3 Stereoscopic media Digital Visual
More informationLecture 8 Active stereo & Volumetric stereo
Lecture 8 Active stereo & Volumetric stereo Active stereo Structured lighting Depth sensing Volumetric stereo: Space carving Shadow carving Voxel coloring Reading: [Szelisky] Chapter 11 Multi-view stereo
More informationMinimizing Noise and Bias in 3D DIC. Correlated Solutions, Inc.
Minimizing Noise and Bias in 3D DIC Correlated Solutions, Inc. Overview Overview of Noise and Bias Digital Image Correlation Background/Tracking Function Minimizing Noise Focus Contrast/Lighting Glare
More informationDepth. Common Classification Tasks. Example: AlexNet. Another Example: Inception. Another Example: Inception. Depth
Common Classification Tasks Recognition of individual objects/faces Analyze object-specific features (e.g., key points) Train with images from different viewing angles Recognition of object classes Analyze
More informationImportant concepts in binocular depth vision: Corresponding and non-corresponding points. Depth Perception 1. Depth Perception Part II
Depth Perception Part II Depth Perception 1 Binocular Cues to Depth Depth Information Oculomotor Visual Accomodation Convergence Binocular Monocular Static Cues Motion Parallax Perspective Size Interposition
More informationlecture 10 - depth from blur, binocular stereo
This lecture carries forward some of the topics from early in the course, namely defocus blur and binocular disparity. The main emphasis here will be on the information these cues carry about depth, rather
More information5LSH0 Advanced Topics Video & Analysis
1 Multiview 3D video / Outline 2 Advanced Topics Multimedia Video (5LSH0), Module 02 3D Geometry, 3D Multiview Video Coding & Rendering Peter H.N. de With, Sveta Zinger & Y. Morvan ( p.h.n.de.with@tue.nl
More information3D Scanning. Qixing Huang Feb. 9 th Slide Credit: Yasutaka Furukawa
3D Scanning Qixing Huang Feb. 9 th 2017 Slide Credit: Yasutaka Furukawa Geometry Reconstruction Pipeline This Lecture Depth Sensing ICP for Pair-wise Alignment Next Lecture Global Alignment Pairwise Multiple
More informationBinocular cues to depth PSY 310 Greg Francis. Lecture 21. Depth perception
Binocular cues to depth PSY 310 Greg Francis Lecture 21 How to find the hidden word. Depth perception You can see depth in static images with just one eye (monocular) Pictorial cues However, motion and
More informationCS5670: Computer Vision
CS5670: Computer Vision Noah Snavely Light & Perception Announcements Quiz on Tuesday Project 3 code due Monday, April 17, by 11:59pm artifact due Wednesday, April 19, by 11:59pm Can we determine shape
More informationLecture 8 Active stereo & Volumetric stereo
Lecture 8 Active stereo & Volumetric stereo In this lecture, we ll first discuss another framework for describing stereo systems called active stereo, and then introduce the problem of volumetric stereo,
More informationNon-line-of-sight imaging
Non-line-of-sight imaging http://graphics.cs.cmu.edu/courses/15-463 15-463, 15-663, 15-862 Computational Photography Fall 2017, Lecture 25 Course announcements Homework 6 will be posted tonight. - Will
More informationComputer Vision I. Announcement. Stereo Vision Outline. Stereo II. CSE252A Lecture 15
Announcement Stereo II CSE252A Lecture 15 HW3 assigned No class on Thursday 12/6 Extra class on Tuesday 12/4 at 6:30PM in WLH Room 2112 Mars Exploratory Rovers: Spirit and Opportunity Stereo Vision Outline
More informationEE795: Computer Vision and Intelligent Systems
EE795: Computer Vision and Intelligent Systems Spring 2012 TTh 17:30-18:45 FDH 204 Lecture 12 130228 http://www.ee.unlv.edu/~b1morris/ecg795/ 2 Outline Review Panoramas, Mosaics, Stitching Two View Geometry
More information