Resolution. Super-Resolution Imaging. Problem

Transcription

1 Resolution Super-Resolution Iaging Resolution: Sallest easurable detail in a visual presentation Subhasis Chaudhuri Departent of Electrical Engineering Indian institute of Technology Bobay Powai, Mubai INDIA Spatial Resolution: spacing of pixels in an iage easured in pixels per inch (ppi) High Spatial Resolution: Pixel density is high. (Larger no of pixels in an iage) HR Applications : Medical Iaging, Satellite iaging, reote sensing etc Why SR? 1. Cost 2. Shot noise Proble Super-Resolution (SR): Obtain high resolution fro several low resolution observations of the sae scene. ( iniizes aliasing and blurring). Conventional Interpolation Methods : Nearest Neighbor or zero order hold or pixel replica, Bilinear, Bicubic Disadvantage : Single iage used. Do not consider the aliasing or blurring.

2 The Idea! Illustration of Effects SR Model Illustration

3 HR LR Transforation SR Restoration LR HR Transforation Aliasing Exaple

4 HR Restoration Different types of Cues to Solve Super-resolution Proble Motion Blur Zoo Photoetry Learning based techniques Proble Definition Y 1 Y 2 Y 3 Y 1 : Least zooed iage SR USING ZOOM CUE q 1 q 2 q 2 Z Y 3 : Most zooed iage q : Zoo factor Proble : Given Y 1, Y 2,,Y p ( Y p has highest resolution), obtain the resolution of Y 1 at resolution of Y p Assuption ade : Zoo factors are known. noise is i.i.d gaussian

5 Motivation An application of SR. Model for low resolution iage Observation odel : y = D C ( z z ) + n = 1... p n 1 (k,l) Uses zoo as a cue. Resolution enhanceent in reote sensing data. Z(k,l) View Cropping C 2 Zoo out q 1 q 2 Zoo out q 2 y 1 (k,l) n 2 (k,l) y 2 (k,l) n 3 (k,l) View Cropping C 3 y 3 (k,l) Solution Markov Rando Field ( MRF) Use MAP a axiu a posteriori estiation z ˆ = arg ax p( z / y, y,..., y 1 2 p ) Models the a priori probability of context dependent entities : iage pixels, depth etc. z Assue to be an MRF (Markov Rando field) Cost to be iniized Regularizes the solution. Reconstruction sooth. ε = λ p = 1 y 2 D C ( z z ) + c c C V ( z) Include line fields.

6 Experiental Results Experiental Results (contd.) Observed iages of Nidhi captured with 3 different zoo settings. Observed iages of a house captured with 3 different zoo settings. Zooed Nidhi iage fored by using successive zero-order hold expansion. Super-Resolved Nidhi iage. Zooed house iage fored by using successive zero-order hold expansion. Super-Resolved house iage. Experients with Zoo Estiation SR with estiated zoo (a) (b) (c) Observed iages of Nidhi captured captured with three different Unknown zoo settings. Zoo factor = 1.72 Estiated Iage obtained by aligning (b) and (c) Zooed Nidhi iage fored by using successive Bicubic expansion. Super-resolved iage

7 SR with MRF Paraeters Estiation Based on Maxiu likelihood estiation LEARNING OF PRIORS FROM ZOOMED OBSERVATIONS θˆ = arg ax P( Z = z θ ) θ θ T = [ β1, β 2 ] for first order neighborhood θ = [ β β β β ] T 1, 2, 3, 4 for second order neighborhood Assuption ade : Entire scene is statistically hoogeneous SR with SAR (Siultaneous auto regressive) paraeters estiation Experients with MRF paraeters Estiation Model: z( s) = θ ( r) z( s + r) + w( s) rε D Where D is the set of neighbors of pixel at site s w (.) is i. i. d noise sequence with zero ean 2 and variance σ θ ( r) = θ ( r) Observed iages of texture with three different zoo settings MRF paraeters learnt fro ost zooed observation using Maxiu pseudolikelihood (MPL) estiation Yu and Cheng PRL,2003

8 SR with MRF Paraeters estiated (cont.) SR with SAR Paraeters estiated Bilinear Interpolation Proposed Approach Super-resolved iage SAR paraeters learnt fro ost zooed observation using axiu likelihood (ML) criterion Kashyap and Chellappa IEEE IT, 1983 An Experient with a zoo factor of 2 Experients with MRF and SAR paraeters Estiation Observed iages of a texture with two different zoo settings Bilinear Interpolation Proposed MRF based Approach

9 Experients with paraeters Estiation (contd.) Observed iages of texture with three different zoo settings Proposed SAR based Approach SR with MRF Paraeters Estiated Experient with SAR paraeters Estiation Bilinear Interpolation Proposed Approach Super-resolved iage

10 Experients with paraeters Estiation (contd.) SR with MRF Paraeters Estiated Observed iages of flower captured with three different unknown zoo settings. Bilinear interpolation Super-resolved iage Conclusion The super-resolved iage is odeled as MRF or as an SAR and an MAP estiate and regularization based approach are used to solve the proble. Reconstruction not very good at zooed out portions as expected. SR IMAGE AND STRUCTURE: USE OF PHOTOMETRIC CUE

11 Proble Definition: Given the photoetric easureents obtain the super-resolved iage as well as dense depth ap. Advantage: No need for iage registration. 3D shape preservation is used as constraint. Model Used: y= FC( D, z, R) + n, = 1,, p Assuptions ade : Light Source directions are known Reflectance odel is known Illustration of SR using Photoetric Cue Regularization 1. MRF-based Approach Using Photoetric Stereo z = ρ ( x r, y r) nˆ( x r, y r). sˆ p, q, and albedo are odeled as separate MRF s Integrability constraint iposed Data consistency check included

12 Experiental Results Final cost iniized : ε = Reflectance odel error + Integrability constraint + Soothness priors + Data consistency error ε = z ρ( x r, y r) nˆ( x r, y r).ˆ s + λ ( p y q )(1 l )(1 v ) x + U ( z) + U ( p) + U ( q) + U ( ρ) p q 2 2 An observed iage of dog Jodu with the light source position ( , ) + α y HDz 2 Bilinearly Interpolated Jodu Super-resolved Jodu using generalized interpolation Experiental Results (contd.) Experiental Results (contd.) Super-resolved Jodu using the MRF-based approach Super-resolved, synthesized view of Jodu for source position (0.0, 0.0) for which the observation is not captured. SR depth ap of Jodu using the Generalized interpolation SR depth ap obtained using the MRF-based approach

13 Experiental Results (contd.) Experiental Results (contd.) An observed shoe iage captured with light source position (0.4663, ) Super-resolved shoe iage using generalized interpolation Super-resolved shoe iage using MRF-based approach Super-resolved depth ap of shoe iage using the generalized interpolation Super-resolved depth ap using the MRF-based approach 2. Variational Approach Results Conclusions The super-resolved iage z, surface gradients p, q and the albedo are odeled as separate MRFs and the regularization based technique is used to solve the proble. Proble also solved using variational approach where soothness of the function is used as regularization ter : Faster Coputations. SR depth ap of Jodu using the variational approach SR depth ap of shoe iage using the variational approach

14 JOINT BLIND RESTORATION AND SURFACE RECOVERY IN PHOTOMETRIC STEREO Proble: Siultaneous estiation of scene structure and restoration of iages fro blurred observations captured under different light source positions keeping both caera and object stationary. Model Used: g = H ( σ ) f ( p, q, ρ) + n, = 1,..., k A restoration and shape recovery proble Solution Illustration of Proposed Method (PSF Unknown) Cost = odel consistency ter for p observations + soothness ters iniize with respect to surface gradients and albedo.

15 Experiental Results (Known blur) Experiental Results (cont.) Focused Jodu iage for two different source positions Blurred (pill-box) Jodu iage for the sae source positions Experiental Results (cont.) Experiental Results (cont.) Estiated Jodu for the sae source positions True depth ap Depth ap due to blurred observations using photoetric stereo Estiated depth ap using proposed approach

16 Experiental Results (Unknown Blur) Experiental Results (cont.) Observed iage of Jodu with an arbitrary caera defocus for a particular light source position Restored Jodu using proposed approach Blind Deconvolution Estiated blur paraeter = Recovered depth esh plot using photoetric stereo Recovered depth using proposed approach Conclusions A blind restoration and structure recovery proble is addressed using photoetric easureents. Results obtained show perceptual as well as quantifiable iproveents over standard PS (photoetric stereo), Lucy-Richardson algorith for restoration and blind deconvolution. USE OF LEARNT WAVELET PRIOR

17 Proble Definition Approach Given a low resolution iage and a set of high resolution training iages learn the high frequencies fro the training data set and obtain SR. Learn the wavelet coefficients at finer scales of the unknown high resolution iage fro high resolution training set. Final cost used for optiization: data fitting ter + wavelet prior + soothness prior Learning: Experiental Results Low resolution iage Bilinearly interpolated iage Super-resolved iage

18 Experiental Results (cont.) Conclusions Low resolution iage A learning based technique for super-resolution using a single low resolution iage is described. Learning represents the next challenging frontier for coputer vision. Bilinear interpolation Super-resolved iage