Multidimensional image retargeting

Transcription

1 Multidimensional image retargeting 9:00: Introduction 9:10: Dynamic range retargeting Tone mapping Apparent contrast and brightness enhancement 10:45: Break 11:00: Color retargeting 11:30: LDR to HDR 12:20: Temporal retargeting - Part I 14:15: Temporal retargeting - Part II 15:00: Spatial resolution retargeting 16:00: Break 16:15: Image and video quality assessment 17:00: Stereo content retargeting 17:45: Q&A 18:00: End 12:45: Break Course web page:

2 Stereo content retargeting Piotr Didyk MPI Informatik

3 Why stereo? Images are no longer flat Improves realism Images are not longer flat Better layout separation Reproduced view dependent effects Improves material perception

4 History of stereo 1838: different images for both eyes 1890: patent on 3D movies 1900: tripod for taking 3D pictures 1915: exhibition of 3D images 1922: 3D movie 1923: 3D movie with stereo sound 1952: 3D movie in color 90 s: IMAX cinemas, TV series 2003: feature film in 3D for IMAX now: became very popular

5 # 3D productions Number of 3D productions Short films Feature Films Why didn t get popular? year

6 Early 3D production Expensive hardware Lack of standardized format Impossible at home Lack of interesting content

7 # 3D productions Number of 3D productions Short films Feature Films Now year

8 Stereo in daily life Anaglyph Shutter glasses Autostereoscopic

9 Current 3D production Great content: Beautiful shots with complex depth Computer generated special effects 3D is coming to our homes: Equipment is getting less expensive 3D games / TV New better 3D equipment: Shutter glasses Polarized glasses Autostereoscopic displays are getting better

10 Stereo on a flat display Different image for each eye Left Eye Right eye

11 Depth perception We see depth due to depth cues. Stereoscopic depth cues: binocular disparity

12 Depth perception We see depth due to depth cues. Stereoscopic depth cues: binocular disparity Ocular depth cues: accommodation, vergence Vergence

13 Depth perception We see depth due to depth cues. Stereoscopic depth cues: binocular disparity Ocular depth cues: accommodation, vergence Pictorial depth cues: occlusion, size, shadows

14 Depth contrast Cues sensitivity Personal space Action space Vista space Occlusion 0.01 Relative size 0.1 Relative density Depth [meters] Perceiving layout and knowing distances: The integration, relative potency, and contextual use of different information about depth by Cutting and Vishton [1995]

15 Depth perception We see depth due to depth cues. Stereoscopic depth cues: binocular disparity Ocular depth cues: accommodation, vergence Pictorial depth cues: occlusion, size, shadows Challenge: Consistency is required!

16 Simple conflict example Present cues: Size Shadows Perspective Occlusion

17 Disparity & occlusion conflict Objects in front

18 Disparity & occlusion conflict Disparity & occlusion conflict

19 Depth perception We see depth due to depth cues. Stereoscopic depth cues: binocular disparity Ocular depth cues: accommodation, vergence Pictorial depth cues: occlusion, size, shadows Require 3D space We cheat our Human Visual System! Reproducible on a flat displays

20 Viewing discomfort Cheating our HVS Screen Accommodation (focal plane) Object in right eye Vergence Pixel disparity Depth Object perceived in 3D Comfort zone Object in left eye

21 Viewing discomfort

22 Comfort zones Comfort zone size depends on: Presented content Viewing condition Simple scene m 2 20 m 70 cm Controlling Perceived Depth in Stereoscopic Images by Jones et al. 2001

23 Comfort zones Comfort zone size depends on: Presented content Viewing condition Simple scene, user allowed to look away from screen m m 70 cm Controlling Perceived Depth in Stereoscopic Images by Jones et al. 2001

24 Comfort zones Comfort zone size depends on: Presented content Viewing condition Difficult scene cm 8 15 cm 70 cm Controlling Perceived Depth in Stereoscopic Images by Jones et al. 2001

25 Comfort zones Comfort zone size depends on: Presented content Viewing condition Difficult scene, user allowed to look away from screen 11 cm 6 15 cm 70 cm Controlling Perceived Depth in Stereoscopic Images by Jones et al. 2001

26 Comfort zones Comfort zone size depends on: Presented content Viewing condition Screen distance Other factors: Distance between eyes Depth of field Temporal coherence The zone of comfort: Predicting visual discomfort with stereo displays by Shibata et al. 2011

27 Depth manipulation Comfort zone Viewing Scene manipulation discomfort Viewing comfort

28 Camera manipulations Viewer/Display space N Display L eye R eye E d f d n W F Z Controlling Perceived Depth in Stereoscopic Images by Jones et al. 2001

29 Camera manipulations Camera/Scene space F Virtual display L camera N R camera A d f d n W Z Controlling Perceived Depth in Stereoscopic Images by Jones et al. 2001

30 Camera manipulations Camera/Scene space N Display F Virtual display F L camera N L eye d f d f E d n W R camera A d n W R eye Z Z The parameters can be the same may cause discomfort Z, W, E, d f, d n = Z, W, A, d f, d n Different parameters for capturing the scene Z, W, E, d f, d n Z, W, A, d f, d n Controlling Perceived Depth in Stereoscopic Images by Jones et al. 2001

31 Camera manipulations L eye E Display N Game controller: F d f d n W L camera R camera A N F d f d n W Virtual display R eye Z Z Define the disparity limits OSCAM - Optimized Stereoscopic Camera Control for Interactive 3D by Oscam et al Calculate appropriate camera parameters Adjustment in each frame Controlling Perceived Depth in Stereoscopic Images by Jones et al Evaluating methods for controlling depth perception in stereoscopic cinematography by Sun et al. 2009

32 Camera manipulations General procedure: 1. Define viewing condition 2. Adjust cameras parameters 3. Capturing Displaying on different device: (captured content) Potential discomfort Recapturing?

33 Pixel disparity Zero disparity on the screen plane Bigger disparities in front and behind screen Left + right view

34 Stereo content Left view Right view Can we have pixel disparity / depth?

35 Sources of pixel disparity Stereo image pair Pixel disparity map Rendering Only image pair Usually available Computer vision technique

36 Disparity manipulations Stereo image pair Pixel disparity map Modified pixel disparity Image-based rendering Adjusted stereo pair Adaptive Image-based Stereo View Synthesis by Didyk et al Nonlinear Disparity Mapping for Stereoscopic 3D by Lang et al. 2010

37 Stereoscopy from Light Fields Light Field Scene Left Right Multi-Perspective Stereoscopy from Light Fields by Kim et al. 2011

38 Stereoscopy from Light Fields Stereo image pair Light Field Multi-Perspective Stereoscopy from Light Fields by Kim et al. 2011

39 Camera separation Stereoscopy from Light Fields Multi-Perspective Stereoscopy from Light Fields by Kim et al. 2011

40 Stereoscopy from Light Fields Multi-Perspective Stereoscopy from Light Fields by Kim et al. 2011

41 Disparity manipulations Nonlinear Disparity Mapping for Stereoscopic 3D by Lang et al. 2010

42 Output pixel disparity Disparity manipulations Mapping function Pixel disparity map Input pixel disparity Function: Liner Logarithmic Content dependent Other possibilities: Gradient domain Local operators Modified pixel disparity Nonlinear Disparity Mapping for Stereoscopic 3D by Lang et al. 2010

43 Saliency map Input stereo image Saliency map Disparity importance Disparity mapping function Nonlinear Disparity Mapping for Stereoscopic 3D by Lang et al. 2010

44 Output disparity Saliency map Input disparity Nonlinear Disparity Mapping for Stereoscopic 3D by Lang et al. 2010

45 Scene manipulation

46 Misperception P Z captured object P Z perceived object left image right image X X Parameters are the same Image Distortions in Stereoscopic Video Systems by Woods et al. 1993

47 Misperception P Z captured object P Z left image right image X X Eyes position and interoccular distance changed Image Distortions in Stereoscopic Video Systems by Woods et al. 1993

48 Misperception Eye separation = 65 mm - 5 mm + 5 mm Image Distortions in Stereoscopic Video Systems by Woods et al. 1993

49 Misperception Screen width = 300 mm mm mm Image Distortions in Stereoscopic Video Systems by Woods et al. 1993

50 Misperception Viewing distance = 1 m m + 1 m Image Distortions in Stereoscopic Video Systems by Woods et al. 1993

51 Misperception Misperceptions in Stereoscopic Displays: A Vision Science Perspective by Held et al. 2008

52 3D image prediction

53 Depth perception Stereoscopic depth cues: binocular disparity Ocular depth cues: accommodation, vergence Model Pictorial depth cues: occlusion, size, shadows A perceptual model for disparity by Didyk et al. 2011

54 Disparity perception Depth difference Depth

55 Disparity perception Is it noticeable? How significant is the difference? Depth difference Depth

56 Disparity perception Is it noticeable? α How significant is the difference? β Depth disparity = α β Depth difference

57 One just-noticeable difference Detection threshold Just No noticeable difference (1 JND) Depth

58 How big is the detection threshold? For sinusoidal depth corrugation β α disparity = α β Sensitivity to horizontal and vertical corrugations defined by binocular disparity. by Bradshaw et al. 1999

59 Disparity perception How significant is the difference? Is it noticeable? Depth difference Depth

60 Discrimination threshold depends Just Existing on the previous amplitude noticeable depth difference Depth

61 Disparity perception Sensitivity to depth changes depends on: Spatial frequency of disparity corrugation Existing disparity (sinusoid amplitude)

62 Threshold Measurements Parameter space: Sinusoid in depth 1. Sample the space 3. Measure thresholds Experiment with adjustment task A perceptual model for disparity by Didyk et al. 2011

63 Measurements Thresholds measurement: Two sinusoidal corrugations Which has more depth? (left/right) Amplitude adjustment (PEST with 2AFC) 12 participants 300+ samples A perceptual model for disparity by Didyk et al. 2011

64 Threshold [arcmin] Model 3. Fit analytic function Frequency [cpd] Amplitude [arcmin]

65 Threshold [arcmin] Threshold [arcmin] The HVS response Disparity sensitivity The HVS response? Frequency [cpd] Amplitude (disparity)[arcmin] Disparity [arcmin]

66 Threshold [arcmin] The HVS response Disparity sensitivity Disparity [arcmin] Depth

67 Threshold [arcmin] The HVS response Disparity sensitivity 1 JND 1 JND 1 JND 1 JND 4 JND How significant is the difference? Disparity [arcmin]

68 Threshold [arcmin] Response [JND] The HVS response Transducers Frequency [cpd] Amplitude [arcmin] Disparity [arcmin] A transducer function for threshold and suprathreshold human vision by Wilson 1980 A perceptual framework for contrast processing of high dynamic range images by Mantiuk et al. 2005

69 Perceptual space Reality is more complex: We 10px show so far: strong (disparity, frequency) [arcmin, cpd] -10px 3D scene with pixel disparity [pixels] HVS response [JND] Map of HVS response [JND] weak

70 Perceptual space Our problem: Problems: 10px strong Pixel disparity [pixels] Disparity [arcmin] Sinusoidal patterns Complex images -10px 3D scene with pixel disparity [pixels] Map of HVS response [JND] weak

71 Pixel disparity to disparity β α disparity = α β

72 Pixel disparity to disparity Interaxial Screen distance β Pixel disparity α Pixel disparity viewing conditions, pixel disparity vergence

73 Vergence to disparity Disparity [arcmin] How do people deal with luminance? Vergence [arcmin]

74 Luminance (contrast perception) Perceptual space (Perceived contrast) Luminance

75 Luminance (contrast perception) Works because: Disparity / Luminance similarity: Different frequencies are processed separately. Luminance Vergence For disparity is similar. Luminance contrast Disparity Disparity is processed in independent channels. Seeing in depth by Howard and Rogers 2002 Lowpass filters Perceptual Contrast decomposed operations into frequency bands

76 Vergence to disparity Vergence [arcmin] Lowpass filters Differences

77 Vergence to disparity Lowpass filters Differences We can process frequencies independently Vergence Disparity

78 Perceptual model Perceptual space A perceptual model for disparity by Didyk et al. 2011

79 Disparity metric

80 Disparity metric strong Original pixel disparity For Luminance: Original vergence Modified pixel disparity Modified vergence A visual discrimination model for imaging system design and development by Lubin 1995 Difference map weak A perceptual model for disparity by Didyk et al. 2011

81 Disparity manipulations Manipulations in perceptual space: The HVS is taken into account Efficient disparity reduction Important disparities preserved Nonlinear Disparity Mapping for Stereoscopic 3D by Lang et al. 2010

82 Disparity manipulation Disparity manipulations A perceptual model for disparity by Didyk et al. 2011

83 Disparity manipulation Disparity manipulations A perceptual model for disparity by Didyk et al. 2011

84 Inverse model Invertible Perceptual space

85 Inverse model Invertible Perceptual space

86 Disparity manipulation Disparity manipulations

87 Disparity manipulation Standard technique In perceptual space strong weak Perceived distortions Perceived distortions Important disparities preservation

88 Personalization Disparity perception depends on: Equipment User

89 Personalization Original disparity Adjusted disparity Perceptual space A perceptual model for disparity by Didyk et al. 2011

90 Personalization = All users perceive the same regardless: Equipment Disparity sensitivity

91 Backward-compatible stereo Backward-compatible Standard 2D stereo image stereo

92 Cornsweet illusion Similar perceived contrast Luminance range reduced Cornsweet illusion works for depth: A Craik-O'Brien-Cornsweet illusion for visual depth by Anstis et al. 1997

93 Backward-compatible stereo Standard stereo Backward-compatible stereo 3D impression preserved No artifacts when special equipment is unavailable A perceptual model for disparity by Didyk et al. 2011

94 Backward-compatible stereo 3D impression preserved No artifacts when special equipment is unavailable A perceptual model for disparity by Didyk et al. 2011

95 Conclusions Stereo perception is complex phenomena Important aspects: Viewing conditions Viewer Equipment Temporal coherence Different adjustment techniques: Camera adjustment Pixel disparity mapping operators Perceptual space

96 Multidimensional image retargeting 9:00: Introduction 9:10: Dynamic range retargeting Tone mapping Apparent contrast and brightness enhancement 10:45: Break 11:00: Color retargeting 11:30: LDR to HDR 12:20: Temporal retargeting - Part I 14:15: Temporal retargeting - Part II 15:00: Spatial resolution retargeting 16:00: Break 16:15: Image and video quality assessment 17:00: Stereo content retargeting 17:45: Q&A 18:00: End 12:45: Break Course web page: