A comparative review of tone-mapping algorithms for high dynamic range video

Transcription

1 A comparative review of tone-mapping algorithms for high dynamic range video A comparative review of tone-mapping algorithms for high dynamic range video State of the art report Gabriel Eilertsen, Rafał Mantiuk, Jonas Unger Linköping University, Sweden University of Cambridge, UK

2 Overview Introduction (15 min) HDR video (capturing, displays, distribution) Tone-mapping (basic algorithms) Historical overview Video tone-mapping (40 min) Introduction Categorization Operators Video specific tone-mapping problems Objective evaluation (20 min) Measures Results Conclusions/summary (5 min) 2

3 Introduction

4 HDR vs. LDR LDR 8-bit integer representation Screen referred Gamma corrected Display model: L d (l) =L (L max L black )+L black + L refl. HDR Floating point Can encode entire range of visible luminances HDR (scene-referred) TMO L 1 d Scene referred (linear ) Used for e.g. IBL, automotive applications, medical imaging LDR (display-referred) Tone-mapping for transformation between the formats Intermediate format 4

5 HDR capturing Static images Physically based rendering Exposure bracketing Video Temporal multiplexing Multiple cameras Multiple sensors (20-24 f-stops) Professional one-sensor cameras ( f-stops) RED, ARRI TOCCI M. D., KISER C., TOCCI N., SEN P.: A versatile hdr video production system. ACM Trans. Graphics 30, 4 (2011), 41:1 41:10. KRONANDER J., GUSTAVSON S., BONNET G., YNNERMAN A., UNGER J.: A unified framework for multi-sensor hdr video recon- struction. Signal Processing : Image Communications 29, 2 (2014),

6 HDR displays Standard LCD display cd/m 2 peak Dual modulation Back-lit by low-res image cd/m 2 peak Brightside DR37-P, Dolby pulsar, Sim2 HDR47 HDR TVs ~1000 cd/m 2 peak Tone-mapping still essential! SEETZEN H., HEIDRICH W., STUERZLINGER W., WARD G., WHITEHEAD L., TRENTACOSTE M., GHOSH A., VOROZCOVS A.: High dynamic range display systems. ACM Trans. Graph. 23, 3 (Aug. 2004),

7 HDR distribution Rapidly increasing interest from TV industry Standardization initiatives PQ transform (SMPTE ST-2084), Hybrid Log-Gamma ITU-R Rec. BT.2100 MPEG HDR10, Dolby vision Input Color transformation and chromatic downsampling Luminance transformation Quantization Video encoding HDR video RGB 4:4:4 Storage Distribution Output HDR video RGB 4:4:4 Inverse color transformation and chromatic upsampling Inverse luminance transformation Conversion to floating point Video decoding MANTIUK R., KRAWCZYK G., MYSZKOWSKI K., SEIDEL H.: Perception-motivated high dynamic range video encoding. ACM Transactions on Graphics (TOG) (2004). 7

8 Luma HDRv Open source (BSD license) Library C++ API for encoding and decoding of HDR video Applications Encoding of HDR video frames Decoding of HDR video to separate frames Playback of HDR video Dependencies Matroska media container VP9 (libvpx) from Google EILERTSEN G., MANTIUK R., UNGER J.: A high dynamic range video codec optimized by large-scale testing. In Proceedings of IEEE International Conference on Image Processing (ICIP) (2016). 8

9 Tone-mapping principles Tone-curve Scaling Logarithmic/exponential Sigmoid Histogram based Other Processing Global - constant tone-curve Local - spatially varying tone-curve Intent Visual system simulators (VSS) Scene reproduction operators (SRP) Best subjective quality operators (BSQ) 9

10 Tone-mapping intents HDR Scenereferred Tonemapping BSQ Boosted contrast, sharpness, details, color saturation etc. SRP Compress and clip color gamut and dynamic range VSS Glare, loss of acuity, color perception in low light LDR Displayreferred Evaluation Subjective metric Fidelity metric Perceptual metric 10

11 HDR input Tone mapping Edge-pres. filter Base layer Tone-curve Edge pres. filter: Bilateral filter Tone-curve: Sigmoid S-curve Detail layer

12 The tone-mapping pipeline 12

13 The tone-mapping pipeline 13

14 The tone-mapping pipeline 14

15 The tone-mapping pipeline 15

16 The tone-mapping pipeline 16

17 The first work 60s Dynamic range reduction with local considerations 80s OPPENHEIM A., SCHAFER R., STOCKHAM T.: Nonlinear filtering of multiplied and convolved signals. IEEE Transactions on Audio and Electroacoustics 16, 3 (Sep 1968), Matching of perceptual attributes Lighting design 90s MILLER N. J., NGAI P. Y., MILLER D. D.: The application of computer graphics in lighting design. Journal of the Illuminating Engineering Society 14, 1 (1984), UPSTILL S. D.: The Realistic Presentation of Synthetic Images: Image Processing in Computer Graphics. PhD thesis, AAI Tone-mapping of computer generated images The first tone-mapping operator TUMBLIN J., RUSHMEIER H.: Tone reproduction for realistic images. Computer Graphics and Applications, IEEE 13, 6 (1993),

18 Evolution of tone-mapping Occurrences of the terms in the literature within Google Books, according to the Google NGram viewer tone mapping high dynamic range Occurrences Year Audio tone-mapping HDR tone-mapping 18

19 Video tone-mapping

20 Video tone-mapping Introduction Difference to tone-mapping of static images Evaluation experiment from 2013 Categorization Operators Description of video TMOs Examples Video specific tone-mapping problems Temporal coherence Noise visibility 20

21 Video tone-mapping Introduction: differences compared to tone-mapping of static images

22 Video tone-mapping Tone-mapping well-explored Many algorithms, for a wide range of purposes Temporally adaptive operators have been around for 2 decades High-quality camera captured HDR video Dedicated camera systems for HDR video Appeared about 5 years ago Different properties compared to static multi-exposure FERWERDA J. A., PATTANAIK S. N., SHIRLEY P., GREEN- BERG D. P.: A model of visual adaptation for realistic image synthesis. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques (New York, NY, USA, 1996), SIGGRAPH 96, ACM, pp PATTANAIK S. N., TUMBLIN J., YEE H., GREENBERG D. P.: Time-dependent visual adaptation for fast realistic image display. In Proc. SIGGRAPH 00 (2000), Annual Conference Series, pp DURAND F., DORSEY J.: Rendering Techniques 2000: Proceedings of the Eurographics Workshop in Brno, Czech Republic, June 26 28, Springer Vienna, Vienna, 2000, ch. Interactive Tone Mapping, pp

23 Video tone-mapping The new challenges in tone-mapping of camera captured HDR video: Complicated transitions, both spatially and over time Camera noise Skin tones Computational complexity Memory consumption Parameter tuning 23

24 Evaluation of TMOs for HDR video 11 video TMOs (with explicit model for temporal adaptation) HDR capture (or rendering) Tone mapping Edi ng User guided enhancements (color grading etc.) Tone mapping Categorization from intent Physical scene (or 3D model) LDR display HDR display Visual system simulators (VSS) Scene reproduction operators (SRP) Best subjective quality operators (BSQ) Qualitative evaluation Subjective pair-wise comparison experiment Measure HVS observa on Fidelity metric Appearance metric Scene could be recalled from memory, and not actually observed Measure Memorized scene Non-reference metric Fidelity metric Media Transforma on Comparison EILERTSEN, G., WANAT, R., MANTIUK, R. K., UNGER, J. (2013): Evaluation of tone mapping operators for HDR-video. Computer Graphics Forum 32, 7,

25 Qualitative evaluation 25

26 Qualitative evaluation Unacceptable Virtual exposures TMO Retina model TMO Very visible Visible Barely visible Invisible Consistency Flickering Ghosting Noise Consistency Flickering Ghosting Noise Unacceptable Very visible Visible Local adaptation TMO Color apperance TMO Hallway Hallway 2 Exhibition area Driving simulator Students Window Barely visible Invisible Consistency Flickering Ghosting Noise Consistency Flickering Ghosting Noise 26

27 Subjective evaluation Quality [JND] tone-mapping operators 5 HDR video sequences Pairwise, non-reference, comparisons Visual adaptation TMO Cone model TMO Mal adaptation TMO Display adaptive TMO Time adaptation TMO Temporal coherence TMO Camera TMO 2 1 Generally global operators favored over local ones 0 Hallway Exhibition area Driving simulator Students Window 27

28 HDR video tone-mapping, requirements 1. Temporal model free from artifacts such as flickering, ghosting and disturbing (too noticeable) temporal color changes. 2. Local processing to achieve sufficient dynamic range compression in all circumstances while maintaining a good level of detail and contrast. 3. Efficient algorithms, since large amount of data need processing, and turnaround times should be kept as short as possible. 4. Low need for parameter tuning. 5. Calibration of input data should be kept to a minimum, e.g. without the need of considering scaling of data. 6. Capability of generating high quality results for a wide range of video inputs with highly different characteristics. 7. Explicit treatment of noise and color. 28

29 Video tone-mapping Categorization of video tone-mapping operators used in the STAR report

30 Video TMO categorization 1. Tone-curve Transformation function used for tone-mapping Linear, logarithmic/exponential, sigmoid, histogram based 2. Processing Global: Spatially uniform tone-curve Local: Spatially varying transformation 3. Intent Visual system simulator (VSS), best subjective quality(bsq), scene reproduction (SRP) 4. Temporal filter Global: filtering of global tone-mapping parameters Local: filtering of local parameters or pixel values 5. Speed Offline, interactive, real-time 30

31 Video tone-mapping Existing methods that include an explicit model for temporal adaptation

32 Operator Categorization Estimated performance Tone-curve Intent Processing Temporal filter Speed Temporal coherence Contrast Exposure Noise treatment Visual adaptation TMO [FPSG96] Linear VSS Global Global Real-time Interactive walk-through TMO [DD00] Linear VSS Local Global Real-time Time-adaptation TMO [PTYG00] Sigmoid VSS Global Global Real-time Temporal exposure TMO [KUWS03] Sigmoid BSQ Local Global Interactive Time-dependent GPU TMO [GWWH03] Sigmoid BSQ Local Global Real-time Adaptive temporal TMO [RJIH04] Sigmoid BSQ Global Global Interactive Local adaptation TMO [LSC04] Sigmoid VSS Local Local 1 Offline Mal-adaptation TMO [IFM05] Hist. based 2 VSS Global Global Interactive Perceptual effects TMO [KMS05b] Sigmoid VSS Local Global Real-time Gradient domain TMO [WRA05] Grad. domain 3 BSQ Local Local 4 Offline Virtual exposures TMO [BM05] Logarithmic BSQ Local Local 5 Offline Cone model TMO [vh06] Sigmoid VSS Global Local 1 Offline Block matching TMO [LK07] Grad. domain 3 BSQ Local Local 4 Offline Display adaptive TMO [MDK08] Hist. based 2 SRP Global Global Offline Retina model TMO [BAHC09] Sigmoid VSS Local Local 1 Offline Local mal-adaptation TMO [PCA 10] Sigmoid VSS Local Local Real-time Flicker reduction TMO [GKEE11] 6 SRP 6 Global 7 Offline Real-time automatic TMO [KRTT12] Sigmoid BSQ Global Global Real-time Color appearance TMO [RPK 12] Sigmoid SRP Local Global Offline Temporal coherence TMO [BBC 12] 6 SRP 6 Global 7 Offline Zonal coherence TMO [BCTB14b] 6 SRP Local Local 7 Offline Motion path filtering TMO [ASC 14] Logarithmic BSQ Local Local 8 Offline Hybrid TMO [SLF 15] Sigm./hist. based BSQ Local Global Real-time Noise-aware TMO [EMU15] Hist. based 2 SRP Local Global Real-time Noisy log sensor TMO [LSRJ16] Hist. based 2 SRP Global Global Real-time K-means clustering TMO [Osk16] Hist. based 2 SRP Global Global Interactive 1 Per-pixel low-pass filter 2 Content-adaptive tone-curve formulated using the image histogram 3 Tone-curve applied in the gradient domain, based on the gradient magnitudes 4 3D Poisson solver 5 Per-pixel bilateral filter 6 Applicable to any TMO, alleviating global temporal incoherence 7 Scaling in a post-processing step 8 Per-pixel filtering along motion paths 32

33 Video TMOs Visual adaptation TMO Time-adaptation TMO Local adaptation TMO Mal-adaptation TMO Virtual exposures TMO Cone model TMO Display adaptive TMO Retina model TMO Color appearance TMO Temporal coherence TMO Zonal coherence TMO Motion path filtering TMO Noise-aware TMO 33

34 Video TMOs Visual adaptation TMO Time-adaptation TMO Local adaptation TMO Mal-adaptation TMO Tone-curve Intent Processing Temp. filter Speed Linear VSS Global Global Real-time Visual response model built from psycho-physical experimental data. Predicts the behavior of the HVS in terms of visibility, color appearance and clarity. Accounts for visual acuity and time-adaptation. Virtual exposures TMO Cone model TMO Display adaptive TMO Retina model TMO Color appearance TMO Temporal coherence TMO Zonal coherence TMO Motion path filtering TMO Noise-aware TMO FERWERDA J. A., PATTANAIK S. N., SHIRLEY P., GREENBERG D. P.: A model of visual adaptation for realistic image synthesis. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques (New York, NY, USA, 1996), SIGGRAPH 96, ACM, pp

35 Video TMOs Visual adaptation TMO Time-adaptation TMO Local adaptation TMO Mal-adaptation TMO Tone-curve Intent Processing Temp. filter Speed Sigmoid VSS Global Global Real-time Modelling of time-adaptation mechanisms of the HVS. Forward and inverse adaptation and appearance models. Time adaptation with exponential smoothing fitted to experimental data. Virtual exposures TMO Cone model TMO Display adaptive TMO Retina model TMO Color appearance TMO Temporal coherence TMO Zonal coherence TMO Motion path filtering TMO Noise-aware TMO PATTANAIK S. N., TUMBLIN J., YEE H., GREENBERG D. P.: Timedependent visual adaptation for fast realistic image display. In Proc. SIGGRAPH 00 (2000), Annual Conference Series, pp

36 Video TMOs Visual adaptation TMO Time-adaptation TMO Local adaptation TMO Mal-adaptation TMO Tone-curve Intent Processing Temp. filter Speed Sigmoid VSS Local Local Offline Separate responses for each pixel, based on experimental data. Bilateral filter used for deriving local adaptation levels. Adaptation levels filtered over time using exponential filters. Virtual exposures TMO Cone model TMO Display adaptive TMO Retina model TMO Color appearance TMO Temporal coherence TMO Zonal coherence TMO Motion path filtering TMO Noise-aware TMO LEDDA P., SANTOS L. P., CHALMERS A.: A local model of eye adaptation for high dynamic range images. In Proc. International Conference on Computer Graphics, Virtual Reality, Visualisation and Interaction in Africa 3 (2004), pp

37 Video TMOs Visual adaptation TMO Time-adaptation TMO Local adaptation TMO Mal-adaptation TMO Tone-curve Intent Processing Temp. filter Speed Hist. based VSS Global Global Interactive Includes the state of mal-adaptation of the HVS. Time model accounts for neural mechanisms and photo-pigment bleaching. Temporal adaptation as in the Time-adaptation TMO. Virtual exposures TMO Cone model TMO Display adaptive TMO Retina model TMO Color appearance TMO Temporal coherence TMO Zonal coherence TMO Motion path filtering TMO Noise-aware TMO IRAWAN P., FERWERDA J. A., MARSCHNER S. R.: Perceptually based tone mapping of high dynamic range image streams. In Proc. Eurographics Conference on Rendering Techniques 16 (2005), pp

38 Video TMOs Visual adaptation TMO Time-adaptation TMO Local adaptation TMO Mal-adaptation TMO Tone-curve Intent Processing Temp. filter Speed Logarithmic BSQ Local Local Offline Temporal filtering approach for denoising. Base layer transformed using logarithmic tone-curve. Bilateral filter both spatially and perpixel over time. Virtual exposures TMO Cone model TMO Display adaptive TMO Retina model TMO Color appearance TMO Temporal coherence TMO Zonal coherence TMO Motion path filtering TMO Noise-aware TMO BENNETT E. P., MCMILLAN L.: Video enhancement using per- pixel virtual exposures. ACM Trans. Graphics 24, 3 (2005),

39 Video TMOs Visual adaptation TMO Time-adaptation TMO Local adaptation TMO Mal-adaptation TMO Tone-curve Intent Processing Temp. filter Speed Sigmoid VSS Global Local Offline Uses actual quantitative measurements on the cells in the macaque retina. Used in a dynamic model of primate cones and horizontal cells. Temporal per-pixel low-pass filtering. Virtual exposures TMO Cone model TMO Display adaptive TMO Retina model TMO Color appearance TMO Temporal coherence TMO Zonal coherence TMO Motion path filtering TMO Noise-aware TMO VAN HATEREN J. H.: Encoding of high dynamic range video with a model of human cones. ACM Trans. Graphics 25 (2006),

40 Video TMOs Visual adaptation TMO Time-adaptation TMO Local adaptation TMO Mal-adaptation TMO Tone-curve Intent Processing Temp. filter Speed Hist. based SRP Global Global Offline Optimization to maximize simlilarity between HDR and tone-mapped image viewed on a specific display. Display model used to model emitted display luminance. Tone-curve low-pass filtered over time. Virtual exposures TMO Cone model TMO Display adaptive TMO Retina model TMO Color appearance TMO Temporal coherence TMO Zonal coherence TMO Motion path filtering TMO Noise-aware TMO MANTIUK R., DALY S., KEROFSKY L.: Display adaptive tone mapping. ACM Trans. Graphics 27, 3 (2008), 68:1 68:10. 40

41 Video TMOs Visual adaptation TMO Time-adaptation TMO Local adaptation TMO Mal-adaptation TMO Tone-curve Intent Processing Temp. filter Speed Sigmoid VSS Local Local Offline Model of the early stages of the HVS. Spatio-temporal local filtering of the retina. Color processing by mosaicing/demosaicing before/after tone-mapping. Virtual exposures TMO Cone model TMO Display adaptive TMO Retina model TMO Color appearance TMO Temporal coherence TMO Zonal coherence TMO Motion path filtering TMO Noise-aware TMO BENOIT A., ALLEYSSON D., HERAULT J., CALLET P. L.: Spatio-temporal tone mapping operator based on a retina model. In Computational Color Imaging Workshop (CCIW 09) (2009), pp

42 Video TMOs Visual adaptation TMO Time-adaptation TMO Local adaptation TMO Mal-adaptation TMO Tone-curve Intent Processing Temp. filter Speed Sigmoid SRP Local Global Offline Appearance calibration algorithm. Matches image appearance in terms of photo-receptor responses, including local aspects of lightness perception. Leaky integration of adaptation parameters over time. Virtual exposures TMO Cone model TMO Display adaptive TMO Retina model TMO Color appearance TMO Temporal coherence TMO Zonal coherence TMO Motion path filtering TMO Noise-aware TMO REINHARD E., POULI T., KUNKEL T., LONG B., BALLESTAD A., DAMBERG G.: Calibrated image appearance reproduction. ACM Trans. on Graph. 31, 6 (November 2012). 42

43 Video TMOs Visual adaptation TMO Time-adaptation TMO Local adaptation TMO Mal-adaptation TMO Tone-curve Intent Processing Temp. filter Speed - SRP - Global Offline Post-processing method, considering the entire tone-mapped sequence. Key values of each frame anchored to the maximum key value of the sequence, using a per-frame global scaling. Virtual exposures TMO Cone model TMO Display adaptive TMO Retina model TMO Color appearance TMO Temporal coherence TMO Zonal coherence TMO Motion path filtering TMO Noise-aware TMO BOITARD R., BOUATOUCH K., COZOT R., THOREAU D., GRUSON A.: Temporal coherency for video tone mapping. In Proc. SPIE 8499, Applications of Digital Image Processing XXXV (San Diego, 2012), pp D 84990D

44 Video TMOs Visual adaptation TMO Time-adaptation TMO Local adaptation TMO Mal-adaptation TMO Virtual exposures TMO Cone model TMO Recent STAR methods Dedicated HDR video algorithms Focus on problems inherent to camera captured HDR video Attempts to overcome specific problems with the previous methods Display adaptive TMO Retina model TMO Color appearance TMO Temporal coherence TMO Zonal coherence TMO Motion path filtering TMO Noise-aware TMO 44

45 Zonal temporal coherency TMO Enforcing global temporal coherency Anchoring of key to maximum key of sequence Considers all frames of the sequence Attempt to prevent flickering AND brightness incoherencies Applies to any TMO apple HDR F apple HDR F,max = appleldr F apple LDR F,max, L 0 applehdr P d = L d apple LDR P,max apple HDR P,max appleldr P 0 BOITARD R., BOUATOUCH K., COZOT R., THOREAU D., GRUSON A.: Temporal coherency for video tone mapping. In Proc. SPIE 8499, Applications of Digital Image Processing XXXV (San Diego, 2012), pp D 84990D 10. BOITARD R., COZOT R., THOREAU D., BOUATOUCH K.: Zonal brightness coherency for video tone mapping. Signal Processing: Image Communication 29, 2 (2014),

46 Zonal temporal coherency TMO Global temporal coherency TMO has problems with contrasts One key value per zone Zones from histogram segmentation (a) Segmented key value histogram (b) Zone s boundary and key value BOITARD R., COZOT R., THOREAU D., BOUATOUCH K.: Zonal brightness coherency for video tone mapping. Signal Processing: Image Communication 29, 2 (2014),

47 Zonal temporal coherency TMO 47

48 ally maintain local contrast) are temporally unstable [Eilertsen et al. 2013]. Similarly, na ıvely applying image TMOs to individual video frames has been found to produce strong temporal artifacts [Eilertsen et al. 2013; Boitard et al. 2014a]. Global TMOs with relatively simple processing steps are found to be less susceptible to temporal artifacts, however they lack the contrast reproduction capabilities of local operators. The temporal instability of local TMOs underlines the necessity of temporal filtering for local video tone mapping. Motion path filtering TMO Base/detail separation Permeability filter Spatio-temporal filtering through motion paths (optical flow) Temporal coherency Noise reduction Standard tone-curve INPUT HDR Frame Spatiotemporal Temporal Filtering Filtering Base Layer Forward Optical Flow Backward Optical Flow Tone Mapping Detail Layer Tone Mapped Frame OUTPUT Figure 3: Major processing steps of our method. Refer to the text for details. AYDIN T. O., STEFANOSKI N., CROCI S., GROSS M., SMOLIC A.: Temporally Since filtering in the coherent local tone mapping of HDR video. ACM Trans. Graphics 33, 6 (2014), temporal domain is notably more challenging than in the spatial domain, one can assume that the scene is static and filter through a straight line in the temporal dimension [Bennett and48 McMillan 2005]. However this approach generates strong

49 Motion path filtering TMO 49

50 Real-time noise-aware TMO Aimed at addressing the most important challenges, as recognized from the previous evaluation Tone-curve Minimum contrast distortion Local tone-curves Temporal filter (low-pass IIR, edge-stop) Detail preservation/enhancement Fast detail extraction diffusion Temporally consistent Noise-awareness Display adaptation Real-time implementation EILERTSEN G., MANTIUK R. K., UNGER J.: Real-time noise- aware tone mapping. ACM Trans. Graph. 34, 6 (Oct. 2015), 198:1 198:15. 50

51 HDR input Tone mapping Edge-pres. filter (fast detail extraction diffusion) Base layer Detail layer Display adaptation (inverse display model) Local tone-curves Noise-awareness (noise-aware minimum contrast distortion) (scaling of detail layer) Noise model 51

52 Real-time noise-aware TMO 52

53 Real-time noise-aware TMO 53

54 Video tone-mapping Temporal coherence in video tone-mapping

55 Temporal coherence Temporal filtering Parameters (those that depend on image statistics) Tone-curve Per-pixel (low-pass, bilateral, optical flow) 55

56 Temporal coherence Temporal filtering Parameters (those that depend on image statistics) Tone-curve Per-pixel (low-pass, bilateral, optical flow) (Lt )= t+d X G ( (L r )) Non-causal r=t d (Lt )= (L t )+(1 ) (L t 1 ) Causal 56

57 Temporal coherence Temporal filtering Parameters (those that depend on image statistics) Tone-curve Per-pixel (low-pass, bilateral, optical flow) (Lt )= t+d X G ( (L r )) Non-causal r=t d (Lt )= (L t )+(1 ) (L t 1 ) Causal 57

58 Temporal coherence Temporal filtering Parameters (those that depend on image statistics) Tone-curve Per-pixel (low-pass, bilateral, optical flow) Naive: L s,t = 1 Ḡ Xt+d r=t d G ( t r ) L s,r Edge-aware: L s,t = 1 Ḡ Xt+d r=t d G a ( L s,t L s,r ) G b ( t r ) L s,r Motion-compensated: L s,t = 1 H Xt+d r=t d H L s,t L s vs,t$r,r L s vs,t$r,r 58

59 Temporal coherence HDR video sequence from: FROEHLICH J., GRANDINETTI S., EBERHARDT B., WALTER S., SCHILLING A., BRENDEL H.: Creating Cinematic Wide Gamut HDR-Video for the Evaluation of Tone Mapping Operators and HDR-Displays. In Proc. SPIE 9023, Digital Photography X (2014), pp X 90230X

60 Temporal coherence Display adaptive TMO: MANTIUK R., DALY S., KEROFSKY L.: Display adaptive tone mapping. ACM Trans. Graphics 27, 3 (2008), 68:1 68:10. 60

61 Temporal coherence Cone model TMO: VAN HATEREN J. H.: Encoding of high dynamic range video with a model of human cones. ACM Trans. Graphics 25 (2006),

62 Temporal coherence Virtual exposures TMO: BENNETT E. P., MCMILLAN L.: Video enhancement using per- pixel virtual exposures. ACM Trans. Graphics 24, 3 (2005),

63 Temporal coherence Motion path filtering TMO: AYDIN T. O., STEFANOSKI N., CROCI S., GROSS M., SMOLIC A.: Temporally coherent local tone mapping of HDR video. ACM Trans. Graphics 33, 6 (2014),

64 Temporal coherence Real-time noiseaware TMO: EILERTSEN G., MANTIUK R. K., UNGER J.: Real-time noise- aware tone mapping. ACM Trans. Graph. 34, 6 (Oct. 2015), 198:1 198:15. 64

65 Temporal coherence Temporal filtering Parameters (those that depend on image statistics) Tone-curve Per-pixel (low-pass, bilateral, block matching, optical flow) Post-processing Global methods GUTHIER B., KOPF S., EBLE M., EFFELSBERG W.: Flicker reduction in tone mapped high dynamic range video. In Proc. SPIE (2011), vol. 7866, pp C 78660C 15. BOITARD R., BOUATOUCH K., COZOT R., THOREAU D., GRUSON A.: Temporal coherency for video tone mapping. In Proc. SPIE 8499, Applications of Digital Image Processing XXXV (San Diego, 2012), pp D 84990D 10. Local methods LANG M., WANG O., AYDIN T., SMOLIC A., GROSS M.: Practical temporal consistency for image-based graphics applications. ACM Trans. Graph. 31, 4 (July 2012), 34:1 34:8. BONNEEL N., TOMPKIN J., SUNKAVALLI K., SUN D., PARIS S., PFISTER H.: Blind video temporal consistency. ACM Trans. Graph. 34, 6 (Oct. 2015), 196:1 196:9. 65

66 Video tone-mapping Noise visibility in video tone-mapping

67 Noise visibility 1 Log contrast G Noise after tone mapping Display limits Camera noise Human visual system threshold Luminance [cd/m 2 ] 67

68 Noise visibility 68

69 Noise visibility Increase of noise visibility For non-linear mappings For local TMOs Noise reduction Heavy-weight Bilateral, optical flow, higher order (V-BM4D, etc.) Noise-awareness Light-weight Control of noise visibility during tone-mapping 69

70 Objective evaluation

71 Measures Temporal coherence Noise visibility Contrast Exposure 71

72 Objective evaluation Temporal coherence

73 Temporal coherence measure Compare correlation between HDR and tone-mapping (L t,t t )= 1 2d t+d X r=t d L r µ Lt L t T t T r µ Tt In order to account for adaptation Linear regression within temporal window Creates invariance to locally linear changes Final incoherency measure: 1 max(0, (L t,t t )) 73

74 Temporal coherence measure

75 Temporal coherence measure Time-adaptation TMO Local adaptation TMO Virtual exposures TMO Display adaptive TMO Temporal coherence TMO Motion path filtering TMO 75

76 Temporal coherence result Local incoherence Global incoherence Incoherence metric Vis. adapt. TMO Time-adapt. TMO Loc. adapt. TMO Mal-adapt. TMO Virt. exp. TMO Cone model TMO Disp. adapt. TMO Ret. model TMO Color app. TMO Temp. coh. TMO Zonal coh. TMO Motion filt. TMO Noise-aw. TMO Camera TMO Local operators 76

77 Unacceptable Virtual exposures TMO Retina model TMO Very visible Visible Barely visible Invisible Consistency Flickering Ghosting Noise Consistency Flickering Ghosting Noise Unacceptable Very visible Visible Local adaptation TMO Color apperance TMO Hallway Hallway 2 Exhibition area Driving simulator Students Window Barely visible Invisible Consistency Flickering Ghosting Noise Consistency Flickering Ghosting Noise 77

78 Objective evaluation Noise visibility

79 Noise visibility measure ˆL = L + N L ˆL Q(ˆL, L) T = V (L) n(l, T )=Q(ˆL, L) Q( ˆT,T) ˆT = V (ˆL) T Q( ˆT,T) ˆT 79

80 Noise visibility result 15 Noise visibility difference Vis. adapt. TMO Time-adapt. TMO Loc. adapt. TMO Mal-adapt. TMO Virt. exp. TMO Cone model TMO Disp. adapt. TMO Ret. model TMO Color app. TMO Temp. coh. TMO Zonal coh. TMO Motion filt. TMO Noise-aw. TMO Camera TMO 80

81 Noise visibility result 15 Noise visibility difference Vis. adapt. TMO Time-adapt. TMO Loc. adapt. TMO Mal-adapt. TMO Virt. exp. TMO Cone model TMO Disp. adapt. TMO Ret. model TMO Color app. TMO Temp. coh. TMO Zonal coh. TMO Motion filt. TMO Noise-aw. TMO Camera TMO 81

82 Objective evaluation Contrast

83 Contrast measure Global contrast C global (T )= 1 N X s p (G T 2 ) s (G T ) 2 s Local contrast C local (T )= 1 N X s L s T s X p2 s G(T p T s )G(p s) 83

84 Contrast result 0 Contrast difference Local contrast Global contrast Vis. adapt. TMO Time-adapt. TMO Loc. adapt. TMO Mal-adapt. TMO Virt. exp. TMO Cone model TMO Disp. adapt. TMO Ret. model TMO Color app. TMO Temp. coh. TMO Zonal coh. TMO Motion filt. TMO Noise-aw. TMO Camera TMO Local operators 84

85 Objective evaluation Exposure

86 Exposure measure Probability [ ] R 0.02 (L)dL 0 R 1 0 (L)dL Tone-mapped value [ L] R (L)dL R 1 0 (L)dL 86

87 Exposure result Over-exposure Under-exposure # of pixels (%) Vis. adapt. TMO Time-adapt. TMO Loc. adapt. TMO Mal-adapt. TMO Virt. exp. TMO Cone model TMO Disp. adapt. TMO Ret. model TMO Color app. TMO Temp. coh. TMO Zonal coh. TMO Motion filt. TMO Noise-aw. TMO Camera TMO 87

88 Objective evaluation visualization Retained local contrast Lost local contrast Local contrast Virtual exposures TMO Color appearance TMO Cone model TMO Local adaptation TMO Visual adaptation TMO Noise-aware TMO Zonal coherence TMO Motion path filtering TMO Display adaptive TMO Temporal coherence TMO Mal-adaptation TMO Camera TMO Processing Temporal filter Intent Measures Global Local Global Local Visual system simulator Best subjective quality Scene reproduction Exposure Noise visibility Retina model TMO Time adaptation TMO Temporal coherence Temporally incoherent Temporally coherent 88

89 Summary

90 Summary Introduction HDR video (capturing, displays, distribution) Tone-mapping (basic algorithms) Historical overview Video tone-mapping Introduction Categorization Operators Video specific tone-mapping problems Objective evaluation Measures Results 90

91 A comparative review of tone-mapping algorithms for high dynamic range video Thank you!