Chroma Adjustment for HDR Video. Jacob Ström, Per Wennersten Ericsson Research

Transcription

1 Chroma Adjustment for HD Video Jacob Ström, Per Wennersten Ericsson esearch

2 Video representation scene Ericsson Confidential Ericsson A Page 2

3 Video representation scene camera sensor Ericsson Confidential Ericsson A Page 3

4 Video representation Every camera sensor works by registering photons that hit the sensor A linear sensor can be seen as counting the photons scene camera sensor linear At the most basic level, we therefore measure the linear values for, and. Ericsson Confidential Ericsson A Page 4

5 Video representation Problem: A linear representation is not a good way to spend code levels 100 photons to % increase visible difference photons to scene camera sensor linear 0.1% increase not visible Ericsson Confidential Ericsson A Page 5

6 Video representation Problem: A linear representation is not a good way to spend code levels 100 photons to % increase visible difference photons to scene camera sensor linear 0.1% increase not visible too many bits spent here Ericsson Confidential Ericsson A Page 6

7 Video representation Problem: A linear representation is not a good way to spend code levels 100 photons to % increase visible difference photons to too few bits spent here scene camera sensor linear 0.1% increase not visible too many bits spent here Ericsson Confidential Ericsson A Page 7

8 Video representation Solution: Add nonlinearity before coding output = f scene camera sensor linear input max Ericsson Confidential Ericsson A Page 8

9 Video representation Solution: Add nonlinearity before coding scene camera sensor linear Ericsson Confidential Ericsson A Page 9

10 Video representation Solution: Add nonlinearity before coding scene camera sensor linear Ericsson Confidential Ericsson A Page 10

11 Video representation Solution: Add nonlinearity before coding scene camera sensor linear Ericsson Confidential Ericsson A Page 11

12 Video representation Solution: Add nonlinearity before coding scene camera sensor linear Ericsson Confidential Ericsson A Page 12

13 Video representation Solution: Add nonlinearity before coding scene camera sensor linear non-linear These are the variables people are used to handle when dealing with 8 bit colors, e.g. (255, 175, 44) Ericsson Confidential Ericsson A Page 13

14 Video representation For HD, the max is higher and the non-linearity is steeper output scene camera sensor linear non-linear input max Ericsson Confidential Ericsson A Page 14

15 Video representation For HD, the max is higher and the non-linearity is steeper output scene camera sensor linear non-linear input max Ericsson Confidential Ericsson A Page 16

16 Video representation 2 nd problem:, and are correlated. Inefficient to send the same data three times Solution: Use average/difference repr. scene camera sensor linear non-linear Ericsson Confidential Ericsson A Page 17

17 Video representation luma : Y = w + w + w scene camera sensor linear non-linear Ericsson Confidential Ericsson A Page 18

18 Video representation luma : Y = w + w + w Cb ~ Y Cr ~ Y chroma : scene camera sensor linear non-linear Ericsson Confidential Ericsson A Page 19

19 Video representation luma : Y = w + w + w Cb ~ Y Cr ~ Y chroma : scene camera sensor linear non-linear Eye is better at bright/dark than color. Encode luma in full resolution Encode chroma in half resolution Ericsson Confidential Ericsson A Page 20

20 Video representation luma : Y = w + w + w Cb ~ Y Cr ~ Y chroma : scene camera sensor linear non-linear 3 rd problem: Eye detects luminance, not luma. Ericsson Confidential Ericsson A Page 21

21 Video representation luma : Y = w + w + w Cb ~ Y Cr ~ Y chroma : scene camera sensor linear non-linear 3 rd problem: Eye detects luminance, not luma. luminance: Y = w + w + w Ericsson Confidential Ericsson A Page 22

22 Video representation luma : Y = w + w + w Cb ~ Y Cr ~ Y chroma : scene camera sensor luminance: Y = w + w + w Ericsson Confidential Ericsson A Page 23 linear non-linear f(y) 3 rd problem: Eye detects luminance, not luma. This is what the eye is sensitive to. This is how f(y) was constructed. f Y Y

23 Video representation Summary: The eye sees luminance f(y) We encode luma Y Does it matter? Yes, as we will see Why not just encode luminance instead? Constant luminance-ycbcr does this, and IC T C P comes close However, a lot of devices and network equipment can only handle traditional Y CbCr Ericsson Confidential Ericsson A Page 24

24 How do we see color? Luminance f(y) only describes the brightness part of the image The color part can be described by CIELUV chromaticity coordinates u, v Larson [2] uses u, v to determine if two colors are perceptually different. [2].W.Larson, The logluv encoding for full gamut, high dynamic range images, Journal of raphics Tools, vol. 3(1), pp , March Ericsson Confidential Ericsson A Page 25

25 Three spaces What the camera measures What we encode What the eye perceives Ericsson Confidential Ericsson A Page 26

26 Three spaces What the camera measures one scanline Ericsson Confidential Ericsson A Page 27

27 Three spaces What the camera measures gray area plus noise Ericsson Confidential Ericsson A Page 28

28 Three spaces What the camera measures linear Ericsson Confidential Ericsson A Page 29

29 Three spaces What the camera measures What we encode Y Cb Cr linear luma and chroma Ericsson Confidential Ericsson A Page 30

30 Three spaces What the camera measures What we encode What the eye perceives Y Cb Cr f(y) u v linear luma and chroma luminance and chromaticity Ericsson Confidential Ericsson A Page 31

31 Three spaces What the camera measures red area plus noise Ericsson Confidential Ericsson A Page 32

32 Three spaces What the camera measures What we encode What the eye perceives Y Cb Cr f(y) u v Problem 1: Small changes in what we measure becomes a noisy and hard signal to encode, even though we cannot see any noise with the naked eye. Ericsson Confidential Ericsson A Page 33

33 Three spaces What the camera measures What we encode What the eye perceives Y Cb Cr f(y) u v Problem 2: The wild swings in Y CbCr must cancel each other perfectly to get an output that looks calm. If we subsample Cb and Cr but not Y, this balance is disturbed and large error occurs in luminance. Ericsson Confidential Ericsson A Page 34

34 Three spaces What the camera measures Y What we encode What the eye perceives f(y) large, visible errors Cb subsampled u Cr subsampled v Problem 2: The wild swings in Y CbCr must cancel each other perfectly to get an output that looks calm. If we subsample Cb and Cr but not Y, this balance is disturbed and large errors occur in luminance. Ericsson Confidential Ericsson A Page 35

35 example artifacts Original image Ericsson Confidential Ericsson A Page 36 oing to Y CbCr, subsampling and going back

36 Luma adjustment [9] What the camera measures Y What we encode What the eye perceives f(y) large, visible errors Cb subsampled u Cr subsampled v Adapt the luma (Y ) component so that the visible errors in luminance disappears. [9] J. Ström, J. Samuelsson, and K. Dovstam, Luma adjustment for high dynamic range video, DCC Ericsson Confidential Ericsson A Page 37

37 Luma adjustment [9] What the camera measures Y What we encode adjusted What the eye perceives f(y) no visible error Cb subsampled u Cr subsampled v Adapt the luma (Y ) component so that the visible errors in luminance disappears. [9] J. Ström, J. Samuelsson, and K. Dovstam, Luma adjustment for high dynamic range video, DCC Ericsson Confidential Ericsson A Page 38

38 Our approach Why is the Y Cb Cr representation so noisy? Signal near zero is amplified due to large gain of transfer function output input Ericsson Confidential Ericsson A Page 39

39 Our approach Why is the Y Cb Cr representation so noisy? Signal near zero is amplified due to large gain of transfer function output small noise input Ericsson Confidential Ericsson A Page 40 large noise

40 Our approach Why is the Y Cb Cr representation so noisy? Signal near zero is amplified due to large gain of transfer function output large noise input Ericsson Confidential Ericsson A Page 41 small noise

41 Our approach Why is the Y Cb Cr representation so noisy? Signal near zero is amplified due to large gain of transfer function not very noisy Ericsson Confidential Ericsson A Page 42

42 Our approach Why is the Y Cb Cr representation so noisy? Signal near zero is amplified due to large gain of transfer function not very noisy green very noisy Ericsson Confidential Ericsson A Page 43

43 Our approach Why is the Y Cb Cr representation so noisy? Signal near zero is amplified due to large gain of transfer function not very noisy green very noisy all components very noisy Y matrix mult Cb Cr Ericsson Confidential Ericsson A Page 44

44 Main idea What the camera measures What the eye perceives f(y) u v filter here... Ericsson Confidential Ericsson A Page 45

45 Main idea What the camera measures What the eye perceives f(y) u v filter here while keeping track that what we see doesn t change... Ericsson Confidential Ericsson A Page 46

46 How much can we change g? What the eye perceives f Y before f(y after ) 0.5/876 half a quant level f(y) u v Ericsson Confidential Ericsson A Page 48

47 How much can we change g? What the eye perceives f Y before f(y after ) 0.5/876 half a quant level f(y) u before u after 0.5/410 v before v after 0.5/410 Larson [2] u v Ericsson Confidential Ericsson A Page 49

48 How much can we change g? What the eye perceives f Y before f(y after ) 1/876 u before u after 2/410 v before v after 2/410 f(y) u v Ericsson Confidential Ericsson A Page 50

49 Analytic luck For a certain color,,, it is possible to calculate the smallest and largest value of that still statisfy the three formulae: f Y before f(y after ) 1/876 u before u after 2/410 v before v after 2/410 ives min, max for a given,, It is even the case that min and max can be calculated analytically from, and! (Details in the paper.) Ericsson Confidential Ericsson A Page 51

50 COE POCESS 1. Calculate min, max for every pixel 2. Filter with FI filter 1, 1, 1, 1, 1 /5 in x- and y- directions. 3. Clamp every pixel = clamp( filter, min, max ) 4. epeat 2 and 3 Due to the clamping, every pixel is still perceptually equivalent to the original Ericsson Confidential Ericsson A Page 52

51 Main idea What the camera measures What we encode What the eye perceives Y Cb Cr f(y) u v filter here so that what we encode does not vary so much.... while keeping track that what we see doesn t change... Ericsson Confidential Ericsson A Page 53

52 Main idea What the camera measures What we encode What the eye perceives Y f(y) Cb subsampled u Cr subsampled v Now if we subsample chroma... Ericsson Confidential Ericsson A Page 54

53 Main idea What the camera measures What we encode What the eye perceives Y f(y) Cb subsampled u Cr subsampled v Now if we subsample chroma they will not change so much since a subsampled version of a smooth function is similar to the original... Ericsson Confidential Ericsson A Page 55

54 Main idea What the camera measures What we encode What the eye perceives Y f(y) Cb subsampled u Cr subsampled v Now if we subsample chroma they will not change so much since a subsampled version of a smooth function is similar to the original and the visible result is greatly improved. Ericsson Confidential Ericsson A Page 56

55 Traditional processing original frame 389 original frame 390 Ericsson Confidential Ericsson A Page 59

56 Traditional processing original frame 389 original frame 390 frame 389 frame 389 Ericsson Confidential Ericsson A Page 60

57 Traditional processing original frame 389 original frame 390 frame 389 frame 389 Ericsson Confidential Ericsson A Page 61

58 Traditional processing original frame 389 original frame 390 frame 389 frame 389 result after subsampling and compression using QP 21 Ericsson Confidential Ericsson A Page 62

59 Proposed method original frame 389 original frame 390 frame 389 frame 389 result after subsampling and compression using QP 21 Ericsson Confidential Ericsson A Page 63

60 esults: Visual results after compression improve original traditional subsampling luma adjustment proposed scheme Ericsson Confidential Ericsson A Page 64

61 objective results We compared traditional subsampling with the proposed method We used the A sequences from MPE (T.709 in a T.709 container) 2.4% D rate gain in tpsny φ = 2/410 θ = 1/876 tpsny tpsnxyz DE100 L100 All -2.4% -2.0% -0.3% -1.7% Ericsson Confidential Ericsson A Page 65

62 onus: SD We did not expect there to be so much to gain for standard dynamic range images, since the transfer function is less steep. However, it turns out that one of the MPE test sequences suffers from noise amplification and noise suppression. The proposed method removes these artifacts. Ericsson Confidential Ericsson A Page 66

63 SD: Compressed qp 22 noise amplification noise more true to original noise suppression Ericsson Confidential Ericsson A Page 69 traditional subsampling proposed method

64 Objective Measurement A -21% Ericsson Confidential Ericsson A Page 70

65 summary Core idea is to filter image just enough that it is not visible The filtering of components near zero will make Y CbCr representation much less noisy Suppresses noise amplification artifacts Makes neighboring frames more similar Prediction works better D rate change of -2.4% in tpsn Also works on regular standard dynamic range images Avoids noise amplification and gives coding gain Ericsson Confidential Ericsson A Page 71

66 Ericsson Confidential Ericsson A Page 72