A Summary of the Daala Project

Transcription

1 A Summary of the Daala Project Timothy B. Terriberry

2 Why We Care About RF Codec licensing is anti-competitive Licensing regimes are universally discriminatory An excuse for proprietary software (Flash) Web stack is traditionally unencumbered The Internet was based on innovation without asking permission 2 Even just counting the number of copies of software we distribute is cost-prohibitive Ignoring licensing costs creates risks that can show up at any time: a tax on success

3 Current Generation a Mess OpenH264 OS-level APIs Additional work for every platform Can t fix bugs/deficiencies Can t adapt to new use cases (e.g., WebRTC) Audio (AAC) is still problematic 3 Binary plugin, can t be deployed everywhere Attempts at an OpenH264-style licensing hack so far unsuccessful

4 Next Generation? 4 There can be no OpenHEVC Same licensing tricks that allow OpenH264 no longer possible With HEVC Advance, it seems HEVC is now too expensive for many commercial entities Need to make this better before the next-next generation gets here

5 The Royalty-Free Video Challenge Creating good codecs is hard But we don t need many The best implementations of patent-encumbered codecs are already free software Network effects decide 5 Where RF is established, non-free codecs see no adoption (JPEG, PNG, FLAC, ) RF is not enough People care about different things Must be better on all fronts

6 We Did This for Audio 6

7 The Daala Project Goal: Better than HEVC without infringing IPR Need a better strategy than read a lot of patents People don t believe you Analysis is error-prone 7 Try to stay far away from the line, but... One mistake can ruin years of development effort See: H.264 Baseline, VC-1, etc.

8 Strategy Look for some elements common to broad classes of patents 8 Only need to avoid one element in a patent claim to be able to say we don t do that Replace with fundamentally different techniques Higher risk/higher reward than incremental changes Can avoid vast swaths of IPR Creates new challenges others haven t solved Still have to read a lot of patents

9 Differences in Motion Compensation and Residual Coding 9

10 Motion Compensation Motion Compensation Copy blocks from an already encoded frame (offset by a motion vector) Subtract from the current frame Code the residual Input 10 = Reference frame Residual

11 Overlapped Block Motion Compensation 11 Instead of just copying a block, copy a bigger block and blend it with its neighbors Gives smaller, more uniform prediction errors Dates back to early 90's, but now being investigated for both VP10 and H.266 How much bigger? Usual approach is to overlap with the smallest block size (e.g., Dirac) Block size not really variable: same effect as small, fixed blocks, just a more efficient representation

12 Adaptive Repartitioning Zhang et al. (1998): Split large blocks until overlap compatible with smaller neighbors No overlap Fixed overlap Adaptive overlap 12 Competitor s paper

13 Limitations of Adaptive Repartitioning Might still use small windows with large blocks How do you search? 13 2-D dependency on neighbors' sizes NP hard

14 Daala's Approach Restrict adjacent block sizes to differ by no more than a factor of two Use a 4-8 mesh to impose the restriction Data structure for surface simplification in computer graphics Has a good search algorithm (Balmelli 2001) Define interpolation rules based on this restriction 14 Maintains continuity when blocks on one side of an edge are split and blocks on the other aren't

15 4-8 Mesh Start with the largest block size Level 0 15

16 4-8 Mesh Odd levels: can add new MVs to block centers Only if two parents are in the grid (always at L1) Level 1 16

17 4-8 Mesh Odd levels: can add new MVs to block centers Only if two parents are in the grid (always at L1) Level 1 17

18 4-8 Mesh Even levels: can add new MVs to block edges Again, only if two parents are in the grid Level 2 18

19 4-8 Mesh And so on... Level 3 19

20 4-8 Mesh And so on... and so on... Level 4 20

21 Blending Windows for Mismatched Block Sizes 21 Simple sums of bilinear weights

22 Block Size Decision Greedily inserting/removing leaves does poorly Objective not monotonic May have to add several MVs that regress R/D before you can add one that helps a lot General Decimation (Balmelli 2001) Bottom-up, but... Try deleting every MV in the grid 22 If it has children, delete them, too

23 General Decimation Greedy vs. General Decimation applied to surface simplification (Balmelli 2001) 23

24 Displaced Frame Difference Normal thing to do now is subtract this prediction from the input and code the residual The displaced frame difference (DFD) is the term of art for that residual Not in and of itself patentable! 24 At least, not anymore... But found as one element of nearly all patent claims on motion compensation

25 What We Do Instead 25 Perceptual Vector Quantization Based on work in Opus designed to preserve energy (film grain, fine details, etc.)

26 Perceptual Vector Quantization Separate gain (energy) from shape (spectrum) Vector = Magnitude Unit Vector (point on sphere) Potential advantages Can give each piece different rate allocations Free activity masking 26 Preserve energy (contrast) instead of low-passing Can throw away more information in regions of high contrast (relative error is smaller) The gain is what we need to know to do this! Better representation of coefficients

27 What does PVQ have to do with DFDs? Subtracting and coding a residual loses energy preservation But we still want to use predictors 27 The gain no longer represents the energy of the original signal They do a really good job of reducing what we need to code

28 What Does Prediction Really Do? Prediction changes the probability of points near the predictor Highly probable things are cheap to code With DFDs, highly probable means near zero Predicting gains is easy Enumerating points on a sphere near an arbitrary point (to model probabilities) is hard 28 Subtract gain of predictor Solution: Transform the space so we can single out points near the predictor

29 2-D Projection Example Input Input 29

30 2-D Projection Example Input + Prediction Prediction Input 30

31 2-D Projection Example Input + Prediction Compute Householder Reflection Prediction Input 31

32 2-D Projection Example Input + Prediction Compute Householder Reflection Apply Reflection Prediction Input 32

33 2-D Projection Example Input + Prediction Compute Householder Reflection Apply Reflection Compute & code angle Prediction θ Input 33

34 2-D Projection Example 34 Input + Prediction Compute Householder Reflection Apply Reflection Compute & code angle Code other dimensions Prediction θ Input

35 What does this accomplish? Creates another intuitive parameter, θ How much like the predictor are we? θ = 0 use predictor exactly Remaining N-1 dimensions are coded with VQ Instead of subtraction (translation), we re scaling and reflecting 35 We know their magnitude is gain*sin(θ) Whatever else you can say, this is nothing like computing a DFD

36 And it works! FastSSIM for turning on activity masking PSNR for PVQ vs. Scalar Quantization (flat quantization, no activity masking) 36

37 Other Differences... 37

38 Loop Filters Loop filters filter block edges to remove blocking artifacts Adaptive: filter strength depends on the amount of difference across the block edge Not invertible Simple filters used in H.263 (and Theora!) Since H.264 there s been an explosion of complex filter designs 38 Very simple to keep CPU cost low And patents

39 Lapped Transforms Non-adaptive, invertible deblocking post-filter Encoder applies the inverse (a blocking filter) Technique dates back to the 90 s Prefilter Postfilter DCT IDCT P P-1 DCT IDCT P P-1 DCT IDCT P P-1 DCT 39 IDCT

40 Blocking Filter 40 Prefilter makes things blocky

41 Old Lapping Strategy Filter size chosen based on size of smallest block on an edge (to prevent overlap) Filter order chosen to mimic a loop filter s 41 Horizontal edges first

42 Old Lapping Strategy 42 Filter size chosen based on size of smallest block on an edge (to prevent overlap) Filter order chosen to mimic a loop filter s Then vertical Maximal parallelism, minimum buffering

43 Problem #1: Basis Weirdness 43

44 Problem #2: Block size decision Have to know neighbors block sizes to compute lapping size Used a heuristic based on the estimated visibility of ringing to pick block sizes up front Worked okay for still images (at least not obviously broken) Was not making good decisions for inter frames Wanted to try explicit block size RDO (like other encoders) But lapping dependency makes this infeasible

45 Fixed Lapping : Remove the Dependency 45 Always use 8-point lapping (4 pixels on either side of an edge) Except on 4 4 blocks (details in a few slides) Always use 4-point lapping for chroma (because of subsampling)

46 New Filter Order 46 Filter top/bottom superblock (32 32) edges first

47 New Filter Order 47 Filter left/right superblock (32 32) edges next

48 New Filter Order 48 Splitting: Filter interior edges

49 New Filter Order Splitting: Filter interior edges 4 4 blocks: 49 Exterior edges use 8-point filter (from previous levels) Interior edges use 4-point filter (overlaps 8-point filter)

50 Results 50 Big boost in metrics PSNR PSNRHVS SSIM FASTSSIM Almost all from decision Used fixed lapping decision with old lapping scheme and got almost all of the gains RATE (%) DSNR (db) Smaller lapping means less ringing but more blockiness (especially on gradients) Didn t save much on ringing: 4 4 blocks have 12pixel support instead of 8 Added bilinear loop filter to compensate for blockiness

51 Bilinear Loop Filter Not a standard deblocking filter Doesn t look outside of current block! Compare decoded block to bilinear interpolation of corner pixels, blend with optimal Wiener filter gain 2 αq w=min 1, 12 σ 2 ( 51 )

52 Bilinear Loop Filter Not a standard deblocking filter Doesn t look outside of current block! Compare decoded block to bilinear interpolation of corner pixels, blend with optimal Wiener filter gain 2 2 αq w=min 1, 2 12 σ ( 52 )

53 Less Lapping In July, moved to 4-point lapping everywhere Not confident this is good 53 Much less ringing Less detail preservation PSNR PSNRHVS SSIM FASTSSIM RATE (%) DSNR (db) Small metrics changes, big visual changes Help doing visual tests welcome! Thor deblocking experiment Can now use 4-point deblocking filter from Thor instead of lapping (by flipping a #define) Help testing this also welcome

54 Spatial (Intra) Prediction Predict a block from its causal neighbors Explicitly code a direction along which to copy 54 Extend boundary of neighbors into new block along this direction

55 Intra Prediction with Lapped Transforms We can t copy pixels until we undo the lapping 55 We can t undo the lapping until we ve predicted those pixels Don t copy pixels: copy transform coefficients Currently just horizontal and vertical directions Chroma (color) predicted from luma (brightness) Not as good, but we try to make up for it elsewhere (e.g., lapping itself)

56 Binary Arithmetic Coding Code only binary decisions Actual cost in bits depends on probability Very cheap to code one symbol Need to code a lot of symbols (not parallelizable) Probability modeling Non-binary values 56 Simple 1-byte lookup tables Various schemes for converting to binary decisions ( binarization )

57 Non-Binary Arithmetic Coding Code values with up to 16 possibilities Equivalent to 4 binary decisions More expensive, but not 4x more expensive Use, e.g., expected value plus distribution shape (Laplace, Exponential) and compute on the fly Convert things to hex, not binary! 57 Effectively parallel! One byte cannot model 16 probabilities A lot of overheads are per-symbol Often combine multiple values into one symbol

58 How Are We Doing? 58

59 FastSSIM Progress Jan to Sep up and left is better Reduced rate by 82.75% HQ YouTube LQ Video Conference Jan H.265 Jun 59 Apr Feb Sep Nov May

60 PSNR-HVS Progress: Jan to Sep up and left is better Reduced rate by 80.91% HQ YouTube LQ Video Conference Jan May H.265 Jun 60 Sep Nov Apr Feb

61 Are We Compressed Yet? 61 Will run metrics on any git commit (we re happy to add your repository, just ask) Amazon EC2 instances, so results in a few minutes Details on setup at

62 Still Images Not actively working on a still-image format But video keyframes are still images JPEG is RF and already won And Daala will support 10- and 12-bit, 4:4:4, etc., unlike WebP And JPEG issued an open call in February Competition at 2015 Picture Coding Symposium to be used as input Included subjective (human) testing So we submitted Daala

63 PCS2015: Results 63

64 Questions? 64