Video Textures. Arno Schödl Richard Szeliski David H. Salesin Irfan Essa. presented by Marco Meyer. Video Textures

Transcription

1 Arno Schödl Richard Szeliski David H. Salesin Irfan Essa presented by Marco Meyer

2 Motivation Images lack of dynamics Videos finite duration lack of timeless quality of image

3 Motivation Image Video Texture Video endless endless & in motion in motion Applications: dyn. scenes on the web dyn. background user-controlled animation

4 Outline Introduction Representation MAIN SYSTEM Extensions Results Future Work & Discussion

5 Introduction If two frames are similar, we can create a transition I1 I I 2 3 I 4... Find sequence with respect to global structure Smooth the selected transitions

6 Representation VT can be seen as Markov processes P 31 I1 I I P 12 P 23 P 34 I... P 24 Matrix of probabilities matrix mostly sparse Set of explicit links with associated probability

7 Outline Introduction Representation MAIN SYSTEM Extensions Results Analysis Synthesis random video play loop Rendering Future Work & Discussion

8 Analysis Preparation steps brightness equalization video stabilisation We need a measure of similarity L 2 distance distance-matrix D j i D ij = I i I j 2

9 Analysis Probability matrix D ij Map distances to probabilities P ij exp( D i+ 1, j σ ) Normalize so that P ij j P ij =1

10 Analysis Video P ij exp( D i+ 1, j σ ) high σ low σ

11 Analysis Preserving Dynamics

12 Analysis Preserving Dynamics Optical flow computation bad in untextured areas Compare temporally adjacent frames match subsequences instead of single frames filter matrix with a diagonal kernel with weights w k D' ij = m 1 k = m w k D i+ k, j+ k

13 Analysis Preserving Dynamics Dij D' ij j i swing to the right P ij swing to the left P' ij

14 Analysis Dead Ends Only window is considered we might get into dead end try to predict future cost Anticipated future cost - propagate future costs backward '' ' p D = ( ) + α ij D ij k '' '' P jk D jk with P '' i+ 1, ij '' D exp( σ j ) Solve by iteration

15 Analysis Dead Ends Only window is considered we might get into dead end try to predict future cost Anticipated future cost - propagate future costs backward '' ' p D = ( ) + α min ij D ij k D '' jk Solve with Q-Learning

16 Analysis Dead Ends Most effective if solved from last to first row P' ij P '' ij

17 Analysis Pruning Goal to supress non-optimal transitions to save storage space Paradigms 1. keep only local maxima 2. keep those transitions with probabilities above threshold

18 Synthesis Random play begin anywhere but in dead end '' select next frame according to P ij VT is never repeated exactly the same way Video loop fixed length loop for conventional digital video players

19 Synthesis Video loop primitive loop To combine loops ranges must overlap n range = [2,9] length = 7 forward transitions not considered compound loop 0 n Loop length # exec i length _ of _ loop i 2 9 i

20 Synthesis Optimal loop algorithm Multiset describes loop with focus to: length range Straightforward approach: exponentially in #transitions Optimal loop algorithm: select set of transitions (dyn. programming) schedule primitive loops

21 Synthesis Select set of transitions A(2) D(5) C(4) B(3) L A(2) B(3) C(4) D(5) Alg: foreach cell if L>=length of transition of current column find legal lowest cost multiset of length L, where the transition of current column appears at least once

22 Synthesis Select set of transitions A(2) D(5) C(4) B(3) L A(2) B(3) C(4) D(5) 1 B(3) 2 BB(6) D(5) Alg: foreach cell if L>=length of transition of current column find legal lowest cost multiset of length L, where the transition of current column appears at least once

23 Synthesis Select set of transitions A(2) D(5) C(4) B(3) L A(2) B(3) C(4) D(5) 1 B(3) 2 BB(6) D(5) Alg: foreach cell if L>=length of transition of current column find legal lowest cost multiset of length L, where the transition of current column appears at least once

24 Synthesis Select set of transitions A(2) D(5) C(4) B(3) L A(2) B(3) C(4) D(5) 1 B(3) 2 BB(6) D(5) 3 BBB (9) Alg: foreach cell if L>=length of transition of current column find legal lowest cost multiset of length L, where the transition of current column appears at least once

25 Synthesis Select set of transitions A(2) D(5) C(4) B(3) L A(2) B(3) C(4) D(5) 1 B(3) 2 BB(6) D(5) 3 BBB (9) C(4) 4 BBBB(12) DD(10) 5 A(2) BBBBB(15) CD(9) CD(9) 6 AB(5) Alg: foreach cell if L>=length of transition of current column find legal lowest cost multiset of length L, where the transition of current column appears at least once

26 Synthesis Select set of transitions A(2) D(5) C(4) B(3) L A(2) B(3) C(4) D(5) 1 B(3) 2 BB(6) D(5) 3 BBB (9) C(4) 4 BBBB(12) DD(10) 5 A(2) BBBBB(15) CD(9) CD(9) 6 AB(5) Alg: foreach cell if L>=length of transition of current column find legal lowest cost multiset of length L, where the transition of current column appears at least once

27 Synthesis Select set of transitions A(2) D(5) C(4) B(3) L A(2) B(3) C(4) D(5) 1 B(3) 2 BB(6) D(5) 3 BBB (9) C(4) 4 BBBB(12) DD(10) 5 A(2) BBBBB(15) CD(9) CD(9) 6 AB(5) Alg: foreach cell if L>=length of transition of current column find legal lowest cost multiset of length L, where the transition of current column appears at least once

28 Synthesis Select set of transitions A(2) D(5) C(4) B(3) L A(2) B(3) C(4) D(5) 1 B(3) 2 BB(6) D(5) 3 BBB (9) C(4) 4 BBBB(12) DD(10) 5 A(2) BBBBB(15) CD(9) CD(9) 6 AB(5) Alg: foreach cell if L>=length of transition of current column find legal lowest cost multiset of length L, where the transition of current column appears at least once

29 Synthesis Select set of transitions A(2) D(5) C(4) B(3) L A(2) B(3) C(4) D(5) 1 B(3) 2 BB(6) D(5) 3 BBB (9) C(4) 4 BBBB(12) DD(10) 5 A(2) BBBBB(15) CD(9) CD(9) 6 AB(5) Alg: foreach cell if L>=length of transition of current column find legal lowest cost multiset of length L, where the transition of current column appears at least once

30 Synthesis Select set of transitions A(2) D(5) C(4) B(3) L A(2) B(3) C(4) D(5) 1 B(3) 2 BB(6) D(5) 3 BBB (9) C(4) 4 BBBB(12) DD(10) 5 A(2) BBBBB(15) CD(9) CD(9) 6 AB(5) AB(5) Alg: foreach cell if L>=length of transition of current column find legal lowest cost multiset of length L, where the transition of current column appears at least once

31 Synthesis Select set of transitions A(2) D(5) C(4) B(3) L A(2) B(3) C(4) D(5) 1 B(3) 2 BB(6) D(5) 3 BBB (9) C(4) 4 BBBB(12) DD(10) 5 A(2) BBBBB(15) CD(9) CD(9) 6 AB(5) AB(5) CC(8) DDD(15) Alg: foreach cell if L>=length of transition of current column time O(L 2 N 2 ) space find legal lowest cost multiset of length L, where the transition of current column appears at least once O(LN) optionally store pointers + range instead of full description

32 Synthesis Schedule primitive loops D C A B Alg: Given: length =17 transitions ABCDD occur current schedule:

33 Synthesis Schedule primitive loops j D C B i Alg: Schedule transition i j that starts at the end 1 or more ranges Given: length =17 transitions ABCDD occur current schedule: invariant: always in first range A

34 Synthesis Schedule primitive loops j D C B i Alg: Schedule transition i j that starts at the end 1 or more ranges Given: length =17 transitions ABCDD occur current schedule: Schedule any* transition in first range that starts after j invariant: always in first range A D * deterministic or stochastic

35 Synthesis Schedule primitive loops D B j i Given: length =17 transitions ABCDD occur current schedule: Alg: Schedule transition i j that starts at the end 1 or more ranges Schedule any* transition in first range that starts after j invariant: always in first range A D C * deterministic or stochastic

36 Synthesis Schedule primitive loops B j i Given: length =17 transitions ABCDD occur current schedule: Alg: Schedule transition i j that starts at the end 1 or more ranges Schedule any* transition in first range that starts after j invariant: always in first range A D C D * deterministic or stochastic

37 Synthesis Schedule primitive loops j i Given: length =17 transitions ABCDD occur current schedule: Alg: Schedule transition i j that starts at the end 1 or more ranges Schedule any* transition in first range that starts after j invariant: always in first range A D C D B * deterministic or stochastic

38 Rendering Jump cut noticable transitions Cross-fade blur if misaligned Morph common features get aligned

39 Rendering flag

40 Rendering Multi-way cross-fade multiple frames displayed at a time weighted average of all participating frames

41 Outline Introduction Representation MAIN SYSTEM Extensions Results Future Work & Discussion

42 3D Video Texture replace video with video texture create novel view

43 Extensions Motion factorization Divide original video into independently moving parts manual automatic

44 Extensions Video-based Animation Control where, how & how long to replay Add user-controlled term to error function based on: velocity of video sprite position of frame

45 Extensions Video-based Animation

46 Extensions Video Sprites 1. Object extraction (bg subtraction, blue screen) 2. Store velocity of centroid 3. Center object Analysis Synthesis + alpha, direction, speed + velocities

47 Extensions Video Sprites

48 Results Simple algorithm Several extensions & applications User control Combine video textures

49 Results Algorithm fails at complex and high-structured video material

50 Future Work & Discussion Distance metrics Blending Variety Control tools