3D Cinematography Principles and Their Applications to

Transcription

1 3D Cinematography Principles and Their Applications to Stereoscopic Media Processing Chun-Wei Liu, Tz-Huan Huang, Ming-Hsu Chang, Ken-Yi Lee, Chia-Kai Liang, and Yung-Yu Chuang National Taiwan University and Lytro, Inc. 國立台灣大學 National Taiwan University

2 3D is hot today

3 3D display devices

4 3D capturing devices

5 Media life cycle Acquisition iti Presentation ti

6 Media life cycle Enhancement Analysis Organization Acquisition Processing Presentation

7 3D authoring tools Little progress has been made on the software side for the consumer 3D media processing. Many operations that are not important to Many operations that are not important to professionals but essential for consumers.

8 3D authoring tools Consumer Professional VS.

9 Books of 3D cinematography

10 Binocular disparity Left view Right view

11 Binocular disparity

12 Binocular disparity

13 Accommodation and vergence Real world Stereoscopic displays

14 3D cinematography principles 1. Maintain coordination between views. 2. Have a continuous depth chart. 3. Place rest areas between strong 3D shots. 4. Use a shallow depth of field for shots with excessive depth brackets. 5. Be careful about the stereoscopic window.

15 1. Maintain coordination between views Vertical Size Distortioni shift difference difference Diverged

16 2. Have a continuous depth chart

17 2. Have a continuous depth chart Time

18 3. Place rest areas between strong 3D shots Time Rest area

19 4. Shallow DoF for excessive depth brackets

20 4. Shallow DoF for excessive depth brackets In focus Out of focus

21 5. Be careful about the stereoscopic window

22 Guidelines for the principles Temporal media 1 Depth change Re-frame media 5

23 Guidelines for the principles 1. Maintain coordination between views. Almost every stereoscopic media authoring method should obey this fundamental principle. i

24 Guidelines for the principles 2. Have a continuous depth chart. 3. Place rest areas between strong 3D shots. Relevant for temporal media editing Video editing Video retargeting

25 Guidelines for the principles 4. Use a shallow depth of field for shots with excessive depth brackets. When the authoring operations cause changes in the depth brackets Image stitching Photomontage

26 Guidelines for the principles 5. Be careful about the stereoscopic window. When the authoring operators re-frame media Cropping Compositing Resizing

27 Application 1: Stereoscopic video stabilization 國立台灣大學 National Taiwan University

28 Jittered video

29 Video stabilization Our stereoscopic video stabilization approach is based on this method: Video stabilization using robust feature trajectories Ken-Yi Lee, Yung-Yu Chuang, Bing-Yu Chen, and Ming Ouhyoung ICCV 2009

30 Key idea T0 T1 T T4 T5 T6 T7 T3 T2

31 Key idea Before stabilization After stabilization 31

32 Approach Original video Robust Feature Trajectories Stabilized video Stabilization by Optimization 32

33 Robust feature trajectories... 33

34 Stabilization by optimization Trajectory Roughness Video Quality Degradation Distortion Scale Uncovered area 34

35 Amateur stereoscopic video

36 Stereoscopic video stabilization Feature trajectories Joint optimization

37 Robust feature trajectories Feature detection SIFT Feature verification Fundamental Matrix Trajectory tracking Base on Lee et al. in ICCV 2009 Feature trajectories Joint optimization 37

38 Joint optimization Perform the optimization jointly for two views. Incorporate stereoscopic constraints (Principle 1). Feature trajectories Left Right Disparity Joint optimization

39 Joint optimization: feature trajectory Feature trajectories Joint optimization 39

40 Joint optimization: stereoscopic constraint 2 Motion estimation Joint optimization 40

41 Input

42 Naïve solution

43 Our result

44 Comparisons Original Naive Our method

45 30 subjects User study 5 questions, 1-5 scores 3 cases children csiegirl redgirl

46 Questions 1. Stabilization. What do they think about the 2D stabilization? 2. Depth continuity. What do you think about the depth stabilization of the video? 3. Stereoscopic effect. What do you think about the comfort and the stereoscopic effect in the video? 4. Experience. How does the stabilization effect help you experience the video? 5. Acceptance. How much are you willing to adopt the bili i ff? stabilization effect?

47 Results of user study

48 Application 2: Stereoscopic photo slideshow 國立台灣大學 National Taiwan University

49 The order for displaying photos In their temporal order Random User defined

50 No effect Fade-in/fade-out Blending Inter-photo effects

51 No effect Pan Zoom Intra-photo effects

52 Stereoscopic photo slideshow The order for displaying slides Photo ordering (Principle 2) Inter-slide effects Active depth cuts (Principle 2, 3) Intra-slide effects Shallow depth-of-field effects (Principle p 4) )

53 Photo reordering Disparity map ROI map 2

54 Active depth cuts Depth shift: translate two images horizontally to change the depth brackets. Photo 1 Photo 2 Depth Screen bracket No effect, Fade-in/out, Blending, Depth bracket

55 15 subjects User study 5 questions, 1-5 scores 8 photos in the slideshow 6 combinations 1. No effect 2. Blending 3. Fade-in/fade-out 4. Depth shift 5. Depth shift + blending 6. Depth shift + fade-in/fade-out

56 Questions 1. Smoothness. What do you think about the smoothness of photo transitions? i 2. Depth continuity. What do you think about the smoothness of the depth (disparity) variation on the display of each photo? 3. Stereoscopic effect. What do you think about the comfort and stereoscopic effect of the slideshow? 4. Experience. To what extent does the transition effect help you experience the trip? 5. Acceptance. How much are you willing to adopt the transition i effect?

57 Results of the user study

58 Shallow depth-of-field effects Blur image according to the depth Disparity map Near Far

59 Shallow depth-of-field effects Blur image according to the depth Disparity map Focused depth

60 Shallow depth-of-field effects

61 Shallow depth-of-field effects

62 Shallow depth-of-field effects

63 Shallow depth-of-field effects

64 Summary The order for displaying slides If allowed, reorder the photo sequence (Principle 2). Inter-photo effects If there is a large depth jump, use active depth cuts and fade-in/fade-out effect (Principle 2, 3). Intra-photo effects If the depth bracket of the image is excessive, apply shallow depth-of-field blur filtering (Principle 4). p g( p )

65 Video

66 Conclusion Introduce the principles for stereoscopic cinematography. Two applications, stabilization and slideshow, show that these principles are essential for stereoscopic media processing.

67 Thanks for your attention 國立台灣大學 National Taiwan University