INTRODUCTION TO 360 VIDEO. Oliver Wang Adobe Research

Transcription

1 INTRODUCTION TO 360 VIDEO Oliver Wang Adobe Research

2 OUTLINE What is 360 video?

3 OUTLINE What is 360 video? How do we represent it? Formats

4 OUTLINE What is 360 video? How do we represent it? How do we create it? Stitching

5 OUTLINE What is 360 How do we How do we What can we video? represent it? create it? do with it? Stabilization

6 OUTLINE What can t What is 360 How do we How do we What can we we do with video? represent it? create it? do with it? it?? Future Areas

7 OUTLINE What can t What is 360 How do we How do we What can we we do with video? represent it? create it? do with it? it?

8 360 VIDEO VS FULL VR VIDEO 3DoF 6DoF Mozilla Developer Network - WebVR Concepts

9 ONE OF THE MOST POPULAR FORMS OF VR

10 360 VIDEO VS FULL Display Capture VS

11 OUTLINE What can t What is 360 How do we How do we What can we we do with video? represent it? create it? do with it? it?

12 360 VIDEO FORMATS

13 360 VIDEO FORMATS

14 360 VIDEO FORMATS

15 EQUIRECTANGULAR

16 EQUIRECTANGULAR Positive Negative Visually interpretable Distortion on poles Easy to process Low pixel density on horizon

17 CUBEMAPS

18 CUBEMAPS Graphics pipeline friendly Hard to interpret/process Low distortion (25% storage reduction)

19 EQUIANGULAR CUBEMAPS

20 EQUIANGULAR CUBEMAPS Equirectangular Projection Cubemap Equiangular Cubemap

21 EQUIANGULAR CUBEMAPS

22 PYRAMIDS

23 PYRAMIDS

24 360 VIDEO CONTENT CREATION What can t What is 360 How do we How do we What can we we do with video? represent it? create it? do with it? it?

25 MULTI-LENS CAMERAS

26 STITCHING Image Alignment and Stitching [Szeliski, Richard 2005]

27 PARALLAX ARTIFACTS different projection centers Slides courtesy of Jungjin Lee, KAIST

28 PARALLAX COMPENSATION Optical Flow Mesh Warping [Jiang and Gu 2015]

29 HIGH-LEVEL PIPELINE - INPUT Panoramic Video from Unstructured Camera Arrays [Perazzi et al. Eurographics 2015]

30 REFERENCE PROJECTION Slides courtesy of Federico Perazzi, Disney Research

31 PARALLAX COMPENSATION Slides courtesy of Federico Perazzi, Disney Research

32 PARALLAX COMPENSATION Slides courtesy of Federico Perazzi, Disney Research

33 BLENDING Slides courtesy of Federico Perazzi, Disney Research

34 BLENDING A Multiresolution Spline With Application to Image Mosaics [Burt and Adelson 1983] Slides courtesy of Federico Perazzi, Disney Research

35 Slides courtesy of Federico Perazzi, Disney Research

36 Slides courtesy of Federico Perazzi, Disney Research

37 ANOTHER WAY TO THINK ABOUT IT Rich360: Optimized Spherical Representation from Structured Panoramic Camera Arrays [Lee et al. SIGGRAPH 2016] Slides courtesy of Jungjin Lee, KAIST

38 MESH DEFORMATION Deformation of the projection sphere Slides courtesy of Jungjin Lee, KAIST

39 DEFORMABLE SPHERICAL PROJECTION SURFACE Recovered 3D point Slides courtesy of Jungjin Lee, KAIST

40 DEFORMABLE SPHERICAL PROJECTION SURFACE Recovered 3D point Slides courtesy of Jungjin Lee, KAIST

41 DEFORMABLE SPHERICAL PROJECTION SURFACE Slides courtesy of Jungjin Lee, KAIST

42 Comparisons with two state-of-the-art stitching methods GCW: per pixel warping using optical flow [Perazzi et al. 2015] STCPW: mesh-based warping using sparse feature points [Jiang and Gu 2015] Slides courtesy of Jungjin Lee, KAIST

43 NON-UNIFORM RAY SAMPLING Slides courtesy of Jungjin Lee, KAIST

44 NON-UNIFORM RAY SAMPLING Slides courtesy of Jungjin Lee, KAIST

45 Slides courtesy of Jungjin Lee, KAIST

46 Comparisons with the equirectangular projection Rendering resolution: 2K Resolution of the viewing screen: full HD Slides courtesy of Jungjin Lee, KAIST

47 360 Video Processing What can t What is 360 How do we How do we What can we we do with video? represent it? create it? do with it? it? 360 VIDEO STABILIZATION [Johannes Kopf, SIGGRAPH Asia 2016]

48 Slides courtesy of Johannes Kopf, Facebook

49 Slides courtesy of Johannes Kopf, Facebook

50 TRACK FIT MOTION MODEL + SMOOTHING WARP Narrow FOV 2D or 3D Cropped Slides courtesy of Johannes Kopf, Facebook

51 TRACK FIT MOTION MODEL + SMOOTHING WARP Full 360 Hybrid 3D-2D Not Cropped Slides courtesy of Johannes Kopf, Facebook

52 Input video Pure-rotation stabilization Slides courtesy of Johannes Kopf, Facebook

53 Translation effects Rolling shutter/lens deformation

54 Pure-rotation Deformed-rotation Slides courtesy of Johannes Kopf, Facebook

55 CREATING NARROW FOV VIDEOS FROM 360 VIDEO Pano2Vid: Automatic Cinematography for Watching 360 Video [Su et al. ACCV 2016]

56 PANO2VID Input: 360 Video Output: normal-field-of-view (NFOV) Video Slides courtesy of Yu-Chuan Su, UT Austin

57 PANO2VID z C3D [Tran 2015] Train Ω x y ST-glimpses Slides courtesy of Yu-Chuan Su, UT Austin

58 PANO2VID z C3D [Tran 2015] Train Ω x y ST-glimpses Test Evaluate capture-worthiness Construct trajectory Slides courtesy of Yu-Chuan Su, UT Austin

59 HUMAN VALIDATION Slides courtesy of Yu-Chuan Su, UT Austin

60 HUMAN VALIDATION # videos Total length 360 videos hours HumanCam 9, hours AutoCam performs the best Justified by multiple metrics Slides courtesy of Yu-Chuan Su, UT Austin

61 Making 360 Video Watchable in 2D: Learning Videography for Click Free Viewing [Su and Grauman CVPR 2017]

62 FUTURE PROBLEMS What can t What is 360 How do we How do we What can we we do with video? represent it? create it? do with it? it?

63 PERCEPTION Gaze Visualization for Immersive Video Thomas Löwe, Michael Stengel, Emmy-Charlotte Förster, Steve Grogorick, Marcus Magnor Eye Tracking and Visualization (Springer), 2017 Saliency in VR: How do people explore virtual environments? Vincent Sitzmann, Ana Serrano, Amy Pavel, Maneesh Agrawala, Diego Gutierrez, Gordon Wetzstein. Arxiv 2016

64 CINEMATOGRAPHY RULES Constraint on camera motion Zooms Cuts (coming up) Lighting

65 SPATIAL AUDIO Core Sound TetraMic

66 360 VIDEO Oliver Wang Adobe Research