360 video stabilization

Transcription

1 360 video stabilization PÉTER BODNÁR DEPARTMENT OF IMAGE PROCESSING AND COMPUTER GRAPHICS UNIVERSITY OF SZEGED, HUNGARY

2 Outline About 360 video technology Stabilization: motivation and overview The video stabilization process 2D/3D/Hybrid models Keyframe selection Tracking Rotation estimation Warping Implementation, evaluation Limitations, outlook

3 About 360 degree videos Also known as spherical video, immersive video Recorded with one (omnidirectional) or more cameras (camera rig) Viewer can control the viewing direction during playback Good for action sports (biking, skiing), virtual tours, adult videos

4 Demo (Project Luna)

5 Creation Device options: One camera with multiple lenses More cameras with overlapping angles Most commonly in 4K resolution Output: one spherical video (camera feeds stitched and synchronized)

6 Devices GoPro: since 2002, action cameras, video editing software Omni: rig of 6 HERO cameras ( 5 400) Fusion: 5.2K fully spherical videos, release: september Nokia OZO 2015., (professional fimmakers) 8 lenses, FOV: 195 degrees Samsung Gear 2016., $ fisheye lenses Facebook Surround wide angle lens, 3 fisheye, global shutter N/A Others: LUNA, Ricoh Theta

7 Stitching Multiple (overlapping) images (video frames) panorama Done by camera or software: VideoStitch Kolor: Autopano SkyStitch Steps: Feature detection (corners, blobs, Difference of Gaussian of Harris corners) Registration of images Compositing (single shot, surface) Blending (color/exposure correction, etc)

8

9 Playback Monoscopic (2D) video Direction typically controlled with mouse/touch Stereoscopic (3D) video Google Cardboard (smartphone holder) Oculus

10 Publishing Platforms Youtube (Google), March Oculus (Facebook), September Video making Affordable devices (Samsung Gear 360) Availability of stitching software Video editing made easy Platforms for sharing

11 Comparison: 3D, 360, VR 3D video Depth of field Limited viewpoint 360 video No depth of field Control of viewing direction VR: Mobility Interaction

12 VR sickness flight simulator sickness General discomfort, headache, nausea, vomiting, disorientation Causes Video shaking (hand-held cameras) Too much time spent in VR Personal health conditions Solution Reduce camera shaking more comfortable to watch Make videos smoother in time

13 Stabilization Estimate camera position (x,y,z) during the video Calculate rotation for each video frame Recognize undesired rotations (e.g. shaking) Calculate an inverse model to remove undesired rotations Produce the stabilized video

14 2D video stabilization Feature detection in 2D, narrow Field of View (FOV) Feature matching, low-dimensional similarity, homography transformation estimation Inverse transformation

15 3D stabilization Feature detection on image or in 3D space Estimation of 3D transformation (transformation matrix or relaxed model) Inverse transformation

16 2D: solutions Homography transformation: 1998, Morimoto and Chelappa Cannot handle parallax Mesh warps: 2011, Liu et al. Smoothed flow fields: 2014, Liu et al. Not easy to adapt to 3D spherical video

17 3D: solutions 3D model trajectory and scene estimation Buehler et al, Bhat et al, Liu et al, Kopf et al, Drawbacks Complex process, slow Too specific to some situations

18 Facebook: Hybrid model 3D-2D algorithm Uses deformed rotational model Handles parallax 3D can distinguish between translation and rotation movement 2D fast execution Preserves desired rotations in scene 10-20% file size reduction

19 Hybrid algorithm: overview Feature tracking Keyframe selection Keyframe rotation estimation Keyframe rotation correction Inner frame optimization Relaxation of the model Warping

20 Video frame representation for tracking

21

22 Feature tracking Implemented in cube faces Luminance channel (HSV or YCbCr/Yuv) Tracking by Lukas-Kanade method Points that leave a cube face are dropped Tracked points are converted from 2D locations to 3D unit vectors Further steps are processed on those vectors

23 Lukas-Kanade method Optical flow estimation Sparse features Basic equations for selected pixels and neighborhood Least squares criterion Implemented in OpenCV, MATLAB

24 OpenCV library CV: Computer Vision Open-source Up-to-date Available for many platforms (C++, Java, Python, Android) Modular (Core, GUI, Image Processing, Feature extraction, Machine learning, video) Supports CUDA, OpenGL Well-documented (

25 Shi-Tomasi corner detection Moravec corner detection Harris corner detection Shi-Tomasi (Kanade-Tomasi) Image I, patch (u,v) shifted by (x,y), weighted sum of squared differences S: Taylor expansion of I(u+x, v+y) using partial derivatives: Matrix form:

26 Shi-Tomasi corner detection Eigenvalues show presence of corners as:

27 Keyframe selection Detect features (corners) for each frame First and last frame of the video are keyframes For each frame: Detect features Match features for previous keyframe Count matched points If matched points are less than 50%, mark frame as new keyframe If some time is passed, mark frame as keyframe

28 Rotation estimation Obtain the previously matched point pairs for consecutive key frames Estimate the rotation using Nister s five-point correspondance Use RANSAC procedure Implemented in OpenCV and OpenGV libraries

29 About rotation matrices A matrix that rotates a point in Eucledean space Express rotation around individual axes Consecutive rotations matrix multiplication Rotation between keyframes = rotations for each frame multiplied consecutively Correction of keyframe rotation = inverse of the rotation matrix

30 RANSAC Random Sample Consensus Iterative method for mathematical model creation Inliers: a set of data that makes a model Outliers: data not fitting on the model

31 RANSAC - Algorithm Select a subset of the original data (hypothetical inliers) Fit the model Evaluate the fitted model on all data, create the set of inliers (consensus set) Check the size of the consensus set Repeat fixed number of times Re-estimate the model with best fit

32 RANSAC Advantages and disadvantages Advantages Robust estimation Works with contaminated sets (inliers less than 50%) Has solution for disadvantages (optimal RANSAC) Disadvantages No upper bound on time (exhaustive search) May not produce optimal result

33 Point correspondance, Nister Camera calibration Finding the Essential matrix between corresponding points in images Defines a constraint, assumes pinhole camera model Can be used for pose recovery of calibrated cameras Included in OpenCV Pose recovery Essential matrix decomposition Multiple solutions, cheirality check (positive depth of the triangulated points) Included in OpenCV

34 The OpenGV library Geometric Vision C++ library for calibrated camera pose computation Includes many algorithms + benchmark MATLAB, Python support

35 Inner frame trajectories Linear interpolation between keyframes good initial solution Optimization of the length and smoothness of features

36 The Ceres library Non-linear optimalization and modeling Open-source, C++ Users: Google Microsoft Blender

37 Deformed rotation model

38 About barycentric coordinates Introduced in Also known as areal coordinates Positive within the convex hull X 1 X n : vertices of a simplex in affine space (a 1 + +a n )p = a 1 x a n x n a a n = 1

39 Deformed rotation model

40 Summary

41 Advantages Accuracy: 3D analysis estimates keyframe rotation, separates rotation and translation Robustness: 3D estimation is reliable between keyframes Regularization: Keyframes provide the backbone Deformed rotation model gives smoother result Free from artifacts Speed: Rapid convergence of the 2D inner frame optimization Faster than normal playback

42 Image warping 360 video no cropping Equirectangular equirectangular Rotation estimation: the new position of the pixel Inverting the model: get the value to each position (x,y) from (u,v)

43 Rotation filtering Rotation estimation and removal camera orientation does not change Not all rotations need removal

44 Demo (GoPro Hero5 camera)

45 Two-phase tracking Some patches have high distortion on shaky videos Some features are leaving a specific cube face from the cube map Proposal: repeat tracking on stabilized video further boost of smoothness

46 Hyperlapse Increased speed of videos (typically 3-5x) Rotation estimation and correction is highly recommended Simplest speedup solution: frame dropping More sophisticated solution: Estimate camera velocity Make it smooth: estimate motion vector magnitudes, median + low pass filter, timestamp remap for video frames

47 Demo (Microsoft Hyperlapse)

48 Effect on bitrate Original video can be recovered from modified 360 viewers also apply transformation no additional computation required Bitrate is reduced less rotation between encoded frames

49 Quantitative evaluation (Facebook)

50 Limitations High frequency rolling shutter Wobbling due to the deformed model Shots are not handled, unpredictable Static overlays, logos Person filming the scene

51 Space for improvement Sparse tracking dense tracking Keyframe selection Other methods for rotation estimation and filtering Other representation for frames Using color information

52 References Facebook video stabilization: Devices: Image processing and optimalization libraries: Johannes Kopf: 360 video stabilization:

53 Review 1. Which devices are used for 360 video recording? 2. What is video stitching and how it is performed? 3. What is the difference between 3D video, 360 video and VR? 4. What is video stabilization and what are the steps? 5. What are the main features of OpenCV?