FAST: A Framework to Accelerate Super- Resolution Processing on Compressed Videos

Transcription

1 FAST: A Framework to Accelerate Super- Resolution Processing on Compressed Videos Zhengdong Zhang, Vivienne Sze Massachusetts Institute of Technology

2 1 Super-Resolution on Mobile Devices Mobile phones are everywhere Screens are getting larger Run Super-Resolution to Improve the Viewing Experience of Lower-Resolution Content

3 2 Challenges High input resolution 2K 4K High-power 180 W 94 o C Low performance Embedded GPU

4 3 Research Goal SR algorithm FAST SR 15x faster Compressed video Real-time A framework that accelerates any SR algorithm by up to 15x when running on compressed videos

5 4 Using Free Information from Compressed Videos Decode Pixels Block-structure Motion-compensation Compressed video Video as a stack of pixels Representation from compressed video This representation can help SIGNIFICANTLY ACCELERATE super-resolution

6 5 Reviewing Motion Compensation Motion compensation (x, y ) (x, y) Frame 1 Frame 2 Predictor Residual Reconstruction Motion Vector (MV) (x, y) (x, y ) Encode Block info Motion vector Residual Available For FREE

7 6 Transferring the Super-Resolution Results Paste Bicubic High Res Frame 1 High Res Frame 2 SR Algorithm f sr Low Res Frame 1 Low Res Frame 2 motion vector

8 7 Transfer is Lightweight SR SR SRSR SR SR Transfer Low-res video High-res video Low-res video High-res video Transfer allows SR to run on only a subset of frames Fractional Interpolation Bicubic Interpolation Skip Flag The complexity of the transfer is comparable to bicubic interpolation. Transfer N frames, accelerate N times

9 8 Challenge 1: Scene Transition Transfer? Transfer will NOT work if there is a transition of scenes Group-of-Picture (GoP) Structure GoP structure in the compressed video provides video segmentation for free

10 9 Challenge 2: Non-Optimal Prediction by Encoder I(x) I(x) I(x) encode Ground-truth x Low res prediction x Low res residual Low res input x Non-adaptive I(x) sharp SR algorithm x High res prediction I(x) High res residual Bicubic interpolation blur x reconstruct I(x) ringing artifact x High res output FAST stops the transfer on blocks with large residual Adaptive

11 10 Challenge 3: Thresholding the Accumulated Error transfer transfer transfer When a SR result gets transferred multiple times, the error accumulates Bicubic FAST estimates the accumulated error as the accumulated Laplacian of the residual, and stops the transfer when it exceeds a threshold

12 11 Challenge 4: Non-overlapping Blocks and Block Artifacts SRCNN No deblocking With deblocking PSNR SRCNN no deblock deblock FAST applies the deblocking filter to alleviate the blocking effect caused by non-overlapping block division

13 12 Evaluation: Accelerating SRCNN PartyScene RaceHorse BasketballPass Examples of videos in the test set (20 videos for HEVC development) PSNR with 4x acceleration GOP = SRCNN SRCNN with FAST Bicubic PSNR with 16x acceleration GOP = SRCNN SRCNN with FAST Bicubic 4 acceleration with NO PSNR LOSS. 16 acceleration with 0.2 db loss of PSNR

14 13 Visual Evaluation Bicubic SRCNN SRCNN with FAST Ground-truth

15 14 Visual Evaluation SRCNN FAST + SRCNN Bicubic

16 15 Related Work SRCNN (Dong et, al. ECCV 14) Faster SRCNN (Dong et, al. ECCV 16) Our framework can accelerate existing SR algorithms. Video Propagation Network (Jampani et, al. CVPR 17) Spatio-Temporal Networks and Motion Compensation (Caballero et, al., Arxiv 16) More efficient transfer using compressed video information

17 16 Contributions Transfer the SR results guided by motion vector Adaptively perform the transfer by thresholding on the residue, and accumulated Laplacian Accelerating most of the SR algorithm by up to 15x running on videos with minimal PSNR loss SR algorithm FAST SR 15x faster Real-time Compressed video