FAST: A Framework to Accelerate Super- Resolution Processing on Compressed Videos Zhengdong Zhang, Vivienne Sze Massachusetts Institute of Technology http://www.mit.edu/~sze/fast.html
1 Super-Resolution on Mobile Devices Mobile phones are everywhere Screens are getting larger Run Super-Resolution to Improve the Viewing Experience of Lower-Resolution Content
2 Challenges High input resolution 2K 4K High-power 180 W 94 o C Low performance Embedded GPU
3 Research Goal SR algorithm FAST SR 15x faster Compressed video Real-time A framework that accelerates any SR algorithm by up to 15x when running on compressed videos
4 Using Free Information from Compressed Videos Decode Pixels Block-structure Motion-compensation Compressed video Video as a stack of pixels Representation from compressed video This representation can help SIGNIFICANTLY ACCELERATE super-resolution
5 Reviewing Motion Compensation Motion compensation (x, y ) (x, y) Frame 1 Frame 2 Predictor Residual Reconstruction Motion Vector (MV) (x, y) (x, y ) Encode 0100110110 1101100111 Block info 0111111001 Motion vector 0010111000 1011001000 Residual 101011 Available For FREE
6 Transferring the Super-Resolution Results Paste Bicubic High Res Frame 1 High Res Frame 2 SR Algorithm f sr Low Res Frame 1 Low Res Frame 2 motion vector
7 Transfer is Lightweight SR SR SRSR SR SR Transfer Low-res video High-res video Low-res video High-res video Transfer allows SR to run on only a subset of frames Fractional Interpolation Bicubic Interpolation Skip Flag The complexity of the transfer is comparable to bicubic interpolation. Transfer N frames, accelerate N times
8 Challenge 1: Scene Transition Transfer? Transfer will NOT work if there is a transition of scenes Group-of-Picture (GoP) Structure GoP structure in the compressed video provides video segmentation for free
9 Challenge 2: Non-Optimal Prediction by Encoder I(x) I(x) I(x) encode Ground-truth x Low res prediction x Low res residual Low res input x Non-adaptive I(x) sharp SR algorithm x High res prediction I(x) High res residual Bicubic interpolation blur x reconstruct I(x) ringing artifact x High res output FAST stops the transfer on blocks with large residual Adaptive
10 Challenge 3: Thresholding the Accumulated Error transfer transfer transfer When a SR result gets transferred multiple times, the error accumulates Bicubic FAST estimates the accumulated error as the accumulated Laplacian of the residual, and stops the transfer when it exceeds a threshold
11 Challenge 4: Non-overlapping Blocks and Block Artifacts SRCNN No deblocking With deblocking PSNR 32.9 32.8 32.7 32.87 32.84 32.6 32.5 32.53 32.4 32.3 SRCNN no deblock deblock FAST applies the deblocking filter to alleviate the blocking effect caused by non-overlapping block division
12 Evaluation: Accelerating SRCNN PartyScene RaceHorse BasketballPass Examples of videos in the test set (20 videos for HEVC development) 31.5 31 30.5 30 29.5 29 PSNR with 4x acceleration GOP = 4 31.04 31.04 SRCNN SRCNN with FAST 29.87 Bicubic 31 30.5 30 29.5 29 PSNR with 16x acceleration GOP = 16 30.89 SRCNN 30.65 SRCNN with FAST 29.77 Bicubic 4 acceleration with NO PSNR LOSS. 16 acceleration with 0.2 db loss of PSNR
13 Visual Evaluation Bicubic SRCNN SRCNN with FAST Ground-truth
14 Visual Evaluation SRCNN FAST + SRCNN Bicubic
15 Related Work SRCNN (Dong et, al. ECCV 14) Faster SRCNN (Dong et, al. ECCV 16) Our framework can accelerate existing SR algorithms. Video Propagation Network (Jampani et, al. CVPR 17) Spatio-Temporal Networks and Motion Compensation (Caballero et, al., Arxiv 16) More efficient transfer using compressed video information
16 Contributions Transfer the SR results guided by motion vector Adaptively perform the transfer by thresholding on the residue, and accumulated Laplacian Accelerating most of the SR algorithm by up to 15x running on videos with minimal PSNR loss SR algorithm FAST SR 15x faster Real-time Compressed video