Bidirectional Recurrent Convolutional Networks for Video Super-Resolution

Size: px
Start display at page:

Download "Bidirectional Recurrent Convolutional Networks for Video Super-Resolution"

Transcription

1 Bidirectional Recurrent Convolutional Networks for Video Super-Resolution Qi Zhang & Yan Huang Center for Research on Intelligent Perception and Computing (CRIPAC) National Laboratory of Pattern Recognition (NLPR) Institute of Automation, Chinese Academy of Sciences (CASIA) May 10, 2017

2 2 CRIPAC CRIPAC mainly focuses on the following research topics related to national public security. Biometrics Image and Video Analysis Big Data and Multi-modal Computing Content Security and Authentication Sensing and Information Acquisition CRAPIC receives regular fundings from various Government departments or agencies. It is also supported by funds of R&D projects from many other national and international sources. CRIPAC members publish widely in leading national and international journals and conferences such as IEEE Transactions on PAMI, IEEE Transactions on Image Processing, International Journal of Computer Vision, Pattern Recognition, Pattern Recognition Letters, ICCV, ECCV, CVPR, ACCV, ICPR, ICIP, etc.

3 3 NVAIL Artificial Intelligence Laboratory Researches on artificial intelligence and deep learning

4 4 Outline 1 Deep Learning 2 Recurrent Convolutional Networks 3 Application to Video Super-Resolution 4 Future Work

5 5 Outline 1 Deep Learning 2 Recurrent Convolutional Networks 3 Application to Video Super-Resolution 4 Future Work

6 6 Deep Neural Networks (DNN) Originate from: simple/complex cell, Hubel and Wiesel efficient error backpropagation, Linnainmaa deep neocognitron, convolution, Fukushima autoencoder, Ballard backpropagation for CNN, Lecun fundamental deep learning problem, Hochreiter deep recurrent neural network, Schmidhuber supervised LSTM RNN, Schmidhuber Two drawbacks: Large numbers of parameters High computational cost Small training set Over-fitting problem

7 Two Recent Developments Big Data Cheap Computation Video surveillance data size (PB) DNN can thus be fitted efficiently 7

8 Deep Learning The Resurgence of DNN Breakthrough in 2006 ImageNet: 74% vs. 85% RNN for sequence analysis Activity recognition, CVPR2015 Video caption, CVPR2015 Deep Learning promotes the fast development areas2014 of various visual computing Representation learning CNN for visual tasks DeepFace, CVPR2014 RCNN for detection, CVPR2014 8

9 9 Outline 1 Deep Learning 2 Recurrent Convolutional Networks 3 Application to Video Super-Resolution 4 Future Work

10 10 Deep Neural Networks (DNN) y x R d, h R n, W R d n h = σ xw, σ t = 1 1+e t h W x Sigmoid function σ t

11 11 Recurrent Neural Networks (RNN) y Temporal dependency modeling y h h 1 U h 2 U h 3 W W W W x x 1 x 2 x 3 DNN RNN x t R d, h t R n, W R d n, U R n n h t = σ x t W + h t 1 U

12 12 Recurrent Convolutional Networks (RCN) DNN: Deep Neural Networks RNN: Recurrent Neural Networks CNN: Convolutional Neural Networks DNN CNN Convolutional Sequential Sequential RNN RCN Convolutional

13 13 Applications of RCN Video SR, NIPS15 & TPAMI17 Scene Labeling, NIPS15 Weather Nowcasting, NIPS15 Action Recognition, ICLR15 Object Recognition, CVPR15 Person ReID, CVPR16

14 14 Outline 1 Deep Learning 2 Recurrent Convolutional Networks 3 Application to Video Super-Resolution 4 Future Work

15 Video Super-Resolution Display High-resolution devices High-resolution videos Display Super-resolution: denoising, deblurring, upscaling Low-resolution videos A great need for super resolving low-resolution videos 15

16 Two Main Approaches (1/2) 1. Single-Image super-resolution [1-6] One-to-One scheme, super resolve each video frame independently Ignore the intrinsic temporal dependency relation of video frames Low computational complexity, fast [1] Dong et al., Learning a deep convolutional network for image super resolution. ECCV, [2] Timofte et al., Anchored neighborhood regression for fast example-based super resolution. ICCV, [3] Zeyde et al., On single image scale-up using sparse-representations. Curves and Surfaces, [4] Yang et al., Image super-resolution via sparse representation. IEEE TIP, [5] Bevilacqua et al., Low-complexity single-image super resolution. BMVC, [6] Chang et al., Super-resolution through neighbor embedding. CVPR,

17 Two Main Approaches (2/2) 2. Multi-Frame super-resolution [7-11] Many-to-One scheme, use multiple adjacent frames to super resolve a frame Model the temporal dependency relation by motion estimation High computational complexity, slow [7] Liu and Sun, On bayesian adaptive video super resolution. IEEE PAMI, [8] Takeda et al., Super-resolution without explicit subpixel motion estimation. IEEE TIP, [9] Mitzel et al., Video super resolution using duality based tv-l 1 optical flow. PR, [10] Protter et al. Generalizing the nonlocal-means to super-resolution reconstruction. IEEE TIP, [11] Fransens et al., Optical flow based super-resolution: A probabilistic approach. CVIU,

18 Motivation RNN: Recurrent Neural Networks SR: Super-Resolution RNN can model long-term contextual information of temporal sequences well Convolutional operation can scale to full videos of any spatial size and temporal step Propose bidirectional recurrent convolutional networks, different from vanilla RNN: 1. Commonly-used full connections are replaced with weight -sharing convolutions 2. Conditional convolutions are added for learning visual-temporal dependency relation 18

19 19 Bidirectional Recurrent Convolutional Networks learn spatial dependency between a low-resolution frame and its highresolution result model long-term temporal dependency relation across video frames enhance visual-temporal dependency relation modeling

20 Learning Define an end-to-end mapping O from low-resolution frames X to high-resolution frames Y Learning proceeds by optimizing the Mean Square Error (MSE) between predicted frames O(X) and Y stochastic gradient descent L = O X Y 2 small learning rate in the output layer: 1e-4 20

21 Experiments Train the model on 25 YUV format video sequences volume-based training number of volumes: roughly 41,000 volume size: Test on a variety of real world videos severe motion blur motion aliasing complex motions Training videos Testing videos 21

22 PSNR Comparison PSNR: peak signal-to-noise ratio Table1: The results of PSNR (db) and test time (sec) on the test video sequences. Surpass state-of-the-art methods in PSNR, due to the effective [1] Video enhancer. version [4] Bevilacqua et al., Low-complexity single-image super resolution. BMVC, [5] Chang et al., Super-resolution through neighbor embedding. CVPR, [6] Dong temporal et al., Learning a dependency deep convolutional network modelling for image super resolution. ECCV, [20] Takeda et al., Super-resolution without explicit subpixel motion estimation. IEEE TIP, [22] Timofte et al., Anchored neighborhood regression for fast example-based super resolution. ICCV, [24] Yang et al., Image super-resolution via sparse representation. IEEE TIP, [25] Zeyde et al., On single image scale-up using sparse-representations. Curves and Surfaces,

23 23 Model Architecture Investigate the impact of our model architecture on the performance Take a simplified network containing only feedfoward (v) convolution as a benchmark Study its variants by successively adding the bidirectional (b), recurrent (r)and conditional (t) schemes Table1: The results of PSNR (db) by variants of BRCN on the testing video sequences.

24 24 Running Time Figure: Speed vs. PSNR for all the comparison methods. Outperform both single-image and multi-frame SR methods Achieve comparable speed with the fastest single-image SR methods

25 25 Closeup Comparison Our method is able to recover more image details than others, Figure: Comparison among original frames (2th, 3th and 4th frames, from the top row to the bottom) of the Dancing video and super resolved results by Bicubic, 3DSKR, ANR and BRCN, under respectively. severe motion conditions

26 26 Example Upscaling factor: Comparison: Bicubic (top) Ours (bottom)

27 Conclusion Bidirectional Recurrent Convolutional Networks bidirectional recurrent and conditional convolutions an end-to-end framework, without pre/post-processing well performance and fast speed For more details, please refer to the following papers: 1. Yan Huang, Wei Wang, and Liang Wang, Bidirectional Recurrent Convolutional Networks for Multi-Frame Super-Resolution. Advances in Neural Information Processing Systems (NIPS), pp , Yan Huang, Wei Wang, and Liang Wang, Video Super-Resolution via Bidirectional Recurrent Convolutional Networks, IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2017, Accepted 27

28 28 Outline 1 Deep Learning 2 Recurrent Convolutional Networks 3 Application to Video Super-Resolution 4 Future Work

29 Future Work For performance improvement extend our model to have a deeper architecture, e.g., based on 19 layers VGG net incorporate some effective strategies, e.g., motion ensemble and residual connection For speed acceleration replace the used pre-upsampling by learning diverse upsampling filters with deconvolution layers Others collect a large-scale high-resolution video dataset, and try to learn our model directly from raw videos 29

30 30 Acknowledgement NVAIL Artificial Intelligence Laboratory Sponsor excellent hardware resources

31 THANK YOU

Bidirectional Recurrent Convolutional Networks for Multi-Frame Super-Resolution

Bidirectional Recurrent Convolutional Networks for Multi-Frame Super-Resolution Bidirectional Recurrent Convolutional Networks for Multi-Frame Super-Resolution Yan Huang 1 Wei Wang 1 Liang Wang 1,2 1 Center for Research on Intelligent Perception and Computing National Laboratory of

More information

LSTM and its variants for visual recognition. Xiaodan Liang Sun Yat-sen University

LSTM and its variants for visual recognition. Xiaodan Liang Sun Yat-sen University LSTM and its variants for visual recognition Xiaodan Liang xdliang328@gmail.com Sun Yat-sen University Outline Context Modelling with CNN LSTM and its Variants LSTM Architecture Variants Application in

More information

arxiv: v2 [cs.cv] 11 Nov 2016

arxiv: v2 [cs.cv] 11 Nov 2016 Accurate Image Super-Resolution Using Very Deep Convolutional Networks Jiwon Kim, Jung Kwon Lee and Kyoung Mu Lee Department of ECE, ASRI, Seoul National University, Korea {j.kim, deruci, kyoungmu}@snu.ac.kr

More information

A Novel Multi-Frame Color Images Super-Resolution Framework based on Deep Convolutional Neural Network. Zhe Li, Shu Li, Jianmin Wang and Hongyang Wang

A Novel Multi-Frame Color Images Super-Resolution Framework based on Deep Convolutional Neural Network. Zhe Li, Shu Li, Jianmin Wang and Hongyang Wang 5th International Conference on Measurement, Instrumentation and Automation (ICMIA 2016) A Novel Multi-Frame Color Images Super-Resolution Framewor based on Deep Convolutional Neural Networ Zhe Li, Shu

More information

Machine Learning 13. week

Machine Learning 13. week Machine Learning 13. week Deep Learning Convolutional Neural Network Recurrent Neural Network 1 Why Deep Learning is so Popular? 1. Increase in the amount of data Thanks to the Internet, huge amount of

More information

IMAGE SUPER-RESOLUTION BASED ON DICTIONARY LEARNING AND ANCHORED NEIGHBORHOOD REGRESSION WITH MUTUAL INCOHERENCE

IMAGE SUPER-RESOLUTION BASED ON DICTIONARY LEARNING AND ANCHORED NEIGHBORHOOD REGRESSION WITH MUTUAL INCOHERENCE IMAGE SUPER-RESOLUTION BASED ON DICTIONARY LEARNING AND ANCHORED NEIGHBORHOOD REGRESSION WITH MUTUAL INCOHERENCE Yulun Zhang 1, Kaiyu Gu 2, Yongbing Zhang 1, Jian Zhang 3, and Qionghai Dai 1,4 1 Shenzhen

More information

Introduction. Prior work BYNET: IMAGE SUPER RESOLUTION WITH A BYPASS CONNECTION NETWORK. Bjo rn Stenger. Rakuten Institute of Technology

Introduction. Prior work BYNET: IMAGE SUPER RESOLUTION WITH A BYPASS CONNECTION NETWORK. Bjo rn Stenger. Rakuten Institute of Technology BYNET: IMAGE SUPER RESOLUTION WITH A BYPASS CONNECTION NETWORK Jiu Xu Yeongnam Chae Bjo rn Stenger Rakuten Institute of Technology ABSTRACT This paper proposes a deep residual network, ByNet, for the single

More information

One Network to Solve Them All Solving Linear Inverse Problems using Deep Projection Models

One Network to Solve Them All Solving Linear Inverse Problems using Deep Projection Models One Network to Solve Them All Solving Linear Inverse Problems using Deep Projection Models [Supplemental Materials] 1. Network Architecture b ref b ref +1 We now describe the architecture of the networks

More information

Recovering Realistic Texture in Image Super-resolution by Deep Spatial Feature Transform. Xintao Wang Ke Yu Chao Dong Chen Change Loy

Recovering Realistic Texture in Image Super-resolution by Deep Spatial Feature Transform. Xintao Wang Ke Yu Chao Dong Chen Change Loy Recovering Realistic Texture in Image Super-resolution by Deep Spatial Feature Transform Xintao Wang Ke Yu Chao Dong Chen Change Loy Problem enlarge 4 times Low-resolution image High-resolution image Previous

More information

Convolutional Neural Networks. Computer Vision Jia-Bin Huang, Virginia Tech

Convolutional Neural Networks. Computer Vision Jia-Bin Huang, Virginia Tech Convolutional Neural Networks Computer Vision Jia-Bin Huang, Virginia Tech Today s class Overview Convolutional Neural Network (CNN) Training CNN Understanding and Visualizing CNN Image Categorization:

More information

arxiv: v1 [cs.cv] 3 Jan 2017

arxiv: v1 [cs.cv] 3 Jan 2017 Learning a Mixture of Deep Networks for Single Image Super-Resolution Ding Liu, Zhaowen Wang, Nasser Nasrabadi, and Thomas Huang arxiv:1701.00823v1 [cs.cv] 3 Jan 2017 Beckman Institute, University of Illinois

More information

arxiv: v1 [cs.cv] 8 Feb 2018

arxiv: v1 [cs.cv] 8 Feb 2018 DEEP IMAGE SUPER RESOLUTION VIA NATURAL IMAGE PRIORS Hojjat S. Mousavi, Tiantong Guo, Vishal Monga Dept. of Electrical Engineering, The Pennsylvania State University arxiv:802.0272v [cs.cv] 8 Feb 208 ABSTRACT

More information

Proceedings of the International MultiConference of Engineers and Computer Scientists 2018 Vol I IMECS 2018, March 14-16, 2018, Hong Kong

Proceedings of the International MultiConference of Engineers and Computer Scientists 2018 Vol I IMECS 2018, March 14-16, 2018, Hong Kong , March 14-16, 2018, Hong Kong , March 14-16, 2018, Hong Kong , March 14-16, 2018, Hong Kong , March 14-16, 2018, Hong Kong TABLE I CLASSIFICATION ACCURACY OF DIFFERENT PRE-TRAINED MODELS ON THE TEST DATA

More information

Efficient Module Based Single Image Super Resolution for Multiple Problems

Efficient Module Based Single Image Super Resolution for Multiple Problems Efficient Module Based Single Image Super Resolution for Multiple Problems Dongwon Park Kwanyoung Kim Se Young Chun School of ECE, Ulsan National Institute of Science and Technology, 44919, Ulsan, South

More information

OPTICAL Character Recognition systems aim at converting

OPTICAL Character Recognition systems aim at converting ICDAR 2015 COMPETITION ON TEXT IMAGE SUPER-RESOLUTION 1 Boosting Optical Character Recognition: A Super-Resolution Approach Chao Dong, Ximei Zhu, Yubin Deng, Chen Change Loy, Member, IEEE, and Yu Qiao

More information

Single Image Super-Resolution via Iterative Collaborative Representation

Single Image Super-Resolution via Iterative Collaborative Representation Single Image Super-Resolution via Iterative Collaborative Representation Yulun Zhang 1(B), Yongbing Zhang 1, Jian Zhang 2, aoqian Wang 1, and Qionghai Dai 1,3 1 Graduate School at Shenzhen, Tsinghua University,

More information

DCGANs for image super-resolution, denoising and debluring

DCGANs for image super-resolution, denoising and debluring DCGANs for image super-resolution, denoising and debluring Qiaojing Yan Stanford University Electrical Engineering qiaojing@stanford.edu Wei Wang Stanford University Electrical Engineering wwang23@stanford.edu

More information

Image Super-Resolution Using Dense Skip Connections

Image Super-Resolution Using Dense Skip Connections Image Super-Resolution Using Dense Skip Connections Tong Tong, Gen Li, Xiejie Liu, Qinquan Gao Imperial Vision Technology Fuzhou, China {ttraveltong,ligen,liu.xiejie,gqinquan}@imperial-vision.com Abstract

More information

arxiv: v1 [cs.cv] 14 Jul 2017

arxiv: v1 [cs.cv] 14 Jul 2017 Temporal Modeling Approaches for Large-scale Youtube-8M Video Understanding Fu Li, Chuang Gan, Xiao Liu, Yunlong Bian, Xiang Long, Yandong Li, Zhichao Li, Jie Zhou, Shilei Wen Baidu IDL & Tsinghua University

More information

Multilayer and Multimodal Fusion of Deep Neural Networks for Video Classification

Multilayer and Multimodal Fusion of Deep Neural Networks for Video Classification Multilayer and Multimodal Fusion of Deep Neural Networks for Video Classification Xiaodong Yang, Pavlo Molchanov, Jan Kautz INTELLIGENT VIDEO ANALYTICS Surveillance event detection Human-computer interaction

More information

arxiv: v4 [cs.cv] 25 Mar 2018

arxiv: v4 [cs.cv] 25 Mar 2018 Frame-Recurrent Video Super-Resolution Mehdi S. M. Sajjadi 1,2 msajjadi@tue.mpg.de Raviteja Vemulapalli 2 ravitejavemu@google.com 1 Max Planck Institute for Intelligent Systems 2 Google Matthew Brown 2

More information

arxiv: v1 [cs.cv] 6 Nov 2015

arxiv: v1 [cs.cv] 6 Nov 2015 Seven ways to improve example-based single image super resolution Radu Timofte Computer Vision Lab D-ITET, ETH Zurich timofter@vision.ee.ethz.ch Rasmus Rothe Computer Vision Lab D-ITET, ETH Zurich rrothe@vision.ee.ethz.ch

More information

DENSE BYNET: RESIDUAL DENSE NETWORK FOR IMAGE SUPER RESOLUTION. Bjo rn Stenger2

DENSE BYNET: RESIDUAL DENSE NETWORK FOR IMAGE SUPER RESOLUTION. Bjo rn Stenger2 DENSE BYNET: RESIDUAL DENSE NETWORK FOR IMAGE SUPER RESOLUTION Jiu Xu1 Yeongnam Chae2 1 Bjo rn Stenger2 Ankur Datta1 Rakuten Institute of Technology, Boston Rakuten Institute of Technology, Tokyo 2 ABSTRACT

More information

COMP 551 Applied Machine Learning Lecture 16: Deep Learning

COMP 551 Applied Machine Learning Lecture 16: Deep Learning COMP 551 Applied Machine Learning Lecture 16: Deep Learning Instructor: Ryan Lowe (ryan.lowe@cs.mcgill.ca) Slides mostly by: Class web page: www.cs.mcgill.ca/~hvanho2/comp551 Unless otherwise noted, all

More information

CNN for Low Level Image Processing. Huanjing Yue

CNN for Low Level Image Processing. Huanjing Yue CNN for Low Level Image Processing Huanjing Yue 2017.11 1 Deep Learning for Image Restoration General formulation: min Θ L( x, x) s. t. x = F(y; Θ) Loss function Parameters to be learned Key issues The

More information

An Attention-Based Approach for Single Image Super Resolution

An Attention-Based Approach for Single Image Super Resolution An Attention-Based Approach for Single Image Super Resolution Yuan Liu 1,2,3, Yuancheng Wang 1, Nan Li 1,Xu Cheng 4, Yifeng Zhang 1,2,3,, Yongming Huang 1, Guojun Lu 5 1 School of Information Science and

More information

UDNet: Up-Down Network for Compact and Efficient Feature Representation in Image Super-Resolution

UDNet: Up-Down Network for Compact and Efficient Feature Representation in Image Super-Resolution UDNet: Up-Down Network for Compact and Efficient Feature Representation in Image Super-Resolution Chang Chen Xinmei Tian Zhiwei Xiong Feng Wu University of Science and Technology of China Abstract Recently,

More information

Seven ways to improve example-based single image super resolution

Seven ways to improve example-based single image super resolution Seven ways to improve example-based single image super resolution Radu Timofte CVL, D-ITET, ETH Zurich radu.timofte@vision.ee.ethz.ch Rasmus Rothe CVL, D-ITET, ETH Zurich rrothe@vision.ee.ethz.ch Luc Van

More information

Multimodal Gesture Recognition using Multi-stream Recurrent Neural Network

Multimodal Gesture Recognition using Multi-stream Recurrent Neural Network Multimodal Gesture Recognition using Multi-stream Recurrent Neural Network Noriki Nishida and Hideki Nakayama Machine Perception Group Graduate School of Information Science and Technology The University

More information

Deep Networks for Image Super-Resolution with Sparse Prior

Deep Networks for Image Super-Resolution with Sparse Prior Deep Networks for Image Super-Resolution with Sparse Prior Zhaowen Wang Ding Liu Jianchao Yang Wei Han Thomas Huang Beckman Institute, University of Illinois at Urbana-Champaign, Urbana, IL Adobe Research,

More information

Yiqi Yan. May 10, 2017

Yiqi Yan. May 10, 2017 Yiqi Yan May 10, 2017 P a r t I F u n d a m e n t a l B a c k g r o u n d s Convolution Single Filter Multiple Filters 3 Convolution: case study, 2 filters 4 Convolution: receptive field receptive field

More information

Learning a Deep Convolutional Network for Image Super-Resolution

Learning a Deep Convolutional Network for Image Super-Resolution Learning a Deep Convolutional Network for Image Super-Resolution Chao Dong 1, Chen Change Loy 1, Kaiming He 2, and Xiaoou Tang 1 1 Department of Information Engineering, The Chinese University of Hong

More information

Pixel-level Generative Model

Pixel-level Generative Model Pixel-level Generative Model Generative Image Modeling Using Spatial LSTMs (2015NIPS) L. Theis and M. Bethge University of Tübingen, Germany Pixel Recurrent Neural Networks (2016ICML) A. van den Oord,

More information

Fast and Accurate Image Super-Resolution Using A Combined Loss

Fast and Accurate Image Super-Resolution Using A Combined Loss Fast and Accurate Image Super-Resolution Using A Combined Loss Jinchang Xu 1, Yu Zhao 1, Yuan Dong 1, Hongliang Bai 2 1 Beijing University of Posts and Telecommunications, 2 Beijing Faceall Technology

More information

arxiv: v1 [cs.cv] 7 May 2018

arxiv: v1 [cs.cv] 7 May 2018 arxiv:1805.02335v1 [cs.cv] 7 May 2018 Skeleton-Based Action Recognition with Spatial Reasoning and Temporal Stack Learning Chenyang Si 1,3, Ya Jing 1,3, Wei Wang 1,3,, Liang Wang 1,2,3, and Tieniu Tan

More information

Recurrent Convolutional Neural Networks for Scene Labeling

Recurrent Convolutional Neural Networks for Scene Labeling Recurrent Convolutional Neural Networks for Scene Labeling Pedro O. Pinheiro, Ronan Collobert Reviewed by Yizhe Zhang August 14, 2015 Scene labeling task Scene labeling: assign a class label to each pixel

More information

arxiv: v2 [cs.cv] 14 May 2018

arxiv: v2 [cs.cv] 14 May 2018 ContextVP: Fully Context-Aware Video Prediction Wonmin Byeon 1234, Qin Wang 1, Rupesh Kumar Srivastava 3, and Petros Koumoutsakos 1 arxiv:1710.08518v2 [cs.cv] 14 May 2018 Abstract Video prediction models

More information

Image Upscaling and Fuzzy ARTMAP Neural Network

Image Upscaling and Fuzzy ARTMAP Neural Network IOSR Journal of Computer Engineering (IOSR-JCE) e-issn: 2278-0661,p-ISSN: 2278-8727, Volume 17, Issue 4, Ver. II (July Aug. 2015), PP 79-85 www.iosrjournals.org Image Upscaling and Fuzzy ARTMAP Neural

More information

Structured Prediction using Convolutional Neural Networks

Structured Prediction using Convolutional Neural Networks Overview Structured Prediction using Convolutional Neural Networks Bohyung Han bhhan@postech.ac.kr Computer Vision Lab. Convolutional Neural Networks (CNNs) Structured predictions for low level computer

More information

Multi-Input Cardiac Image Super-Resolution using Convolutional Neural Networks

Multi-Input Cardiac Image Super-Resolution using Convolutional Neural Networks Multi-Input Cardiac Image Super-Resolution using Convolutional Neural Networks Ozan Oktay, Wenjia Bai, Matthew Lee, Ricardo Guerrero, Konstantinos Kamnitsas, Jose Caballero, Antonio de Marvao, Stuart Cook,

More information

Learning Visual Semantics: Models, Massive Computation, and Innovative Applications

Learning Visual Semantics: Models, Massive Computation, and Innovative Applications Learning Visual Semantics: Models, Massive Computation, and Innovative Applications Part II: Visual Features and Representations Liangliang Cao, IBM Watson Research Center Evolvement of Visual Features

More information

Spatial Localization and Detection. Lecture 8-1

Spatial Localization and Detection. Lecture 8-1 Lecture 8: Spatial Localization and Detection Lecture 8-1 Administrative - Project Proposals were due on Saturday Homework 2 due Friday 2/5 Homework 1 grades out this week Midterm will be in-class on Wednesday

More information

A Novel Image Super-resolution Reconstruction Algorithm based on Modified Sparse Representation

A Novel Image Super-resolution Reconstruction Algorithm based on Modified Sparse Representation , pp.162-167 http://dx.doi.org/10.14257/astl.2016.138.33 A Novel Image Super-resolution Reconstruction Algorithm based on Modified Sparse Representation Liqiang Hu, Chaofeng He Shijiazhuang Tiedao University,

More information

Study of Residual Networks for Image Recognition

Study of Residual Networks for Image Recognition Study of Residual Networks for Image Recognition Mohammad Sadegh Ebrahimi Stanford University sadegh@stanford.edu Hossein Karkeh Abadi Stanford University hosseink@stanford.edu Abstract Deep neural networks

More information

Deep Learning. Deep Learning. Practical Application Automatically Adding Sounds To Silent Movies

Deep Learning. Deep Learning. Practical Application Automatically Adding Sounds To Silent Movies http://blog.csdn.net/zouxy09/article/details/8775360 Automatic Colorization of Black and White Images Automatically Adding Sounds To Silent Movies Traditionally this was done by hand with human effort

More information

Deep learning for object detection. Slides from Svetlana Lazebnik and many others

Deep learning for object detection. Slides from Svetlana Lazebnik and many others Deep learning for object detection Slides from Svetlana Lazebnik and many others Recent developments in object detection 80% PASCAL VOC mean0average0precision0(map) 70% 60% 50% 40% 30% 20% 10% Before deep

More information

A DEEP DICTIONARY MODEL FOR IMAGE SUPER-RESOLUTION. Jun-Jie Huang and Pier Luigi Dragotti

A DEEP DICTIONARY MODEL FOR IMAGE SUPER-RESOLUTION. Jun-Jie Huang and Pier Luigi Dragotti A DEEP DICTIONARY MODEL FOR IMAGE SUPER-RESOLUTION Jun-Jie Huang and Pier Luigi Dragotti Communications and Signal Processing Group CSP), Imperial College London, UK ABSTRACT Inspired by the recent success

More information

Asynchronous Parallel Learning for Neural Networks and Structured Models with Dense Features

Asynchronous Parallel Learning for Neural Networks and Structured Models with Dense Features Asynchronous Parallel Learning for Neural Networks and Structured Models with Dense Features Xu SUN ( 孙栩 ) Peking University xusun@pku.edu.cn Motivation Neural networks -> Good Performance CNN, RNN, LSTM

More information

RTSR: Enhancing Real-time H.264 Video Streaming using Deep Learning based Video Super Resolution Spring 2017 CS570 Project Presentation June 8, 2017

RTSR: Enhancing Real-time H.264 Video Streaming using Deep Learning based Video Super Resolution Spring 2017 CS570 Project Presentation June 8, 2017 RTSR: Enhancing Real-time H.264 Video Streaming using Deep Learning based Video Super Resolution Spring 2017 CS570 Project Presentation June 8, 2017 Team 16 Soomin Kim Leslie Tiong Youngki Kwon Insu Jang

More information

Object detection with CNNs

Object detection with CNNs Object detection with CNNs 80% PASCAL VOC mean0average0precision0(map) 70% 60% 50% 40% 30% 20% 10% Before CNNs After CNNs 0% 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 year Region proposals

More information

Boosting face recognition via neural Super-Resolution

Boosting face recognition via neural Super-Resolution Boosting face recognition via neural Super-Resolution Guillaume Berger, Cle ment Peyrard and Moez Baccouche Orange Labs - 4 rue du Clos Courtel, 35510 Cesson-Se vigne - France Abstract. We propose a two-step

More information

Single Image Super Resolution of Textures via CNNs. Andrew Palmer

Single Image Super Resolution of Textures via CNNs. Andrew Palmer Single Image Super Resolution of Textures via CNNs Andrew Palmer What is Super Resolution (SR)? Simple: Obtain one or more high-resolution images from one or more low-resolution ones Many, many applications

More information

Progressive Neural Architecture Search

Progressive Neural Architecture Search Progressive Neural Architecture Search Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, Kevin Murphy 09/10/2018 @ECCV 1 Outline Introduction

More information

DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution and Fully Connected CRFs

DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution and Fully Connected CRFs DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution and Fully Connected CRFs Zhipeng Yan, Moyuan Huang, Hao Jiang 5/1/2017 1 Outline Background semantic segmentation Objective,

More information

Flow-Based Video Recognition

Flow-Based Video Recognition Flow-Based Video Recognition Jifeng Dai Visual Computing Group, Microsoft Research Asia Joint work with Xizhou Zhu*, Yuwen Xiong*, Yujie Wang*, Lu Yuan and Yichen Wei (* interns) Talk pipeline Introduction

More information

Video Compression Using Recurrent Convolutional Neural Networks

Video Compression Using Recurrent Convolutional Neural Networks Video Compression Using Recurrent Convolutional Neural Networks Cedric Yue Sik Kin Electrical Engineering cedyue@stanford.edu Berk Coker Computer Science bcoker@stanford.edu Abstract The demand for video

More information

Artificial Neural Networks. Introduction to Computational Neuroscience Ardi Tampuu

Artificial Neural Networks. Introduction to Computational Neuroscience Ardi Tampuu Artificial Neural Networks Introduction to Computational Neuroscience Ardi Tampuu 7.0.206 Artificial neural network NB! Inspired by biology, not based on biology! Applications Automatic speech recognition

More information

Encoder-Decoder Networks for Semantic Segmentation. Sachin Mehta

Encoder-Decoder Networks for Semantic Segmentation. Sachin Mehta Encoder-Decoder Networks for Semantic Segmentation Sachin Mehta Outline > Overview of Semantic Segmentation > Encoder-Decoder Networks > Results What is Semantic Segmentation? Input: RGB Image Output:

More information

Hallucinating Very Low-Resolution Unaligned and Noisy Face Images by Transformative Discriminative Autoencoders

Hallucinating Very Low-Resolution Unaligned and Noisy Face Images by Transformative Discriminative Autoencoders Hallucinating Very Low-Resolution Unaligned and Noisy Face Images by Transformative Discriminative Autoencoders Xin Yu, Fatih Porikli Australian National University {xin.yu, fatih.porikli}@anu.edu.au Abstract

More information

Computer Vision Lecture 16

Computer Vision Lecture 16 Computer Vision Lecture 16 Deep Learning for Object Categorization 14.01.2016 Bastian Leibe RWTH Aachen http://www.vision.rwth-aachen.de leibe@vision.rwth-aachen.de Announcements Seminar registration period

More information

Recovering Realistic Texture in Image Super-resolution by Deep Spatial Feature Transform Supplementary Material

Recovering Realistic Texture in Image Super-resolution by Deep Spatial Feature Transform Supplementary Material Recovering Realistic Texture in Image Super-resolution by Deep Spatial Feature Transform Supplementary Material Xintao Wang 1 Ke Yu 1 Chao Dong 2 Chen Change Loy 1 1 CUHK - SenseTime Joint Lab, The Chinese

More information

Text Recognition in Videos using a Recurrent Connectionist Approach

Text Recognition in Videos using a Recurrent Connectionist Approach Author manuscript, published in "ICANN - 22th International Conference on Artificial Neural Networks, Lausanne : Switzerland (2012)" DOI : 10.1007/978-3-642-33266-1_22 Text Recognition in Videos using

More information

Deep Back-Projection Networks For Super-Resolution Supplementary Material

Deep Back-Projection Networks For Super-Resolution Supplementary Material Deep Back-Projection Networks For Super-Resolution Supplementary Material Muhammad Haris 1, Greg Shakhnarovich 2, and Norimichi Ukita 1, 1 Toyota Technological Institute, Japan 2 Toyota Technological Institute

More information

Deep Learning in Visual Recognition. Thanks Da Zhang for the slides

Deep Learning in Visual Recognition. Thanks Da Zhang for the slides Deep Learning in Visual Recognition Thanks Da Zhang for the slides Deep Learning is Everywhere 2 Roadmap Introduction Convolutional Neural Network Application Image Classification Object Detection Object

More information

FAST: A Framework to Accelerate Super-Resolution Processing on Compressed Videos

FAST: A Framework to Accelerate Super-Resolution Processing on Compressed Videos FAST: A Framework to Accelerate Super-Resolution Processing on Compressed Videos Zhengdong Zhang, Vivienne Sze Massachusetts Institute of Technology {zhangzd, sze}@mit.edu Abstract State-of-the-art super-resolution

More information

Deep learning for dense per-pixel prediction. Chunhua Shen The University of Adelaide, Australia

Deep learning for dense per-pixel prediction. Chunhua Shen The University of Adelaide, Australia Deep learning for dense per-pixel prediction Chunhua Shen The University of Adelaide, Australia Image understanding Classification error Convolution Neural Networks 0.3 0.2 0.1 Image Classification [Krizhevsky

More information

Digital Image Restoration

Digital Image Restoration Digital Image Restoration Blur as a chance and not a nuisance Filip Šroubek sroubekf@utia.cas.cz www.utia.cas.cz Institute of Information Theory and Automation Academy of Sciences of the Czech Republic

More information

Machine Learning. Deep Learning. Eric Xing (and Pengtao Xie) , Fall Lecture 8, October 6, Eric CMU,

Machine Learning. Deep Learning. Eric Xing (and Pengtao Xie) , Fall Lecture 8, October 6, Eric CMU, Machine Learning 10-701, Fall 2015 Deep Learning Eric Xing (and Pengtao Xie) Lecture 8, October 6, 2015 Eric Xing @ CMU, 2015 1 A perennial challenge in computer vision: feature engineering SIFT Spin image

More information

Code Mania Artificial Intelligence: a. Module - 1: Introduction to Artificial intelligence and Python:

Code Mania Artificial Intelligence: a. Module - 1: Introduction to Artificial intelligence and Python: Code Mania 2019 Artificial Intelligence: a. Module - 1: Introduction to Artificial intelligence and Python: 1. Introduction to Artificial Intelligence 2. Introduction to python programming and Environment

More information

Bilevel Sparse Coding

Bilevel Sparse Coding Adobe Research 345 Park Ave, San Jose, CA Mar 15, 2013 Outline 1 2 The learning model The learning algorithm 3 4 Sparse Modeling Many types of sensory data, e.g., images and audio, are in high-dimensional

More information

Tutorial on Keras CAP ADVANCED COMPUTER VISION SPRING 2018 KISHAN S ATHREY

Tutorial on Keras CAP ADVANCED COMPUTER VISION SPRING 2018 KISHAN S ATHREY Tutorial on Keras CAP 6412 - ADVANCED COMPUTER VISION SPRING 2018 KISHAN S ATHREY Deep learning packages TensorFlow Google PyTorch Facebook AI research Keras Francois Chollet (now at Google) Chainer Company

More information

arxiv: v1 [cs.cv] 18 Dec 2018 Abstract

arxiv: v1 [cs.cv] 18 Dec 2018 Abstract SREdgeNet: Edge Enhanced Single Image Super Resolution using Dense Edge Detection Network and Feature Merge Network Kwanyoung Kim, Se Young Chun Ulsan National Institute of Science and Technology (UNIST),

More information

Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution

Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution Wei-Sheng Lai 1 Jia-Bin Huang 2 Narendra Ahuja 3 Ming-Hsuan Yang 1 1 University of California, Merced 2 Virginia Tech 3 University

More information

Deep Neural Networks:

Deep Neural Networks: Deep Neural Networks: Part II Convolutional Neural Network (CNN) Yuan-Kai Wang, 2016 Web site of this course: http://pattern-recognition.weebly.com source: CNN for ImageClassification, by S. Lazebnik,

More information

Diffusion Convolutional Recurrent Neural Network: Data-Driven Traffic Forecasting

Diffusion Convolutional Recurrent Neural Network: Data-Driven Traffic Forecasting Diffusion Convolutional Recurrent Neural Network: Data-Driven Traffic Forecasting Yaguang Li Joint work with Rose Yu, Cyrus Shahabi, Yan Liu Page 1 Introduction Traffic congesting is wasteful of time,

More information

Single Image Super Resolution - When Model Adaptation Matters

Single Image Super Resolution - When Model Adaptation Matters JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 1 Single Image Super Resolution - When Model Adaptation Matters arxiv:1703.10889v1 [cs.cv] 31 Mar 2017 Yudong Liang, Radu Timofte, Member, IEEE,

More information

Anchored Neighborhood Regression for Fast Example-Based Super-Resolution

Anchored Neighborhood Regression for Fast Example-Based Super-Resolution Anchored Neighborhood Regression for Fast Example-Based Super-Resolution Radu Timofte 1,2, Vincent De Smet 1, and Luc Van Gool 1,2 1 KU Leuven, ESAT-PSI / iminds, VISICS 2 ETH Zurich, D-ITET, Computer

More information

Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization. Presented by: Karen Lucknavalai and Alexandr Kuznetsov

Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization. Presented by: Karen Lucknavalai and Alexandr Kuznetsov Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization Presented by: Karen Lucknavalai and Alexandr Kuznetsov Example Style Content Result Motivation Transforming content of an image

More information

Prediction of Pedestrian Trajectories Final Report

Prediction of Pedestrian Trajectories Final Report Prediction of Pedestrian Trajectories Final Report Mingchen Li (limc), Yiyang Li (yiyang7), Gendong Zhang (zgdsh29) December 15, 2017 1 Introduction As the industry of automotive vehicles growing rapidly,

More information

RECURRENT NEURAL NETWORKS

RECURRENT NEURAL NETWORKS RECURRENT NEURAL NETWORKS Methods Traditional Deep-Learning based Non-machine Learning Machine-Learning based method Supervised SVM MLP CNN RNN (LSTM) Localizati on GPS, SLAM Self Driving Perception Pedestrian

More information

Restricted Boltzmann Machines. Shallow vs. deep networks. Stacked RBMs. Boltzmann Machine learning: Unsupervised version

Restricted Boltzmann Machines. Shallow vs. deep networks. Stacked RBMs. Boltzmann Machine learning: Unsupervised version Shallow vs. deep networks Restricted Boltzmann Machines Shallow: one hidden layer Features can be learned more-or-less independently Arbitrary function approximator (with enough hidden units) Deep: two

More information

Computer Vision Lecture 16

Computer Vision Lecture 16 Announcements Computer Vision Lecture 16 Deep Learning Applications 11.01.2017 Seminar registration period starts on Friday We will offer a lab course in the summer semester Deep Robot Learning Topic:

More information

Fast and Accurate Single Image Super-Resolution via Information Distillation Network

Fast and Accurate Single Image Super-Resolution via Information Distillation Network Fast and Accurate Single Image Super-Resolution via Information Distillation Network Recently, due to the strength of deep convolutional neural network (CNN), many CNN-based SR methods try to train a deep

More information

Deep Video Super-Resolution Network Using Dynamic Upsampling Filters Without Explicit Motion Compensation

Deep Video Super-Resolution Network Using Dynamic Upsampling Filters Without Explicit Motion Compensation Deep Video Super-Resolution Network Using Dynamic Upsampling Filters Without Explicit Motion Compensation Younghyun Jo Seoung Wug Oh Jaeyeon Kang Seon Joo Kim Yonsei University Abstract Video super-resolution

More information

Learning to Match. Jun Xu, Zhengdong Lu, Tianqi Chen, Hang Li

Learning to Match. Jun Xu, Zhengdong Lu, Tianqi Chen, Hang Li Learning to Match Jun Xu, Zhengdong Lu, Tianqi Chen, Hang Li 1. Introduction The main tasks in many applications can be formalized as matching between heterogeneous objects, including search, recommendation,

More information

Single Image Super-resolution using Deformable Patches

Single Image Super-resolution using Deformable Patches Single Image Super-resolution using Deformable Patches Yu Zhu 1, Yanning Zhang 1, Alan L. Yuille 2 1 School of Computer Science, Northwestern Polytechnical University, China 2 Department of Statistics,

More information

A FRAMEWORK OF EXTRACTING MULTI-SCALE FEATURES USING MULTIPLE CONVOLUTIONAL NEURAL NETWORKS. Kuan-Chuan Peng and Tsuhan Chen

A FRAMEWORK OF EXTRACTING MULTI-SCALE FEATURES USING MULTIPLE CONVOLUTIONAL NEURAL NETWORKS. Kuan-Chuan Peng and Tsuhan Chen A FRAMEWORK OF EXTRACTING MULTI-SCALE FEATURES USING MULTIPLE CONVOLUTIONAL NEURAL NETWORKS Kuan-Chuan Peng and Tsuhan Chen School of Electrical and Computer Engineering, Cornell University, Ithaca, NY

More information

Computer Vision Lecture 16

Computer Vision Lecture 16 Computer Vision Lecture 16 Deep Learning Applications 11.01.2017 Bastian Leibe RWTH Aachen http://www.vision.rwth-aachen.de leibe@vision.rwth-aachen.de Announcements Seminar registration period starts

More information

Feature-Fused SSD: Fast Detection for Small Objects

Feature-Fused SSD: Fast Detection for Small Objects Feature-Fused SSD: Fast Detection for Small Objects Guimei Cao, Xuemei Xie, Wenzhe Yang, Quan Liao, Guangming Shi, Jinjian Wu School of Electronic Engineering, Xidian University, China xmxie@mail.xidian.edu.cn

More information

arxiv: v1 [cs.cv] 30 Nov 2018

arxiv: v1 [cs.cv] 30 Nov 2018 Super-Resolution based on Image-Adapted CNN Denoisers: Incorporating Generalization of Training Data and Internal Learning in Test Time arxiv:1811.12866v1 [cs.cv] 30 Nov 2018 Tom Tirer Tel Aviv University,

More information

Human Detection and Tracking for Video Surveillance: A Cognitive Science Approach

Human Detection and Tracking for Video Surveillance: A Cognitive Science Approach Human Detection and Tracking for Video Surveillance: A Cognitive Science Approach Vandit Gajjar gajjar.vandit.381@ldce.ac.in Ayesha Gurnani gurnani.ayesha.52@ldce.ac.in Yash Khandhediya khandhediya.yash.364@ldce.ac.in

More information

arxiv: v1 [cs.cv] 31 Dec 2018 Abstract

arxiv: v1 [cs.cv] 31 Dec 2018 Abstract Image Super-Resolution via RL-CSC: When Residual Learning Meets olutional Sparse Coding Menglei Zhang, Zhou Liu, Lei Yu School of Electronic and Information, Wuhan University, China {zmlhome, liuzhou,

More information

ROBUST INTERNAL EXEMPLAR-BASED IMAGE ENHANCEMENT. Yang Xian 1 and Yingli Tian 1,2

ROBUST INTERNAL EXEMPLAR-BASED IMAGE ENHANCEMENT. Yang Xian 1 and Yingli Tian 1,2 ROBUST INTERNAL EXEMPLAR-BASED IMAGE ENHANCEMENT Yang Xian 1 and Yingli Tian 1,2 1 The Graduate Center, 2 The City College, The City University of New York, New York, Email: yxian@gc.cuny.edu; ytian@ccny.cuny.edu

More information

EnhanceNet: Single Image Super-Resolution Through Automated Texture Synthesis Supplementary

EnhanceNet: Single Image Super-Resolution Through Automated Texture Synthesis Supplementary EnhanceNet: Single Image Super-Resolution Through Automated Texture Synthesis Supplementary Mehdi S. M. Sajjadi Bernhard Schölkopf Michael Hirsch Max Planck Institute for Intelligent Systems Spemanstr.

More information

An Effective Single-Image Super-Resolution Model Using Squeeze-and-Excitation Networks

An Effective Single-Image Super-Resolution Model Using Squeeze-and-Excitation Networks An Effective Single-Image Super-Resolution Model Using Squeeze-and-Excitation Networks Kangfu Mei 1, Juncheng Li 2, Luyao 1, Mingwen Wang 1, Aiwen Jiang 1 Jiangxi Normal University 1 East China Normal

More information

FAST: A Framework to Accelerate Super- Resolution Processing on Compressed Videos

FAST: A Framework to Accelerate Super- Resolution Processing on Compressed Videos FAST: A Framework to Accelerate Super- Resolution Processing on Compressed Videos Zhengdong Zhang, Vivienne Sze Massachusetts Institute of Technology http://www.mit.edu/~sze/fast.html 1 Super-Resolution

More information

SUPPLEMENTARY MATERIAL

SUPPLEMENTARY MATERIAL SUPPLEMENTARY MATERIAL Zhiyuan Zha 1,3, Xin Liu 2, Ziheng Zhou 2, Xiaohua Huang 2, Jingang Shi 2, Zhenhong Shang 3, Lan Tang 1, Yechao Bai 1, Qiong Wang 1, Xinggan Zhang 1 1 School of Electronic Science

More information

DeepFace: Closing the Gap to Human-Level Performance in Face Verification

DeepFace: Closing the Gap to Human-Level Performance in Face Verification DeepFace: Closing the Gap to Human-Level Performance in Face Verification Report on the paper Artem Komarichev February 7, 2016 Outline New alignment technique New DNN architecture New large dataset with

More information

Object Detection Lecture Introduction to deep learning (CNN) Idar Dyrdal

Object Detection Lecture Introduction to deep learning (CNN) Idar Dyrdal Object Detection Lecture 10.3 - Introduction to deep learning (CNN) Idar Dyrdal Deep Learning Labels Computational models composed of multiple processing layers (non-linear transformations) Used to learn

More information

Human Pose Estimation with Deep Learning. Wei Yang

Human Pose Estimation with Deep Learning. Wei Yang Human Pose Estimation with Deep Learning Wei Yang Applications Understand Activities Family Robots American Heist (2014) - The Bank Robbery Scene 2 What do we need to know to recognize a crime scene? 3

More information