Deep Image: Scaling up Image Recognition. Ren Wu Distinguished Scientist, 韧在百度

Transcription

1 Deep Image: Scaling up Image Recognition Ren Wu Distinguished Scientist, 韧在百度

2 The Color of the Dress

3 The Color of the Dress Color Constancy Human vs. Artificial Intelligence

4

5 GTC14 GTC 14: Deep Learning Meets Heterogeneous Computing Big data + Deep learning + High performance computing = Intelligence Big data + Deep learning + Heterogeneous computing = Success

6 Baidu Stock

7 Baidu Q2 14

8 Big Data Storage Processing Webpages Index Update Log >2000PB PB/day 100b-1000b 100b-1000b 1b-10b/day 100TB~1PB/day

9 Computer Chess and Moore s Law

10 Computer Chess and Moore s Law

11 Deep Blue A classic example of application-specific system design comprised of an IBM supercomputer with 480 custom-made VLSI chess chips, running massively parallel search algorithm with highly optimized implementation.

12 Heterogeneous Computing 1993 world #1 Think Machine CM5/ GFlops 2013 Samsung Note 3 smartphone (Qualcomm SnapDragon 800) 129 Gflops 2000 world #1 ASCI White (IBM RS/6000SP) 6MW power, 106 tons 12.3 TFlops 2013 Two MacPro workstation (dual AMD GPUs each) 14 TFlops

13 Deep Learning Applications Speech recognition Image recognition Optical character recognition (OCR) Language translation Web search Computational Ads (CTR)

14 ImageNet Large-Scale Visual Recognition Challenge ImageNet dataset More than 15 million images belonging to about 22,000 categories ILSVRC (ImageNet Large-Scale Visual Recognition Challenge) Classification task: 1.2 million images contains 1,000 categories One of the most challenging computer vision benchmarks Increasing attention both from industry and academic communities * Olga Russakovsky et al. ECCV 2014

15 ImageNet Classification Challenge

16 ImageNet classification Team Year Place Error (top-5) Uses external data SuperVision % no SuperVision st 15.3% ImageNet 22k Clarifai % no Clarifai st 11.2% ImageNet 22k MSRA rd 7.35% no VGG nd 7.32% no GoogLeNet st 6.67% no Slide credit: Yangqing Jia, Google Invincible?

17 Our approach Insights and inspirations 多算胜少算不胜 孙 子 ( BC) 计篇 More calculations win, few calculation lose 元元本本殚 见洽闻 班固 (32-92 AD) 西都赋 Meaning the more you see the more you know 明 足以察秋毫之末 孟 子 ( BC) 梁惠 王上 ability to see very fine details

18 Project Minwa 百度敏娲 Minerva + Athena + 女娲 Athena: Goddess of Wisdom, Warfare, Divine Intelligence, Architecture, and Crafts Minerva: Goddess of wisdom, magic, medicine, arts, commerce and defense 女娲 : 抟 土造 人, 炼 石补天, 婚姻, 乐器 World s Largest Artificial Neural Networks v Pushing the State-of-the-Art v ~ 100x bigger than previous ones v New kind of Intelligence?

19 Hardware/Software Co-design Stochastic gradient decent (SGD) High compute density Scale up, up to 100 nodes High bandwidth low latency GPUs Infiniband 36 nodes, 144 GPUs, 6.9TB Host, 1.7TB Device 0.6 PFLOPS Highly Optimized software stack RDMA/GPU Direct New data partition and communication strategies

20 Minwa

21 Speedup ( wall time for convergence ) GPU 16 GPU 1 GPU Accuracy Accuracy 80% 32 GPU: 8.6 hours 1 GPU: 212 hours Speedup: 24.7x Time (hours) Validation set accuracy for different numbers of GPUs

22 Data Augmentation Never have enough training examples! Key observations Invariant to illuminant of the scene Invariant to observers Augmentation approaches Color casting Optical distortion Rotation and cropping etc 见多识 广

23 The color of the Dress And the Color Constancy Key observations Invariant to illuminant of the scene Invariant to observers Augmentation approaches Color casting Optical distortion Rotation and cropping etc Inspired by the color constancy principal. Essentially, this forces our neural network to develop its own color constancy ability.

24 Data Augmentation Possible variations Augmentation The number of possible changes Color casting Vignetting 1960 Lens distortion 260 Rotation 20 Flipping 2 Cropping 82944(crop size is 224x224, input image size is 512x512) The Deep Image system learned from ~2 billion examples, out of 90 billion possible candidates.

25 Data augmentation vs. Overfitting

26 Examples Bathtub Isopod Indian elephant Ice bear Some hard cases addressed by adding our data augmentation.

27 Multi-scale training Same crop size, different resolution Fixed-size 224*224 Downsized training images Reduces computational costs But not for state-of-the-art Different models trained by different image sizes High-resolution model works 256x256: top % 512x512: top % Multi-scale models are complementary Fused model: 6.97% 明查秋毫 256* *512

28 Multi-scale training Backpack Tricycle Washer Little blue heron

29 Tricycle

30 Model One basic configuration has 16 layers The number of weights in our configuration is 212.7M About 40% bigger than VGG s Team Top-1 val. error Top-5 val. error GoogLeNet % VGG 25.9% 8.0% Deep Image 24.88% 7.42%

31 Compare to state-of-the-art Team Year Place Top-5 test error SuperVision % ISI % VGG % Clarifai % NUS % ZF % GoogLeNet % VGG % MSRA % Andrew Howard % DeeperVision % Deep Image % Deep Image has set the new record of 5.98% top-5 error rate for test dataset, a 10.2% relative improvement than the previous best result.

32 Latest results Team Date Top-5 test error GoogLeNet % Deep Image 01/12/ % Deep Image 02/05/ % Microsoft 02/05/ % Google 03/02/ % Deep Image 03/17/ %

33 Robustness

34

35

36

37

38

39

40

41

42 Major differentiators Customized built supercomputer dedicated for DL Simple, scalable algorithm + Fully optimized software stack Larger models More Aggressive data augmentation Multi-scale, include high-resolution images Brute force + Insights and push for extreme

43 Thank you!