Comprehensive Arm Solutions for Innovative Machine Learning (ML) and Computer Vision (CV) Applications

Transcription

1 Comprehensive Arm Solutions for Innovative Machine Learning (ML) and Computer Vision (CV) Applications Helena Zheng ML Group, Arm Arm Technical Symposia 2017, Taipei

2 Machine Learning is a Subset of Artificial Intelligence AI means many things to many people Artificial Intelligence Machine Learning Perception & Vision Natural Language Processing Knowledge Representation Planning & Navigation Generalized Intelligence ML itself has a lot of depth 3 3

3 Why Artificial Intelligence(AI) Exploding Now Availability of increased data sourced at the edge with ubiquitous powerful compute! Compute 2010 Data zettabyte IP Traffic zettabyte 4 4

4 AI Presents Significant Opportunity for Innovation VR/MR Robotics Drones Shipping & logistics IoT Home, surveillance & analytics Automotive Mobile 5 5

5 Distributing Intelligence Cloud-based training High-performance processing On-device learning Security and privacy for your data AI in your hand & cloud Real-time inference for autonomous systems 6 6

6 Why is On-device ML Driving AI to the Edge? Bandwidth Power Cost Latency Privacy 7 7

7 Arm ML Platform Enables Efficiency Flexibility Freedom 8 8

8 Arm s ML Computing Platform AI Applications Incorporating ML, CV, speech recognition etc. Applications Compute library Neural network frameworks (e.g. TensorFlow, Caffe, AndroidNN) Optional Spirit libraries & model sets Spirit metadata library Stable SW interfaces Arm DS-5 / Keil tools / compilers / drivers CPU CPU GPU Spirit Computer Vision Edge devices SVE 9 9

9 Components of Arm ML Platform Software Specialized Acceleration Hardware 10 10

10 Software Development 11 11

11 Arm Compute Library Faster, advanced processing What is the Arm Compute Library? Functions for CV and deeplearning algorithms Optimized for Arm CPU and GPU OS and platform agnostic No fee, MIT license Delivers faster processing 4.6x faster than stock OpenCV on NEON Offers OpenCV and Open VX compatibility Use as a plug-in backend for your own runtime implementation Available now:

12 Arm Compute Library Partners 13 13

13 Specialized Acceleration 14 14

14 Computer Vision (CV)

15 Spirit for Object Detection and Localization Sensor ISP Spirit CV pre-processor Image stream CPU GPU Sensor interface Feature extraction Classifier 1 Classifier 2 Metadata stream (Regions of interest) Acceleration Beth Ben 16 16

16 Comparison with Neural Network Framework Solutions Yolo Spirit SSD Neural Network 17 17

17 Spirit: Key Features Object localization (>200k locations in full HD) Size variation (~20x for full HD) Scalable to 4K without performance compromise Real time, 60 fps, no dropped frames Invariance to optical distortions Invariance to illumination conditions High occlusion tolerance Suitable for stationary and moving cameras 18 18

18 Comparison with a DSP Spirit uses a form of HOG*/ SVM* baked into an efficient hardware design Using a DSP to achieve the same performance E.g. Pedestrian Detection on a DSP Processed at VGA resolution Achieving 5fps Operating at 40MHz/50MHz Scaling this to Spirit performance levels of 1080p60 would require the DSP to run at 3.24GHz *Histogram of Oriented Gradients *Support Vector Machines

19 ML on Mali GPUs

20 Relative Eneryg Efficency Mali GPUs: Increasing ML Throughput and Efficiency Increasing efficiency SGEMM FP32 Mali-G71 SGEMM FP16 Mali-G72 17% Efficiency gain GEMM depicts core functionality of ML algorithms Mali-G72 has several optimizations to improve ML inference Less power-hungry FMA unit Bigger L1 cache in the execution engine Mali-G72 is the most efficient Mali GPU for machine learning 21 21

21 Hardware 22 22

22 AI Applications at the Edge on Arm Detect plant diseases Sort cucumbers Detect Caltrain delays 23 23

23 Instruction Sets for AI Cortex-A Additional dot product instructions (Cortex-A55 and Cortex-A75) New Scalable Vector Extensions (SVE) instructions Cortex-M Optimized CMSIS-DSP libraries for matrix multiplication Closely-coupled acceleration Improved performance and efficiency (for broader use cases) Flexibility in multi-core computing with Arm DynamIQ technology 25 25

24 DynamIQ: New Cluster Design for New Cores Arm DynamIQ big.little systems: 26 Greater product differentiation and scalability Improved energy efficiency and performance SW compatibility with Energy Aware Scheduling (EAS) Private L2 and shared L3 caches Local cache close to processors L3 cache shared between all cores DynamIQ Shared Unit (DSU) Contains L3, Snoop Control Unit (SCU) and all cluster interfaces Additional instructions for ML 26 SCU ACP Cortex-A75 32b/64b Core Private L2 cache 1b+7L AMBA4 ACE.. Peripheral Port DynamIQ Shared Unit (DSU) 2b+6L 1b+2L 1b+3L Example: DynamIQ big.little configurations Cortex-A55 32b/64b Core Private L2 cache Async Bridges Shared L3 cache 4b+4L 1b+4L

25 New DynamIQ-based CPUs for New Possibilities Cortex-A75 processor >50% more performance compared to current devices Cortex-A55 processor 2.5x higher power efficiency compared to current devices 27 Estimated device performance using SPECINT2006, final device results may vary Comparison using Cortex-A73 at 2.4GHz vs Cortex-A75 at 3GHz 27 Comparison using Cortex-A53 in 28nm devices vs Cortex-A55 in 16nm devices

26 Time (lower is better) MAC/cycle Enhanced Architecture for Emerging Use Cases Computer Vision Machine Learning FP16 FP16 0 Harris Corners (relative to Cortex-A53) Cortex-A53 (FP32) Cortex-A55 (FP32) Cortex-A55(FP16) Cortex-A73 (FP32) Cortex-A75(FP32) Cortex-A75(FP16) General Matrix Multiply (relative to Cortex-A53) Cortex-A53 (FP32) Cortex-A55 (FP16) 8-bit dot product Cortex-A55 (FP32) Cortex-A55 (8-bit) 28 28

27 Introducing the Scalable Vector Extension (SVE) + pred = for (i = 0; i < n; ++i) INDEX i CMPLT n n-2 n-1 n n Gather-load and scatter-store Per-lane predication Predicate-driven loop control and management = + pred = = = Vector partitioning and software-managed speculation Extended floating-point horizontal reductions 29 29

28 Instruction Sets for AI Cortex-A Additional dot product instructions (Cortex-A55 and Cortex-A75) New SVE instructions Cortex-M Optimized CMSIS-DSP libraries for matrix multiplication Closely-coupled acceleration Improved performance and efficiency (for broader use cases) Flexibility in multi-core computing with DynamIQ technology 30 30

29 Relative throughput Software Optimizations: Cortex-M Example Convolution Use of partial im2col to reduce the memory footprint Optimized data dimension layout (Height-Width-Channel) to save im2col overhead Pooling Split into x-pooling and y-pooling instead of window-based 5.1X improvements compared to Caffe-like implementation Activation ReLU: use SIMD within a register, 2.6X improvement compared to Caffe-like implementation Sigmoid and Tanh: use table look-up CIFAR-10 network runtime improvement Conv Pooling ReLU Total Baseline New kernels *Baseline uses CMSIS 1D Conv and Caffe-like Pooling/ReLU The new kernels will be integrated into future versions of CMSIS 4x higher perf 31 31

30 Instruction sets for AI Cortex-A Additional dot product instructions (Cortex-A55 and Cortex-A75) New SVE instructions Cortex-M Optimized CMSIS-DSP libraries for matrix multiplication Closely-coupled acceleration Improved performance and efficiency (for broader use cases) Flexibility in multi-core computing with DynamIQ technology 32 32

31 Interface to Acceleration Logic (1) Configure registers for task (3) Processing completed DynamIQ cluster PP ACP (2) Fetches data from CPU memory (4) Writes result into CPU memory Accelerator (3) Carries out acceleration DynamIQ cluster PP ACP (2) Carries out computation (1) Writes data into CPU memory (4) Reads result from CPU memory or sends data for onward processing I/O agent 33 33

32 Accelerator Flexible Acceleration Platform Closely coupled Independent Off-chip coherent DynamIQ Cluster Level-3 Cache Accelerator Accelerator Accel Cache CCIX XP CMN-600 XP XP Agile System Cache Agile System Cache XP DMC-620 XP DMC-620 XP CCIX 34 34

33 Arm ML Platform Enables Efficiency Flexibility Freedom 35 35

34 Thank You! Danke! Merci! 謝謝! ありがとう! Gracias! Kiitos! 38

35 The Arm trademarks featured in this presentation are registered trademarks or trademarks of Arm Limited (or its subsidiaries) in the US and/or elsewhere. All rights reserved. All other marks featured may be trademarks of their respective owners. 39