Debbie Marr Sr. Principal Engineer and Director Intel Labs / Accelerator Architecture Lab. September 2018

Transcription

1 Debbie Marr Sr. Principal Engineer and Director Intel Labs / Accelerator Architecture Lab September 2018

2 Talk outline Part 1: The Future of ML (HW) Part 2: Efficient Compute for AI/ML Part 3: FPGAs for ML/AI

3 Customization Standardization ( Makimoto s Wave.. updated to 2018) Standardization Standard Discrete Logic Devices Proliferation of compute Standard ISAs Microprocessors Memories Proliferation of applications Standard IEEE p754 fp (1985), Compliant fp processors Proliferation of reliable FP applications OpenGL (1992), Direct3D (1995) Compliant Graphics Engines Proliferation of GUIs, games, apps? Customization Custom LSIs For TVs, calculators, Specialized algorithms Floating point Co-processors, Specialized FP algorithms Graphics accelerators with custom interfaces & drivers, Specialized graphics algorithms ML/AI accelerators, Specialized, rapidly changing algorithms Makimoto s Wave from Tsugio Makimoto s FPT 2002 talk The Hot Decade of Field Programmable Technologies described FPGAs

4 Forces swinging the Pendulum New Usage Models Software / Algorithm Innovations Compute Supply/Demand Imbalance Differentiation Value Addition Cost Effectiveness Software/Algorithm Stabilization Usage Standardization Operational Efficiency Compute Supply/Demand Balanced Customization Standardization Updated Makimoto s Pendulum from Tsugio Makimoto s FPL 1992 talk The Hot Decade of Field Programmable Technologies

5 Forces swinging the Pendulum New Usage Models Software / Algorithm Innovations Compute Supply/Demand Imbalance Differentiation Value Addition Cost Effectiveness Software/Algorithm Stabilization Usage Standardization Operational Efficiency Compute Supply/Demand Balanced Customization Standardization Updated Makimoto s Pendulum from Tsugio Makimoto s FPL 1992 talk The Hot Decade of Field Programmable Technologies

6 Intel Labs: Accelerator Architecture Lab Team Expertise: Hardware architecture, microarchitecture, compiler Workload Focus: Machine Learning and Deep Learning Goal: Research for x Performance, Efficiency Analyze Algorithms & Workloads Traditional ML SVM, recommendation engines, K-means, KNN, etc. Convolutional Neural Networks (CNN) (Low-precision, alexnet, resnet, inception, IR-v2) Memory / Recursive / Sequence Networks (LSTM, GRU, Neural Turing Machines, Differential Neural Computer) Generator Adversarial Networks (GAN) Reinforcement Learning Transfer Learning One-/Zero-shot Deep Learning Research Programs Intel Collaborative Research Institute Computational Intelligence (ICRI-CI) Hardware Accelerator Research Program for Xeon+FPGA (HARP) Intel Research Strategic Alliances for FPGA HW and SW (ISRA FPGA SW & HW) Custom Accelerators Research (ASIC and FPGA) General-Purpose CPU/GPU Microarchitecture Research

7 AAL Publications Apprentice: Using Knowledge Distillation Techniques to Improve Low-Precision Network Accuracy (ICLR 2018, May 2018) WRPN: Wide Reduced-Precision Networks (ICLR 2018, May 2018) A Customizable Matrix Multiplication Framework for Intel HARPv2 Platform (ISFPGA 2018, February 2018) In-Package Domain-Specific ASICs for Intel Stratix 10 FPGAs: A Case Study of Accelerating Deep Learning Using TensorTile ASIC (ISFPGA 2018, February 2018) Exploration of Low Numeric Precision Deep Learning Inference Using Intel FPGAs (ISFPGA 2018, February 2018) Customizable FPGA OpenCL matrix multiply design template for deep neural networks (FPT 2017, December 2017) High performance binary neural networks on the Xeon+FPGA (FPL 2017, September 2017) Accelerating Deep Convolutional Networks Using Low-Precision and Sparsity (ICASSP 2017, March 2017) Can FPGAs Beat GPUs in Accelerating Next-Generation Deep Neural Networks? (ISFPGA 2017, February 2017) Fine-Grained Accelerators for Sparse Machine Learning Workloads (ASPDA 2017, January 2017) Accelerating Binarized Neural Networks: Comparison of FPGA, CPU, GPU, and ASIC (FPT 2016, December 2016) Accelerating Recurrent Neural Networks in Analytics Servers: Comparison of FPGA, CPU, GPU, and ASIC (FPL 2016, September 2016) Hardware Accelerator for Analytics of Sparse Data (DATE 2016, March 2016) A Sparse Matrix Vector Multiply Accelerator for Support Vector Machine (CASES 2015, October 2015)

8 Customization Standardization ( Makimoto s Wave.. updated to 2018) Standardization Standard Discrete Logic Devices Proliferation of compute Standard ISAs Microprocessors Memories Proliferation of applications Standard IEEE p754 fp (1985), Compliant fp processors Proliferation of reliable FP applications OpenGL (1992), Direct3D (1995) Compliant Graphics Engines Proliferation of GUIs, games, apps? Customization Custom LSIs For TVs, calculators, Specialized algorithms Floating point Co-processors, Specialized FP algorithms Graphics accelerators with custom interfaces & drivers, Specialized graphics algorithms ML/AI accelerators, Specialized, rapidly changing algorithms Makimoto s Wave from Tsugio Makimoto s FPT 2002 talk The Hot Decade of Field Programmable Technologies described FPGAs

9 Talk outline Part 1: The Future of ML (HW) Part 2: Efficient Compute for AI/ML Part 3: FPGAs for ML/AI

10 Hardware Accelerator for Analytics of Sparse Data (DATE 2016) Accelerator for classic machine learning algorithms and real-world sparse datasets (text, ratings)

11 Fine-Grained Accelerators for Sparse Machine Learning (ASPDA 2017, January 2017) Accelerator for CSR (compressed sparse row) classic machine learning algorithms and realworld sparse datasets (text, ratings) A tightly-coupled fine-grained HW accelerator can achieve an order of magnitude perf gain.

12 Ternary Neural Networks AAL has best accuracy! (ICASSP 2017) Larger, deeper networks give better accuracies on Imagenet (image recognition) Ternary network models are 16x smaller for the same configuration Ternary network models can be trained to 80+% sparsity without losing accuracy We have the highest accuracy ternary network! Our paper (Oct 2016) is available at: Network Full-precision 2-bit precision Depth Top-1 [4] Top-5 [4] Top-1 Top-5 Resnet Resnet Resnet Resnet Table1. Accuracy of Resnet network on Imagenet dataset for different depths (first column suffix), regular full-precision, and the extremely low-precision 2-bit version. Fig. 4. Training Resnet-34 with low-precision weights The Fi the 34 tor op gr lay ca er

13 DLAC: Deep Learning Hardware Accelerator (ICASSP 2017)

14 WRPN: Wide Reduced-Precision Networks (ICLR 2018, May 2018) Activation more important than weights in some cases (training, batch inference, HD images, RNNs), but quantization can severely degrade accuracy. Our contribution Simple, hardware-friendly quantization Widening and retraining with Apprentice generates models as accurate as full-precision models, enables low-precision hardware implementations with 2-3x perf & efficiency improvements.

15 Apprentice: Low-Precision Networks (ICLR 2018, May 2018) Joint teacher-student training from scratch gives accuracy - better than SOTA low-precision network A trained teacher distilling knowledge to student reduces time to train by 10-20%. Best accuracy: Full-precision teacher and student networks; knowledge distillation to fine-tune student +WRPN: 10% widening can beat full precision baseline

16 HW/SW Co-optimization for Efficient Compute for AI/ML Key Learnings: 1. Key to improving inference accuracy is finding and exploiting new vectors of regularization/generalization (including sparsity, precision, knowledge distillation) 2. Many promising techniques to reduce compute, memory footprint, data movement for both inference and training 3. HW/SW co-optimization can improve performance and efficiency by 2-3 orders of magnitude (esp. for inference - reduced data movement, compression, tiny die area). Notes: Algorithms, usages evolving at rapid rate. Efficiency techniques may be more or less promising depending on capacity/regularization employed in the algorithm/network. Ensembles of algorithms can be much more powerful!

17 Talk outline Part 1: The Future of ML (HW) Part 2: Efficient Compute for AI/ML Part 3: FPGAs for ML/AI

18 What is FPGA? 1000s of Hard M20K SRAMs (2.5KB/SRAM) Sea of Programmable Logic and Routing Extreme degree of customizations Arbitrary bitwidth, mix bitwidths, etc Arbitrary SRAMs compositions (spad, $, fifo,..) X X X X of hard DSPs (floating-point units) X +1000s M20K M20K M20K + M20K e.g., arbitrary DNN architectures Figures courtesy of Gordon Chiu FPGAs are highly customizable at many levels

19 FPGAs everywhere, from edge to cloud Source: Dasu s TSLRP FPGAs are already in strategic points, where data moves across edge/cloud

20 FPGAs already used for Deep Learning Today E.g., MSFT BrainWave (2017) E.g., Baidu s SDA for DNNs (2015) Alibaba and Amazon also offer FPGAs in their cloud Why? Can FPGA really beat GPUs on DL?

21 What s next? Stronger Trends for Customizations Mix of precisions (just a few examples below) Bfloat FP16 FP11 INT8 INT4 Ternary Binary Mix of uses (e.g., context analyses) images Mix of NN layer types (e.g., DeepSpeech2) FC RNN CNN Multi Tenants User1 NN Mix of sparse layers (e.g., NIPS 15) Ensemble of NNs AutoML: Computer-generated custom NNs (e.g., available in Google Cloud) Body? Face? Scene? User2 NN Posture? Who? Expression? UserX NN Relations/context

22 A Few Highlights from Our FPGA Research Various studies of DL on FPGA vs. GPUs Study 1: Recurrent Neural Networks [FPL 16] Study 2: Binarized Neural Networks [FPT 16][FPL 17][ISFPGA 17][ISFPGA 18] Study 3: Co-optimizing algo and FPGA implementations [FCCM 18][ICLR 18] Study 4: Sparse Ternary Networks [ISFPGA 17][ICASSP 17] Study 5: TensorTile: Combining FPGAs + ASICs for DL [FPL 18]

23 Study 1: Recurrent Neural Networks (RNNs) E. Nurvitadhi, J. Sim, D. Sheffield, A. Mishra, S. Krishnan, D. Marr, Accelerating recurrent neural networks in analytics servers: Comparison of FPGA, CPU, GPU, and ASIC, FPL 16 Neural networks with loops We studied GRU, a state-of-the-art variant of RNN FPGA Design Titan X GPU vs. Arria 10 FPGA Speedup over GPU Hot spot: GEMV on small-medium size weight matrices Dependencies across GEMV operations Comparable or better than GPU For RNNs, can beat GPU in pure performance

24 popcnt Study 2: Binarized Neural Network (BNN) vi x +1 Neural networks s with weights +1 or -1 Matrix x Vector, with +1 or -1 W (-1.-1)+(1.1)+(1.1) = (-1.-1)+(1.-1)+(1.-1) = (-1.1)+(1.1)+(1.-1) vo Compute can be done using xnor and bit counting vi 0 1 x 1 Binarized Matrix x Vector W bcnt(xnor(011,011)) = bcnt(xnor(011,000)) = bcnt(xnor(011,110)) vo FPGA Design in0 lutm xnor split add in1 MUL (Binarized) lut0 Binarized Ops used in BNN is very FPGA friendly Arria 10 deliver comparable Binary Ops Performance to Titan X GPU Stratix 10 projected to be at least ~3x better [1] E. Nurvitadhi, D. Sheffield, J. Sim, A. Mishra, et. al, Accelerating binarized neural networks: Comparison of FPGA, CPU, GPU, and ASIC, FPT 16 [2] D. Moss, E. Nurvitadhi, J. Sim, A. Mishra, et. Al., High Performance Binary Neural Networks on the Xeon+FPGA Platform, FPL 17 (to appear) [3] E. Nurvitadhi, G. Venkatesh, J. Sim, D. Marr, et al., Can FPGAs Beat GPUs in Accelerating Next-Generation Deep Neural Networks? ISFPGA 17 out

25 Study 3: Co-Optimizing Algorithms for FPGAs Unlike GPU, FPGA can support myriad of custom precisions We proposed WRPN approach, to trade-off network width, accuracy, and precisions [ICLR 18] We evaluated many variants of ResNets using WRPN for latest Stratix 10 FPGA [FCCM 18] Stratix 10 FPGA est. performance vs. Accuracy on various ResNets ResNet-34 1x Wide ResNet-34 2x Wide ResNet-34 3x Wide Activation Weight Eq TOPS Top-1 Acc Eq TOPS Top-1 Acc Eq TOPs Top-1 Acc FPGA s fine-grain FP32 FP % NR NR NR NR 8-bit 8-bit % 2 NR 1 NR 8-bit Ternary % 11 NR 5 NR 8-bit Binary x TOP/s 0.3% loss 13 NR 6 NR 4-bit 4-bit % % 2 NR 3-bit 3-bit 51 NR 13 NR 4.2x TOP/s 6@ 1.2% lossnr 2-bit 2-bit % % 9 NR 2-bit Ternary % % 11 NR 1-bit 1-bit % % % programmability allows wide range of SW/HW co-optimization [ICLR 18] A. Mishra, E.Nurvitadhi, J. Cook, D. Marr, WRPN: Wide Reduced-Precision Networks, ICRL 2018 [FCCM 18] P. Colangelo, N. Nasiri, E. Nurvitadhi, et al., Exploration of Low Numeric Precision Deep Learning Inference Using Intel FPGAs, FCCM

26 Study 4: Sparse Ternary Networks Ternary NN: neural net with weights of +1,-1,0 I x 0.2 W = (0.1x-1)+(0x0)+(0.2x1) (0.1x0)+(0x-1)+(0.2x-1) (0.1x1)+(0x0)+(0.2x0) = O ImageNet Accuracy Ternary ResNet offers state-of-the-art accuracy [1] Sparse I and/or ternary W Zero-skip scheduler We target Resnet-50- TNN in this study W I acc_in Skip computation on zero values, and no multiply sign add out [1] G. Venkatesh, E. Nurvitadhi, D. Marrl, Accelerating Deep Convolutional Networks using low-precision and sparsity, ICASSP 17

27 Est. Stratix 10 Results [1] Speedup vs. dense design Per layer speedup varies depends on sparsity 1.994x weighted average speedups LayerID TOP/s GOP/s/Watt S10 FPGA performs better, across all frequency targets Titan X GPU theoretical peak Titan X GPU measured on Torch, batch64 Titan X GPU theoretical peak Titan X GPU measured on Torch, batch64 [1] E. Nurvitadhi, G. Venkatesh, J. Sim, D. Marr, et al., Can FPGAs Beat GPUs in Accelerating Next- Generation Deep Neural Networks? ISFPGA 17

28 Package Study 5: Combine FPGAs + ASIC tiles ( chiplets ) [FPL 18] FPGA: flexibility for custom application-specific ops Stratix10 FPGAs already System-in- Package : use 2.5D EMIBs to offer xcvr and memory tiles next to FPGA ASIC ASIC EMIB EMIB FPGA EMIB EMIB ASIC ASIC Intel s 2.5D EMIB for integration ASICs: efficiency for domain-shared ops We propose TensorTile: ASIC tile chiplet for Deep Learning [FPL 18] E. Nurvitadhi, J. Cook, A. Mishra, D. Marr, et. al., In-Package Domain-Specific ASICs for Intel Stratix 10 FPGAs: A Case Study of Accelerating Deep Learning Using TensorTile ASIC, FPL 2018.

29 Very Scalable: Can Mix & Match FPGAs + Tiles Volta: GPU + Tensor Cores 125 TOPs (FP16) 26 MB on-chip RAMs 1.5x and 2.7x better peak TOPs and RAMs Est. 69 TOPs (INT8) in a T-tile Ttile Stratix (378K LEs) S x T-tiles Est. 194 TOPs (FP16) 72MB on-chip RAMs Small Large Small to large FPGAs in Stratix 10 family, with links to 1 or more TensorTiles

30 Case Studies (1) AlexNet (FP16) Inference, low batch (2) Persistent ResNets (INT8) 2x Tensor Tiles S (MX) Intel DLA [ISFPGA 17] DNN footprint (MBs) 6xT-tiles: ~72MB onchip RAMs Volta GPU: ~26MB onchip RAMs BatchSz DNN The 2x tiles improved performance and performance/watt by 4x and 3.3x TensorTiles improve persistence of DNNs

31 Key FPGA Learnings FPGAs are already placed where data flows through compute systems FPGAs shine on low-latency/data-intensive/irregular DL FPGAs + ASICs (e.g. TensorTiles) enable powerful, versatile options Thank you! (And, we re hiring!)

32 Customization Standardization ( Makimoto s Wave.. updated to 2018) Standardization Standard Discrete Logic Devices Proliferation of compute Standard ISAs Microprocessors Memories Proliferation of applications Standard IEEE p754 fp (1985), Compliant fp processors Proliferation of reliable FP applications OpenGL (1992), Direct3D (1995) Compliant Graphics Engines Proliferation of GUIs, games, apps? Customization Custom LSIs For TVs, calculators, Specialized algorithms Floating point Co-processors, Specialized FP algorithms Graphics accelerators with custom interfaces & drivers, Specialized graphics algorithms ML/AI accelerators, Specialized, rapidly changing algorithms Makimoto s Wave from Tsugio Makimoto s FPT 2002 talk The Hot Decade of Field Programmable Technologies described FPGAs

33 Legal Disclaimer INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL'S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. A "Mission Critical Application" is any application in which failure of the Intel Product could result, directly or indirectly, in personal injury or death. SHOULD YOU PURCHASE OR USE INTEL'S PRODUCTS FOR ANY SUCH MISSION CRITICAL APPLICATION, YOU SHALL INDEMNIFY AND HOLD INTEL AND ITS SUBSIDIARIES, SUBCONTRACTORS AND AFFILIATES, AND THE DIRECTORS, OFFICERS, AND EMPLOYEES OF EACH, HARMLESS AGAINST ALL CLAIMS COSTS, DAMAGES, AND EXPENSES AND REASONABLE ATTORNEYS' FEES ARISING OUT OF, DIRECTLY OR INDIRECTLY, ANY CLAIM OF PRODUCT LIABILITY, PERSONAL INJURY, OR DEATH ARISING IN ANY WAY OUT OF SUCH MISSION CRITICAL APPLICATION, WHETHER OR NOT INTEL OR ITS SUBCONTRACTOR WAS NEGLIGENT IN THE DESIGN, MANUFACTURE, OR WARNING OF THE INTEL PRODUCT OR ANY OF ITS PARTS. Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or characteristics of any features or instructions marked reserved or undefined. Intel reserves these for future definition and shall have no responsibility whatsoever for conflicts or incompatibilities arising from future changes to them. The products described in this document may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. The code names presented in this document are only for use by Intel to identify products, technologies, or services in development, that have not been made commercially available to the public, i.e., announced, launched or shipped. They are not "commercial" names for products or services and are not intended to function as trademarks. Results have been estimated or simulated using internal Intel analysis or architecture simulation or modeling, and provided to you for informational purposes. Any differences in your system hardware, software or configuration may affect your actual performance. Contact your local Intel sales office or your distributor to obtain the latest specifications and before placing your product order. Copies of documents which have an order number and are referenced in this document, or other Intel literature may be obtained by calling or by visiting Intel's website at Intel is a trademark of Intel Corporation in the US and other countries. Copyright 2015 Intel Corporation. All rights reserved. * Other brands and names may be claimed as the property of others.

34