TVM: An Automated End-to-End Optimizing Compiler for Deep Learning
|
|
|
- Vernon Jenkins
- 5 years ago
- Views:
Transcription
1 TVM: An Automated End-to-End Optimizing Compiler for Deep Learning Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Meghan Cowan, Haichen Shen, Leyuan Wang, Yuwei Hu, Luis Ceze, Carlos Guestrin, Arvind Krishnamurthy!1
2 Collaborators SAMPL Lab:
3 Beginning of Story!3
4 Beginning of Story Acclerator!3
5 Beginning of Story Acclerator!3
6 Beginning of Story Acclerator // Pseudo-code for convolution program for the VIA accelerator // Virtual Thread 0 0x00: LOAD(PARAM[ 0-71]) // LD@TID0 0x01: LOAD(ACTIV[ 0-24]) // LD@TID0 0x02: LOAD(LDBUF[ 0-31]) // LD@TID0 0x03: PUSH(LD->EX) // LD@TID0 0x04: POP (LD->EX) // EX@TID0 0x05: EXE (ACTIV[ 0-24],PARAM[ 0-71],LDBUF[ 0-31],STBUF[ 0-7]) // EX@TID0 0x06: PUSH(EX->LD) // EX@TID0 0x07: PUSH(EX->ST) // EX@TID0 0x08: POP (EX->ST) // ST@TID0 0x09: STOR(STBUF[ 0-7]) // ST@TID0 0x0A: PUSH(ST->EX) // ST@TID0 // Virtual Thread 1 0x0B: LOAD(ACTIV[25-50]) // LD@TID1 0x0C: LOAD(LDBUF[32-63]) // LD@TID1 0x0D: PUSH(LD->EX) // LD@TID1 0x0E: POP (LD->EX) // EX@TID1 0x0F: EXE (ACTIV[25-50],PARAM[ 0-71],LDBUF[32-63],STBUF[32-39]) // EX@TID1 0x10: PUSH(EX->LD) // EX@TID1 0x11: PUSH(EX->ST) // EX@TID1 0x12: POP (EX->ST) // ST@TID1 0x13: STOR(STBUF[32-39]) // ST@TID1 0x14: PUSH(ST->EX) // ST@TID1 // Virtual Thread 2 0x15: POP (EX->LD) // LD@TID2 0x16: LOAD(PARAM[ 0-71]) // LD@TID2 0x17: LOAD(ACTIV[ 0-24]) // LD@TID2 0x18: LOAD(LDBUF[ 0-31]) // LD@TID2 0x19: PUSH(LD->EX) // LD@TID2 0x1A: POP (LD->EX) // EX@TID2 0x1B: POP (ST->EX) // EX@TID2 0x1C: EXE (ACTIV[ 0-24],PARAM[ 0-71],LDBUF[ 0-31],STBUF[ 0-7]) // EX@TID2 0x1D: PUSH(EX->ST) // EX@TID2 0x1E: POP (EX->ST) // ST@TID2 0x1F: STOR(STBUF[ 0-7]) // ST@TID2 // Virtual Thread 3 0x20: POP (EX->LD) // LD@TID3 0x21: LOAD(ACTIV[25-50]) // LD@TID3 0x22: LOAD(LDBUF[32-63]) // LD@TID3 0x23: PUSH(LD->EX) // LD@TID3 0x24: POP (LD->EX) // EX@TID3 0x25: POP (ST->EX) // EX@TID2 0x26: EXE (ACTIV[25-50],PARAM[ 0-71],LDBUF[32-63],STBUF[32-39]) // EX@TID3 0x27: PUSH(EX->ST) // EX@TID3 0x28: POP (EX->ST) // ST@TID3 0x29: STOR(STBUF[32-39]) // ST@TID3 (a) Blocked convolution program with multiple thread contexts // Convolution access pattern dictated by micro-coded program. // Each register index is derived as a 2-D affine function. // e.g. idx rf = a rf y+b rf x+c rf 0, where c rf 0 is specified by // micro op 0 fields. for y in [0 i) for x in [0 j) rf[idx rf 0 ] += GEVM(act[idx act 0 ], par[idx par 0 ]) rf[idx rf 1 ] += GEVM(act[idx act 1 ], par[idx par 1 ]) rf[idx rf n ] += GEVM(act[idx act n ], par[idx par n ]) (b) Convolution micro-coded program // Max-pool, batch normalization and activation function // access pattern dictated by micro-coded program. // Each register index is derived as a 2D affine function. // e.g. idx dst = a dst y+b dst x+c dst 0, where c dst 0 is specified by // micro op 0 fields. for y in [0 i) for x in [0 j) // max pooling rf[idx dst 0 ] = MAX(rf[idx dst 0 ], rf[idx src 0 ]) rf[idx dst 1 ] = MAX(rf[idx dst 1 ], rf[idx src 1 ]) // batch norm rf[idx dst m ] = MUL(rf[idx dst m ], rf[idx src m ]) rf[idx dst m+1 ] = ADD(rf[idx dst m+1 ], rf[idx src m+1 ]) rf[idx dst m+2 ] = MUL(rf[idx dst m+2 ], rf[idx src m+2 ]) rf[idx dst m+3 ] = ADD(rf[idx dst m+3 ], rf[idx src m+3 ]) // activation rf[idx dst n-1 ] = RELU(rf[idx dst n-1 ], rf[idx src n-1 ]) rf[idx dst n ] = RELU(rf[idx dst n ], rf[idx src n ]) (c) Max pool, batch norm and activation micro-coded program!3
7 Goal: Deploy Deep Learning Everywhere Frameworks!4
8 Goal: Deploy Deep Learning Everywhere Frameworks Explosion of models and frameworks!4
9 Goal: Deploy Deep Learning Everywhere Frameworks Explosion of models and frameworks Explosion of hardware backends!4
10 Goal: Deploy Deep Learning Everywhere Frameworks Explosion of models and frameworks Explosion of hardware backends!4
11 Goal: Deploy Deep Learning Everywhere Frameworks Explosion of models and frameworks Explosion of hardware backends!4
12 Goal: Deploy Deep Learning Everywhere Frameworks Explosion of models and frameworks Explosion of hardware backends!4
13 Goal: Deploy Deep Learning Everywhere Frameworks Explosion of models and frameworks Explosion of hardware backends!4
14 Goal: Deploy Deep Learning Everywhere Frameworks Explosion of models and frameworks Explosion of hardware backends!4
15 Goal: Deploy Deep Learning Everywhere Frameworks Explosion of models and frameworks Explosion of hardware backends!4
16 Goal: Deploy Deep Learning Everywhere Frameworks Explosion of models and frameworks Explosion of hardware backends!4
17 Goal: Deploy Deep Learning Everywhere Frameworks Explosion of models and frameworks Explosion of hardware backends Acclerator!4
18 Goal: Deploy Deep Learning Everywhere Frameworks Explosion of models and frameworks Huge gap between model/frameworks and hardware backends Explosion of hardware backends Acclerator!4
19 Existing Approach Frameworks Hardware!5
20 Existing Approach Frameworks High-level data flow graph Hardware!5
21 Existing Approach Frameworks High-level data flow graph Primitive Tensor operators such as Conv2D Hardware!5
22 Existing Approach Frameworks High-level data flow graph Primitive Tensor operators such as Conv2D eg. cudnn Offload to heavily optimized DNN operator library Hardware!5
23 Existing Approach: Engineer Optimized Tensor Operators!6
24 Existing Approach: Engineer Optimized Tensor Operators!6
25 Existing Approach: Engineer Optimized Tensor Operators Matmul: Operator Specification C = tvm.compute((m, n), lambda y, x: tvm.sum(a[k, y] * B[k, x], axis=k))!6
26 Existing Approach: Engineer Optimized Tensor Operators Matmul: Operator Specification C = tvm.compute((m, n), lambda y, x: tvm.sum(a[k, y] * B[k, x], axis=k))!6
27 Existing Approach: Engineer Optimized Tensor Operators Matmul: Operator Specification C = tvm.compute((m, n), lambda y, x: tvm.sum(a[k, y] * B[k, x], axis=k)) Vanilla Code for y in range(1024): for x in range(1024): C[y][x] = 0 for k in range(1024): C[y][x] += A[k][y] * B[k][x]!6
28 Existing Approach: Engineer Optimized Tensor Operators Matmul: Operator Specification C = tvm.compute((m, n), lambda y, x: tvm.sum(a[k, y] * B[k, x], axis=k)) Loop Tiling for Locality for yo in range(128): for xo in range(128): C[yo*8:yo*8+8][xo*8:xo*8+8] = 0 for ko in range(128): for yi in range(8): for xi in range(8): for ki in range(8): C[yo*8+yi][xo*8+xi] += A[ko*8+ki][yo*8+yi] * B[ko*8+ki][xo*8+xi]!7
29 Existing Approach: Engineer Optimized Tensor Operators Matmul: Operator Specification C = tvm.compute((m, n), lambda y, x: tvm.sum(a[k, y] * B[k, x], axis=k)) Map to Accelerators inp_buffer AL[8][8], BL[8][8] acc_buffer CL[8][8] for yo in range(128): for xo in range(128): vdla.fill_zero(cl) for ko in range(128): vdla.dma_copy2d(al, A[ko*8:ko*8+8][yo*8:yo*8+8]) vdla.dma_copy2d(bl, B[ko*8:ko*8+8][xo*8:xo*8+8]) vdla.fused_gemm8x8_add(cl, AL, BL) vdla.dma_copy2d(c[yo*8:yo*8+8,xo*8:xo*8+8], CL) Human exploration of optimized code!8
30 Limitations of Existing Approach Frameworks cudnn!9
31 Limitations of Existing Approach Frameworks cudnn!9
32 Limitations of Existing Approach Frameworks cudnn!9
33 Limitations of Existing Approach Frameworks cudnn!9
34 Limitations of Existing Approach Frameworks cudnn!9
35 Limitations of Existing Approach Frameworks New operator introduced by operator fusion optimization potentially benefit: 1.5x speedup cudnn!9
36 Limitations of Existing Approach Frameworks New operator introduced by operator fusion optimization potentially benefit: 1.5x speedup cudnn!9
37 Limitations of Existing Approach Frameworks New operator introduced by operator fusion optimization potentially benefit: 1.5x speedup cudnn!9
38 Limitations of Existing Approach Frameworks New operator introduced by operator fusion optimization potentially benefit: 1.5x speedup cudnn Engineering intensive!9
39 Learning-based Learning System Frameworks High-level data flow graph and optimizations Hardware!10
40 Learning-based Learning System Frameworks High-level data flow graph and optimizations Hardware!10
41 Learning-based Learning System Frameworks High-level data flow graph and optimizations Hardware aware Search Space of Optimized Tensor Programs Hardware!10
42 Learning-based Learning System Frameworks High-level data flow graph and optimizations Hardware aware Search Space of Optimized Tensor Programs Machine Learning based Program Optimizer Hardware!10
43 Learning-based Learning System Frameworks High-level data flow graph and optimizations Hardware aware Search Space of Optimized Tensor Programs Machine Learning based Program Optimizer directly generate optimized program for new operator workloads and hardware Hardware!10
44 Learning-based Learning System Frameworks High-level data flow graph and optimizations Hardware aware Search Space of Optimized Tensor Programs Machine Learning based Program Optimizer Hardware!11
45 Learning-based Learning System Frameworks High-level data flow graph and optimizations Hardware aware Search Space of Optimized Tensor Programs Machine Learning based Program Optimizer Hardware!11
46 Hardware-aware Search Space Tensor Expression Language (Specification) C = tvm.compute((m, n), lambda y, x: tvm.sum(a[k, y] * B[k, x], axis=k)) Hardware!12
47 Hardware-aware Search Space Tensor Expression Language (Specification) C = tvm.compute((m, n), lambda y, x: tvm.sum(a[k, y] * B[k, x], axis=k)) Define search space of hardware aware mappings from expression to hardware program Based on Halide s compute/schedule separation Hardware!12
48 Hardware-aware Search Space CPUs Compute Primitives Memory Subsystem L3 scalar vector L2 L2 L1D L1I L1D L1I implicitly managed Loop Transformations Cache Locality Vectorization Reuse primitives from prior work: Halide, Loopy!13
49 Challenge to Support Diverse Hardware Backends CPUs GPUs TPU-like specialized Accelerators!14
50 Hardware-aware Search Space GPUs Compute Primitives Memory Subsystem scalar vector L2 SM SM TX/L1 TX/L1 RF RF RF RF mixed!15
51 Hardware-aware Search Space GPUs Compute Primitives Memory Subsystem scalar vector L2 SM SM TX/L1 TX/L1 RF RF RF RF mixed Shared memory among compute cores!15
52 Hardware-aware Search Space GPUs Compute Primitives Memory Subsystem scalar vector L2 SM SM TX/L1 TX/L1 RF RF RF RF mixed Shared memory among compute cores Use of Shared Thread Memory Cooperation!15
53 Hardware-aware Search Space TPU-like Specialized Accelerators Compute Primitives tensor Memory Subsystem Unified Buffer FIFO Acc explicitly managed!16
54 Hardware-aware Search Space TPU-like Specialized Accelerators Compute Primitives tensor Memory Subsystem Unified Buffer FIFO Acc explicitly managed!16
55 Tensorization Challenge Compute primitives!17
56 Tensorization Challenge Compute primitives scalar!17
57 Tensorization Challenge Compute primitives scalar vector!17
58 Tensorization Challenge Compute primitives scalar vector tensor!17
59 Tensorization Challenge Compute primitives scalar vector tensor Hardware designer: declare tensor instruction interface with Tensor Expression w, x = t.placeholder((8, 8)), t.placeholder((8, 8)) k = t.reduce_axis((0, 8)) y = t.compute((8, 8), lambda i, j: t.sum(w[i, k] * x[j, k], axis=k)) def gemm_intrin_lower(inputs, outputs): ww_ptr = inputs[0].access_ptr( r") xx_ptr = inputs[1].access_ptr("r") zz_ptr = outputs[0].access_ptr("w") compute = t.hardware_intrin("gemm8x8", ww_ptr, xx_ptr, zz_ptr) reset = t.hardware_intrin("fill_zero", zz_ptr) update = t.hardware_intrin("fuse_gemm8x8_add", ww_ptr, xx_ptr, zz_ptr) return compute, reset, update gemm8x8 = t.decl_tensor_intrin(y.op, gemm_intrin_lower) declare behavior lowering rule to generate hardware intrinsics to carry out the computation!17
60 Tensorization Challenge Compute primitives scalar vector tensor Hardware designer: declare tensor instruction interface with Tensor Expression Tensorize: transform program to use tensor instructions w, x = t.placeholder((8, 8)), t.placeholder((8, 8)) k = t.reduce_axis((0, 8)) y = t.compute((8, 8), lambda i, j: t.sum(w[i, k] * x[j, k], axis=k)) def gemm_intrin_lower(inputs, outputs): ww_ptr = inputs[0].access_ptr( r") xx_ptr = inputs[1].access_ptr("r") zz_ptr = outputs[0].access_ptr("w") compute = t.hardware_intrin("gemm8x8", ww_ptr, xx_ptr, zz_ptr) reset = t.hardware_intrin("fill_zero", zz_ptr) update = t.hardware_intrin("fuse_gemm8x8_add", ww_ptr, xx_ptr, zz_ptr) return compute, reset, update gemm8x8 = t.decl_tensor_intrin(y.op, gemm_intrin_lower) declare behavior lowering rule to generate hardware intrinsics to carry out the computation scalar tensor!17
61 Hardware-aware Search Space TPU-like Specialized Accelerators Compute Primitives tensor Memory Subsystem Unified Buffer FIFO Acc explicitly managed!18
62 Hardware-aware Search Space TPU-like Specialized Accelerators Compute Primitives tensor Memory Subsystem Unified Buffer FIFO Acc explicitly managed!18
63 Software Support for Latency Hiding Single Module No Task-Pipelining load inputs load weights matrix multiplication store outputs load inputs load weights matrix multiplication store outputs
64 Software Support for Latency Hiding Single Module No Task-Pipelining load inputs load weights matrix multiplication store outputs load inputs load weights matrix multiplication store outputs load inputs load weights load inputs load weights Multiple-Module Task-Level Pipelining matrix multiplication matrix multiplication store outputs store outputs
65 Software Support for Latency Hiding Single Module No Task-Pipelining load inputs load weights matrix multiplication store outputs load inputs load weights matrix multiplication store outputs load inputs load weights load inputs load weights Multiple-Module Task-Level Pipelining matrix multiplication matrix multiplication Explicit dependency tracking store outputs store outputs managed by software to hide memory latency
66 Hardware-aware Search Space Tensor Expression Language C = tvm.compute((m, n), lambda y, x: tvm.sum(a[k, y] * B[k, x], axis=k)) Hardware!21
67 Hardware-aware Search Space Tensor Expression Language C = tvm.compute((m, n), lambda y, x: tvm.sum(a[k, y] * B[k, x], axis=k)) Primitives in prior work: Loop Thread Cache Halide, Loopy Transformations Bindings Locality Hardware!21
68 Hardware-aware Search Space Tensor Expression Language C = tvm.compute((m, n), lambda y, x: tvm.sum(a[k, y] * B[k, x], axis=k)) Primitives in prior work: Loop Thread Cache Halide, Loopy Transformations Bindings Locality New primitives for GPUs, Thread Latency and enable TPU-like Cooperation Tensorization Hiding Accelerators Hardware!21
69 Hardware-aware Search Space Tensor Expression Language C = tvm.compute((m, n), lambda y, x: tvm.sum(a[k, y] * B[k, x], axis=k)) Loop Transformations Thread Bindings Cache Locality Thread Cooperation Tensorization Latency Hiding Hardware!22
70 Hardware-aware Search Space Tensor Expression Language C = tvm.compute((m, n), lambda y, x: tvm.sum(a[k, y] * B[k, x], axis=k)) Loop Transformations Thread Bindings Cache Locality Thread Cooperation Tensorization Latency Hiding Hardware!22
71 Hardware-aware Search Space Tensor Expression Language C = tvm.compute((m, n), lambda y, x: tvm.sum(a[k, y] * B[k, x], axis=k)) Billions Loop Thread Cache of possible Transformations Bindings Locality optimization Thread Latency choices Cooperation Tensorization Hiding Hardware!22
72 Learning-based Learning System Frameworks High-level data flow graph and optimizations Hardware aware Search Space of Optimized Tensor Programs Machine Learning based Program Optimizer Hardware!23
73 Learning-based Learning System Frameworks High-level data flow graph and optimizations Hardware aware Search Space of Optimized Tensor Programs Machine Learning based Program Optimizer Hardware!23
74 Learning-based Program Optimizer Program Optimizer!24
75 Learning-based Program Optimizer Program Optimizer Code Generator Program!24
76 Learning-based Program Optimizer Program Optimizer Code Generator Program Runtime Measurements!24
77 Learning-based Program Optimizer Program Optimizer Code Generator Program Runtime Measurements High experiment cost, each trial costs ~1second!24
78 Learning-based Program Optimizer Program Optimizer Code Generator Program!25
79 Learning-based Program Optimizer Program Optimizer Code Generator Program Cost Model!25
80 Learning-based Program Optimizer Program Optimizer Code Generator Program Cost Model Need reliable cost model per hardware!25
81 Learning-based Program Optimizer Program Optimizer Code Generator Program!26
82 D<latexit sha1_base64="1z6czjbl0omvztfq+m452ydkcy0=">aaab8nicbvdlssnafl2pr1pfvzdugkvwvrirdfnuhcsk9gftkjpppb06mqkzn0ij/qw3lhrx69e482+ctflo64gbwzn3mueembhcood9o6w19y3nrfj2zwd3b/+genjunirvllwoekp3q2ky4jk1kkng3uqzeoecdcljbe53npg2xmlhncysimli8ohtglbq9wocy0pedjcbvgte3zvdxsv+qwpqodmofvwhiqyxk0gfmabnewkggdhiqwczsj81lcf0qkasz6kkmtnbno88c8+smnqjpe2t6m7v3xsziy2zxqgdzcoazs8x//n6kubxqczlkiktdpfrlaoxlzvf7w65zhtf1bjcnbdzxtommlc0lvvscf7yyaukfvh3vbr/cflr3br1loeetuecflicbtxde1paqcezvmkbg86l8+58lezltrfzdh/gfp4adn2rwg==</latexit> <latexit sha1_base64="1z6czjbl0omvztfq+m452ydkcy0=">aaab8nicbvdlssnafl2pr1pfvzdugkvwvrirdfnuhcsk9gftkjpppb06mqkzn0ij/qw3lhrx69e482+ctflo64gbwzn3mueembhcood9o6w19y3nrfj2zwd3b/+genjunirvllwoekp3q2ky4jk1kkng3uqzeoecdcljbe53npg2xmlhncysimli8ohtglbq9wocy0pedjcbvgte3zvdxsv+qwpqodmofvwhiqyxk0gfmabnewkggdhiqwczsj81lcf0qkasz6kkmtnbno88c8+smnqjpe2t6m7v3xsziy2zxqgdzcoazs8x//n6kubxqczlkiktdpfrlaoxlzvf7w65zhtf1bjcnbdzxtommlc0lvvscf7yyaukfvh3vbr/cflr3br1loeetuecflicbtxde1paqcezvmkbg86l8+58lezltrfzdh/gfp4adn2rwg==</latexit> <latexit sha1_base64="1z6czjbl0omvztfq+m452ydkcy0=">aaab8nicbvdlssnafl2pr1pfvzdugkvwvrirdfnuhcsk9gftkjpppb06mqkzn0ij/qw3lhrx69e482+ctflo64gbwzn3mueembhcood9o6w19y3nrfj2zwd3b/+genjunirvllwoekp3q2ky4jk1kkng3uqzeoecdcljbe53npg2xmlhncysimli8ohtglbq9wocy0pedjcbvgte3zvdxsv+qwpqodmofvwhiqyxk0gfmabnewkggdhiqwczsj81lcf0qkasz6kkmtnbno88c8+smnqjpe2t6m7v3xsziy2zxqgdzcoazs8x//n6kubxqczlkiktdpfrlaoxlzvf7w65zhtf1bjcnbdzxtommlc0lvvscf7yyaukfvh3vbr/cflr3br1loeetuecflicbtxde1paqcezvmkbg86l8+58lezltrfzdh/gfp4adn2rwg==</latexit> <latexit sha1_base64="1z6czjbl0omvztfq+m452ydkcy0=">aaab8nicbvdlssnafl2pr1pfvzdugkvwvrirdfnuhcsk9gftkjpppb06mqkzn0ij/qw3lhrx69e482+ctflo64gbwzn3mueembhcood9o6w19y3nrfj2zwd3b/+genjunirvllwoekp3q2ky4jk1kkng3uqzeoecdcljbe53npg2xmlhncysimli8ohtglbq9wocy0pedjcbvgte3zvdxsv+qwpqodmofvwhiqyxk0gfmabnewkggdhiqwczsj81lcf0qkasz6kkmtnbno88c8+smnqjpe2t6m7v3xsziy2zxqgdzcoazs8x//n6kubxqczlkiktdpfrlaoxlzvf7w65zhtf1bjcnbdzxtommlc0lvvscf7yyaukfvh3vbr/cflr3br1loeetuecflicbtxde1paqcezvmkbg86l8+58lezltrfzdh/gfp4adn2rwg==</latexit> Learning-based Program Optimizer Program Optimizer Code Generator Program Training data!26
83 D<latexit sha1_base64="1z6czjbl0omvztfq+m452ydkcy0=">aaab8nicbvdlssnafl2pr1pfvzdugkvwvrirdfnuhcsk9gftkjpppb06mqkzn0ij/qw3lhrx69e482+ctflo64gbwzn3mueembhcood9o6w19y3nrfj2zwd3b/+genjunirvllwoekp3q2ky4jk1kkng3uqzeoecdcljbe53npg2xmlhncysimli8ohtglbq9wocy0pedjcbvgte3zvdxsv+qwpqodmofvwhiqyxk0gfmabnewkggdhiqwczsj81lcf0qkasz6kkmtnbno88c8+smnqjpe2t6m7v3xsziy2zxqgdzcoazs8x//n6kubxqczlkiktdpfrlaoxlzvf7w65zhtf1bjcnbdzxtommlc0lvvscf7yyaukfvh3vbr/cflr3br1loeetuecflicbtxde1paqcezvmkbg86l8+58lezltrfzdh/gfp4adn2rwg==</latexit> <latexit sha1_base64="1z6czjbl0omvztfq+m452ydkcy0=">aaab8nicbvdlssnafl2pr1pfvzdugkvwvrirdfnuhcsk9gftkjpppb06mqkzn0ij/qw3lhrx69e482+ctflo64gbwzn3mueembhcood9o6w19y3nrfj2zwd3b/+genjunirvllwoekp3q2ky4jk1kkng3uqzeoecdcljbe53npg2xmlhncysimli8ohtglbq9wocy0pedjcbvgte3zvdxsv+qwpqodmofvwhiqyxk0gfmabnewkggdhiqwczsj81lcf0qkasz6kkmtnbno88c8+smnqjpe2t6m7v3xsziy2zxqgdzcoazs8x//n6kubxqczlkiktdpfrlaoxlzvf7w65zhtf1bjcnbdzxtommlc0lvvscf7yyaukfvh3vbr/cflr3br1loeetuecflicbtxde1paqcezvmkbg86l8+58lezltrfzdh/gfp4adn2rwg==</latexit> <latexit sha1_base64="1z6czjbl0omvztfq+m452ydkcy0=">aaab8nicbvdlssnafl2pr1pfvzdugkvwvrirdfnuhcsk9gftkjpppb06mqkzn0ij/qw3lhrx69e482+ctflo64gbwzn3mueembhcood9o6w19y3nrfj2zwd3b/+genjunirvllwoekp3q2ky4jk1kkng3uqzeoecdcljbe53npg2xmlhncysimli8ohtglbq9wocy0pedjcbvgte3zvdxsv+qwpqodmofvwhiqyxk0gfmabnewkggdhiqwczsj81lcf0qkasz6kkmtnbno88c8+smnqjpe2t6m7v3xsziy2zxqgdzcoazs8x//n6kubxqczlkiktdpfrlaoxlzvf7w65zhtf1bjcnbdzxtommlc0lvvscf7yyaukfvh3vbr/cflr3br1loeetuecflicbtxde1paqcezvmkbg86l8+58lezltrfzdh/gfp4adn2rwg==</latexit> <latexit sha1_base64="1z6czjbl0omvztfq+m452ydkcy0=">aaab8nicbvdlssnafl2pr1pfvzdugkvwvrirdfnuhcsk9gftkjpppb06mqkzn0ij/qw3lhrx69e482+ctflo64gbwzn3mueembhcood9o6w19y3nrfj2zwd3b/+genjunirvllwoekp3q2ky4jk1kkng3uqzeoecdcljbe53npg2xmlhncysimli8ohtglbq9wocy0pedjcbvgte3zvdxsv+qwpqodmofvwhiqyxk0gfmabnewkggdhiqwczsj81lcf0qkasz6kkmtnbno88c8+smnqjpe2t6m7v3xsziy2zxqgdzcoazs8x//n6kubxqczlkiktdpfrlaoxlzvf7w65zhtf1bjcnbdzxtommlc0lvvscf7yyaukfvh3vbr/cflr3br1loeetuecflicbtxde1paqcezvmkbg86l8+58lezltrfzdh/gfp4adn2rwg==</latexit> Learning-based Program Optimizer Program Optimizer Code Generator Program Learning Statistical Cost Model Training data!26
84 D<latexit sha1_base64="1z6czjbl0omvztfq+m452ydkcy0=">aaab8nicbvdlssnafl2pr1pfvzdugkvwvrirdfnuhcsk9gftkjpppb06mqkzn0ij/qw3lhrx69e482+ctflo64gbwzn3mueembhcood9o6w19y3nrfj2zwd3b/+genjunirvllwoekp3q2ky4jk1kkng3uqzeoecdcljbe53npg2xmlhncysimli8ohtglbq9wocy0pedjcbvgte3zvdxsv+qwpqodmofvwhiqyxk0gfmabnewkggdhiqwczsj81lcf0qkasz6kkmtnbno88c8+smnqjpe2t6m7v3xsziy2zxqgdzcoazs8x//n6kubxqczlkiktdpfrlaoxlzvf7w65zhtf1bjcnbdzxtommlc0lvvscf7yyaukfvh3vbr/cflr3br1loeetuecflicbtxde1paqcezvmkbg86l8+58lezltrfzdh/gfp4adn2rwg==</latexit> <latexit sha1_base64="1z6czjbl0omvztfq+m452ydkcy0=">aaab8nicbvdlssnafl2pr1pfvzdugkvwvrirdfnuhcsk9gftkjpppb06mqkzn0ij/qw3lhrx69e482+ctflo64gbwzn3mueembhcood9o6w19y3nrfj2zwd3b/+genjunirvllwoekp3q2ky4jk1kkng3uqzeoecdcljbe53npg2xmlhncysimli8ohtglbq9wocy0pedjcbvgte3zvdxsv+qwpqodmofvwhiqyxk0gfmabnewkggdhiqwczsj81lcf0qkasz6kkmtnbno88c8+smnqjpe2t6m7v3xsziy2zxqgdzcoazs8x//n6kubxqczlkiktdpfrlaoxlzvf7w65zhtf1bjcnbdzxtommlc0lvvscf7yyaukfvh3vbr/cflr3br1loeetuecflicbtxde1paqcezvmkbg86l8+58lezltrfzdh/gfp4adn2rwg==</latexit> <latexit sha1_base64="1z6czjbl0omvztfq+m452ydkcy0=">aaab8nicbvdlssnafl2pr1pfvzdugkvwvrirdfnuhcsk9gftkjpppb06mqkzn0ij/qw3lhrx69e482+ctflo64gbwzn3mueembhcood9o6w19y3nrfj2zwd3b/+genjunirvllwoekp3q2ky4jk1kkng3uqzeoecdcljbe53npg2xmlhncysimli8ohtglbq9wocy0pedjcbvgte3zvdxsv+qwpqodmofvwhiqyxk0gfmabnewkggdhiqwczsj81lcf0qkasz6kkmtnbno88c8+smnqjpe2t6m7v3xsziy2zxqgdzcoazs8x//n6kubxqczlkiktdpfrlaoxlzvf7w65zhtf1bjcnbdzxtommlc0lvvscf7yyaukfvh3vbr/cflr3br1loeetuecflicbtxde1paqcezvmkbg86l8+58lezltrfzdh/gfp4adn2rwg==</latexit> <latexit sha1_base64="1z6czjbl0omvztfq+m452ydkcy0=">aaab8nicbvdlssnafl2pr1pfvzdugkvwvrirdfnuhcsk9gftkjpppb06mqkzn0ij/qw3lhrx69e482+ctflo64gbwzn3mueembhcood9o6w19y3nrfj2zwd3b/+genjunirvllwoekp3q2ky4jk1kkng3uqzeoecdcljbe53npg2xmlhncysimli8ohtglbq9wocy0pedjcbvgte3zvdxsv+qwpqodmofvwhiqyxk0gfmabnewkggdhiqwczsj81lcf0qkasz6kkmtnbno88c8+smnqjpe2t6m7v3xsziy2zxqgdzcoazs8x//n6kubxqczlkiktdpfrlaoxlzvf7w65zhtf1bjcnbdzxtommlc0lvvscf7yyaukfvh3vbr/cflr3br1loeetuecflicbtxde1paqcezvmkbg86l8+58lezltrfzdh/gfp4adn2rwg==</latexit> Learning-based Program Optimizer Program Optimizer Code Generator Program Learning Statistical Cost Model Training data Adapt to hardware type by learning Make prediction in 1ms level!26
85 Effectiveness of ML based Model !27
86 Effectiveness of ML based Model One Conv2D Layer of ResNet18 on Titan X !27
87 Effectiveness of ML based Model One Conv2D Layer of ResNet18 on Titan X Number of Trials!27
88 Effectiveness of ML based Model 1.50 Relative Speedup Baseline: CuDNN 0.25 One Conv2D Layer of ResNet18 on Titan X Number of Trials!27
89 Effectiveness of ML based Model 1.50 Relative Speedup TVM: Random Search Baseline: CuDNN 0.25 One Conv2D Layer of ResNet18 on Titan X Number of Trials!28
90 Effectiveness of ML based Model 1.50 TVM: ML-based Model Relative Speedup TVM: Random Search Baseline: CuDNN 0.25 One Conv2D Layer of ResNet18 on Titan X Number of Trials!29
91 Learning-based Learning System Frameworks High-level data flow graph and optimizations Hardware aware Search Space of Optimized Tensor Programs Machine Learning based Program Optimizer Hardware!30
92 End to End Inference Performance (Nvidia Titan X) Tensorflow Apache MXNet Tensorflow-XLA Time(ms) ResNet-18 MobileNet 0.0 LSTM LM DQN DCGAN
93 End to End Inference Performance (Nvidia Titan X) Backed by cudnn Tensorflow Apache MXNet Tensorflow-XLA Time(ms) ResNet-18 MobileNet 0.0 LSTM LM DQN DCGAN
94 End to End Inference Performance (Nvidia Titan X) Tensorflow Apache MXNet TVM: without graph optimizations Tensorflow-XLA Time(ms) ResNet-18 MobileNet 0.0 LSTM LM DQN DCGAN
95 End to End Inference Performance (Nvidia Titan X) Tensorflow Tensorflow-XLA Apache MXNet TVM: without graph optimizations TVM: all optimizations Time(ms) ResNet-18 MobileNet 0.0 LSTM LM DQN DCGAN
96 End to End Inference Performance (Nvidia Titan X) Tensorflow Tensorflow-XLA Apache MXNet TVM: without graph optimizations TVM: all optimizations Competitive on standard models Time(ms) ResNet-18 MobileNet 0.0 LSTM LM DQN DCGAN
97 End to End Inference Performance (Nvidia Titan X) Tensorflow Tensorflow-XLA Apache MXNet TVM: without graph optimizations TVM: all optimizations Competitive on standard models Time(ms) Bigger gap on less conventional models 0.0 ResNet-18 MobileNet 0.0 LSTM LM DQN DCGAN
98 End to End Performance(ARM Cortex-A53) Tensorflow Lite TVM w/o graph opt TVM Time(ms) ResNet-18 MobileNet! DQN
99 End to End Performance(ARM Cortex-A53) Specially optimized for Embedded system(arm) Tensorflow Lite TVM w/o graph opt TVM Time(ms) ResNet-18 MobileNet! DQN
100 End to End Performance(ARM GPU) ARMComputeLib TVM w/o graph opt TVM Time (ms) float32 ResNet-18 float16 float32 MobileNet float16 float32 float16 DQN!35
101 Supporting New Specialized Accelerators Frameworks High-level data flow graph and optimizations Hardware aware Search Space of Optimized Tensor Programs Machine Learning based Program Optimizer LLVM CUDA!36
102 Supporting New Specialized Accelerators Frameworks High-level data flow graph and optimizations Hardware aware Search Space of Optimized Tensor Programs Machine Learning based Program Optimizer LLVM CUDA VTA: Open, Customizable Deep Learning Accelerator Edge FPGA Data Center FPGA ASIC!36
103 TVM/VTA: Full Stack Open Source System High-level Optimizations Tensor Program Search Space ML-based Optimizer!37
104 TVM/VTA: Full Stack Open Source System High-level Optimizations Tensor Program Search Space ML-based Optimizer VTA MicroArchitecture!37
105 TVM/VTA: Full Stack Open Source System High-level Optimizations Tensor Program Search Space ML-based Optimizer VTA Hardware/Software Interface (ISA) VTA MicroArchitecture!37
106 TVM/VTA: Full Stack Open Source System High-level Optimizations Tensor Program Search Space ML-based Optimizer VTA Runtime & JIT Compiler VTA Hardware/Software Interface (ISA) VTA MicroArchitecture!37
107 TVM/VTA: Full Stack Open Source System High-level Optimizations Tensor Program Search Space ML-based Optimizer VTA Runtime & JIT Compiler VTA Hardware/Software Interface (ISA) VTA MicroArchitecture VTA Simulator!37
108 TVM/VTA: Full Stack Open Source System High-level Optimizations Tensor Program Search Space ML-based Optimizer VTA Runtime & JIT Compiler JIT compile accelerator micro code Support heterogenous devices, 10x better than CPU on the same board. VTA Hardware/Software Interface (ISA) VTA MicroArchitecture VTA Simulator Move hardware complexity to software!37
109 TVM/VTA: Full Stack Open Source System High-level Optimizations Tensor Program Search Space ML-based Optimizer VTA Runtime & JIT Compiler JIT compile accelerator micro code Support heterogenous devices, 10x better than CPU on the same board. VTA Hardware/Software Interface (ISA) VTA MicroArchitecture VTA Simulator Move hardware complexity to software }compiler, driver, hardware design!37 full stack open source
110 TVM: Learning-based Learning System High-level Optimizations Tensor Program Search Space ML-based Optimizer Check it out! LLVM CUDA VTA
TVM Stack Overview. Tianqi Chen
TVM Stack Overview Tianqi Chen Beginning of TVM Story Acclerator Beginning of TVM Story Acclerator Beginning of TVM Story Beginning of TVM Story Acclerator // Pseudo-code for convolution program for the
VTA: Open & Flexible DL Acceleration. Thierry Moreau TVM Conference, Dec 12th 2018
VTA: Open & Flexible DL Acceleration Thierry Moreau TVM Conference, Dec 12th 2018 TVM Stack High-Level Differentiable IR Tensor Expression IR LLVM CUDA Metal TVM Stack High-Level Differentiable IR Tensor
End to End Optimization Stack for Deep Learning
End to End Optimization Stack for Deep Learning Presenter: Tianqi Chen Paul G. Allen School of Computer Science & Engineering University of Washington Collaborators University of Washington AWS AI Team
Automatic Code Generation TVM Stack
Automatic Code Generation TVM Stack CSE 599W Spring TVM stack is an active project by saml.cs.washington.edu and many partners in the open source community The Gap between Framework and Hardware Frameworks
TVM: An Automated End-to-End Optimizing Compiler for Deep Learning
TVM: An Automated End-to-End Optimizing Compiler for Deep Learning Tianqi Chen and Thierry Moreau, University of Washington; Ziheng Jiang, University of Washington, AWS; Lianmin Zheng, Shanghai Jiao Tong
AutoTVM & Device Fleet
AutoTVM & Device Fleet ` Learning to Optimize Tensor Programs Frameworks High-level data flow graph and optimizations Hardware Learning to Optimize Tensor Programs Frameworks High-level data flow graph
arxiv: v2 [cs.lg] 20 May 2018
![arxiv: v2 [cs.lg] 20 May 2018](/thumbs/81/82954790.jpg "arxiv: v2 [cs.lg] 20 May 2018")
TVM: An Automated End-to-End Optimizing Compiler for Deep Learning Tianqi Chen 1, Thierry Moreau 1, Ziheng Jiang 2, Lianmin Zheng 3, Eddie Yan 1 Meghan Cowan 1, Haichen Shen 1, Leyuan Wang 4, Yuwei Hu
Optimizing CNN Inference on CPUs
Optimizing CNN Inference on CPUs Yizhi Liu, Yao Wang, Yida Wang With others in AWS AI Agenda Deep learning inference optimization Optimization on Intel CPUs Evaluation Make DL inference easier and faster
arxiv: v1 [cs.lg] 11 Jul 2018
![arxiv: v1 [cs.lg] 11 Jul 2018](/thumbs/90/102842714.jpg "arxiv: v1 [cs.lg] 11 Jul 2018")
VTA: An Open Hardware-Software Stack for Deep Learning Thierry Moreau, Tianqi Chen, Ziheng Jiang, Luis Ceze, Carlos Guestrin, Arvind Krishnamurthy Paul G. Allen School of Computer Science & Engineering,
Relay: a high level differentiable IR. Jared Roesch TVMConf December 12th, 2018
Relay: a high level differentiable IR Jared Roesch TVMConf December 12th, 2018!1 This represents months of joint work with lots of great folks:!2 TVM Stack Optimization Relay High-Level Differentiable
Deep Learning Accelerators
Deep Learning Accelerators Abhishek Srivastava (as29) Samarth Kulshreshtha (samarth5) University of Illinois, Urbana-Champaign Submitted as a requirement for CS 433 graduate student project Outline Introduction
Research Faculty Summit Systems Fueling future disruptions
Research Faculty Summit 2018 Systems Fueling future disruptions Wolong: A Back-end Optimizer for Deep Learning Computation Jilong Xue Researcher, Microsoft Research Asia System Challenge in Deep Learning
Scalable Distributed Training with Parameter Hub: a whirlwind tour
Scalable Distributed Training with Parameter Hub: a whirlwind tour TVM Stack Optimization High-Level Differentiable IR Tensor Expression IR AutoTVM LLVM, CUDA, Metal VTA AutoVTA Edge FPGA Cloud FPGA ASIC
Deep Learning Compiler
Deep Learning Compiler AWS AI Acknowledgement Amazon Sagemaker Neo Enables developers to train machine learning models once and run them anywhere in the cloud and at the edge Hardware targets Intel CPU,
Recurrent Neural Networks. Deep neural networks have enabled major advances in machine learning and AI. Convolutional Neural Networks
Deep neural networks have enabled major advances in machine learning and AI Computer vision Language translation Speech recognition Question answering And more Problem: DNNs are challenging to serve and
Unified Deep Learning with CPU, GPU, and FPGA Technologies
Unified Deep Learning with CPU, GPU, and FPGA Technologies Allen Rush 1, Ashish Sirasao 2, Mike Ignatowski 1 1: Advanced Micro Devices, Inc., 2: Xilinx, Inc. Abstract Deep learning and complex machine
The HammerBlade: An ML-Optimized Supercomputer for ML and Graphs
The HammerBlade: An ML-Optimized Supercomputer for ML and Graphs Prof. Michael B. Taylor (PI) University of Washington Prof. Adrian Sampson Cornell University Prof. Luis Ceze University of Washington Prof.
HKG OpenCL Support by NNVM & TVM. Jammy Zhou - Linaro
HKG18-417 OpenCL Support by NNVM & TVM Jammy Zhou - Linaro Agenda OpenCL Overview OpenCL in NNVM & TVM Current Status OpenCL Introduction Open Computing Language Open standard maintained by Khronos with
Two FPGA-DNN Projects: 1. Low Latency Multi-Layer Perceptrons using FPGAs 2. Acceleration of CNN Training on FPGA-based Clusters
Two FPGA-DNN Projects: 1. Low Latency Multi-Layer Perceptrons using FPGAs 2. Acceleration of CNN Training on FPGA-based Clusters *Argonne National Lab +BU & USTC Presented by Martin Herbordt Work by Ahmed
Xilinx ML Suite Overview
Xilinx ML Suite Overview Yao Fu System Architect Data Center Acceleration Xilinx Accelerated Computing Workloads Machine Learning Inference Image classification and object detection Video Streaming Frame
Nvidia Jetson TX2 and its Software Toolset. João Fernandes 2017/2018
Nvidia Jetson TX2 and its Software Toolset João Fernandes 2017/2018 In this presentation Nvidia Jetson TX2: Hardware Nvidia Jetson TX2: Software Machine Learning: Neural Networks Convolutional Neural Networks
NVIDIA GPU CLOUD DEEP LEARNING FRAMEWORKS
TECHNICAL OVERVIEW NVIDIA GPU CLOUD DEEP LEARNING FRAMEWORKS A Guide to the Optimized Framework Containers on NVIDIA GPU Cloud Introduction Artificial intelligence is helping to solve some of the most
High Performance Computing
High Performance Computing 9th Lecture 2016/10/28 YUKI ITO 1 Selected Paper: vdnn: Virtualized Deep Neural Networks for Scalable, MemoryEfficient Neural Network Design Minsoo Rhu, Natalia Gimelshein, Jason
Speculations about Computer Architecture in Next Three Years. Jan. 20, 2018
Speculations about Computer Architecture in Next Three Years shuchang.zhou@gmail.com Jan. 20, 2018 About me https://zsc.github.io/ Source-to-source transformation Cache simulation Compiler Optimization
Processing-in-Memory for Energy-efficient Neural Network Training: A Heterogeneous Approach
2018 51st Annual IEEE/ACM International Symposium on Microarchitecture Processing-in-Memory for Energy-efficient Neural Network Training: A Heterogeneous Approach Jiawen Liu*, Hengyu Zhao*, Matheus Almeida
Can FPGAs beat GPUs in accelerating next-generation Deep Neural Networks? Discussion of the FPGA 17 paper by Intel Corp. (Nurvitadhi et al.
Can FPGAs beat GPUs in accelerating next-generation Deep Neural Networks? Discussion of the FPGA 17 paper by Intel Corp. (Nurvitadhi et al.) Andreas Kurth 2017-12-05 1 In short: The situation Image credit:
GPU-Accelerated Deep Learning
GPU-Accelerated Deep Learning July 6 th, 2016. Greg Heinrich. Credits: Alison B. Lowndes, Julie Bernauer, Leo K. Tam. PRACTICAL DEEP LEARNING EXAMPLES Image Classification, Object Detection, Localization,
Improving Performance of Machine Learning Workloads
Improving Performance of Machine Learning Workloads Dong Li Parallel Architecture, System, and Algorithm Lab Electrical Engineering and Computer Science School of Engineering University of California,
A GPU-Inspired Soft Processor for High- Throughput Acceleration
A GPU-Inspired Soft Processor for High- Throughput Acceleration Jeffrey Kingyens and J. Gregory Steffan Electrical and Computer Engineering University of Toronto 1 FGPA-Based Acceleration In-socket acceleration
Inference Optimization Using TensorRT with Use Cases. Jack Han / 한재근 Solutions Architect NVIDIA
Inference Optimization Using TensorRT with Use Cases Jack Han / 한재근 Solutions Architect NVIDIA Search Image NLP Maps TensorRT 4 Adoption Use Cases Speech Video AI Inference is exploding 1 Billion Videos
Lecture 6: Optimize for Hardware Backends. CSE599G1: Spring 2017
Lecture 6: Optimize for Hardware Backends CSE599G1: Spring 2017 Where are we High level Packages User API Programming API Gradient Calculation (Differentiation API) System Components Computational Graph
Master Informatics Eng.
Advanced Architectures Master Informatics Eng. 2018/19 A.J.Proença Data Parallelism 3 (GPU/CUDA, Neural Nets,...) (most slides are borrowed) AJProença, Advanced Architectures, MiEI, UMinho, 2018/19 1 The
Deep learning in MATLAB From Concept to CUDA Code
Deep learning in MATLAB From Concept to CUDA Code Roy Fahn Applications Engineer Systematics royf@systematics.co.il 03-7660111 Ram Kokku Principal Engineer MathWorks ram.kokku@mathworks.com 2017 The MathWorks,
Shrinath Shanbhag Senior Software Engineer Microsoft Corporation
Accelerating GPU inferencing with DirectML and DirectX 12 Shrinath Shanbhag Senior Software Engineer Microsoft Corporation Machine Learning Machine learning has become immensely popular over the last decade
Neural Network Exchange Format
Copyright Khronos Group 2017 - Page 1 Neural Network Exchange Format Deploying Trained Networks to Inference Engines Viktor Gyenes, specification editor Copyright Khronos Group 2017 - Page 2 Outlook The
Graph Streaming Processor
Graph Streaming Processor A Next-Generation Computing Architecture Val G. Cook Chief Software Architect Satyaki Koneru Chief Technology Officer Ke Yin Chief Scientist Dinakar Munagala Chief Executive Officer
Deep Learning on Modern Architectures. Keren Zhou 4/17/2017
Deep Learning on Modern Architectures Keren Zhou 4/17/2017 HPC Software Stack Application Algorithm Data Layout CPU GPU MIC Others HPC Software Stack Deep Learning Algorithm Data Layout CPU GPU MIC Others
Onto Petaflops with Kubernetes
Onto Petaflops with Kubernetes Vishnu Kannan Google Inc. vishh@google.com Key Takeaways Kubernetes can manage hardware accelerators at Scale Kubernetes provides a playground for ML ML journey with Kubernetes
Multi-dimensional Parallel Training of Winograd Layer on Memory-Centric Architecture
The 51st Annual IEEE/ACM International Symposium on Microarchitecture Multi-dimensional Parallel Training of Winograd Layer on Memory-Centric Architecture Byungchul Hong Yeonju Ro John Kim FuriosaAI Samsung
PRIME: A Novel Processing-in-memory Architecture for Neural Network Computation in ReRAM-based Main Memory
Scalable and Energy-Efficient Architecture Lab (SEAL) PRIME: A Novel Processing-in-memory Architecture for Neural Network Computation in -based Main Memory Ping Chi *, Shuangchen Li *, Tao Zhang, Cong
Open Standards for Vision and AI Peter McGuinness NNEF WG Chair CEO, Highwai, Inc May 2018
Copyright Khronos Group 2018 - Page 1 Open Standards for Vision and AI Peter McGuinness NNEF WG Chair CEO, Highwai, Inc peter.mcguinness@gobrach.com May 2018 Khronos Mission E.g. OpenGL ES provides 3D
A Cross-Input Adaptive Framework for GPU Program Optimizations
A Cross-Input Adaptive Framework for GPU Program Optimizations Yixun Liu, Eddy Z. Zhang, Xipeng Shen Computer Science Department The College of William & Mary Outline GPU overview G-Adapt Framework Evaluation
Versal: AI Engine & Programming Environment
Engineering Director, Xilinx Silicon Architecture Group Versal: Engine & Programming Environment Presented By Ambrose Finnerty Xilinx DSP Technical Marketing Manager October 16, 2018 MEMORY MEMORY MEMORY
Wu Zhiwen.
Wu Zhiwen zhiwen.wu@intel.com Agenda Background information OpenCV DNN module OpenCL acceleration Vulkan backend Sample 2 What is OpenCV? Open Source Compute Vision (OpenCV) library 2500+ Optimized algorithms
When MPPDB Meets GPU:
When MPPDB Meets GPU: An Extendible Framework for Acceleration Laura Chen, Le Cai, Yongyan Wang Background: Heterogeneous Computing Hardware Trend stops growing with Moore s Law Fast development of GPU
Embarquez votre Intelligence Artificielle (IA) sur CPU, GPU et FPGA
Embarquez votre Intelligence Artificielle (IA) sur CPU, GPU et FPGA Pierre Nowodzienski Engineer pierre.nowodzienski@mathworks.fr 2018 The MathWorks, Inc. 1 From Data to Business value Make decisions Get
Big Data Systems on Future Hardware. Bingsheng He NUS Computing
Big Data Systems on Future Hardware Bingsheng He NUS Computing http://www.comp.nus.edu.sg/~hebs/ 1 Outline Challenges for Big Data Systems Why Hardware Matters? Open Challenges Summary 2 3 ANYs in Big
Slalom: Fast, Verifiable and Private Execution of Neural Networks in Trusted Hardware
Slalom: Fast, Verifiable and Private Execution of Neural Networks in Trusted Hardware Florian Tramèr (joint work with Dan Boneh) Stanford security lunch June 13 th Trusted execution of ML: 3 motivating
TensorFlow: A System for Learning-Scale Machine Learning. Google Brain
TensorFlow: A System for Learning-Scale Machine Learning Google Brain The Problem Machine learning is everywhere This is in large part due to: 1. Invention of more sophisticated machine learning models
Operator Vectorization Library A TensorFlow Plugin
Operator Vectorization Library A TensorFlow Plugin Matthew Pickett, Karen Brems, Florian Raudies Hewlett Packard Labs HPE-2016-94 Keyword(s): Machine learning; GPU acceleration; TensorFlow Abstract: TensorFlow
MIXED PRECISION TRAINING OF NEURAL NETWORKS. Carl Case, Senior Architect, NVIDIA
MIXED PRECISION TRAINING OF NEURAL NETWORKS Carl Case, Senior Architect, NVIDIA OUTLINE 1. What is mixed precision training with FP16? 2. Considerations and methodology for mixed precision training 3.
Scaling Distributed Machine Learning
Scaling Distributed Machine Learning with System and Algorithm Co-design Mu Li Thesis Defense CSD, CMU Feb 2nd, 2017 nx min w f i (w) Distributed systems i=1 Large scale optimization methods Large-scale
GPU Coder: Automatic CUDA and TensorRT code generation from MATLAB
GPU Coder: Automatic CUDA and TensorRT code generation from MATLAB Ram Kokku 2018 The MathWorks, Inc. 1 GPUs and CUDA programming faster Performance CUDA OpenCL C/C++ GPU Coder MATLAB Python Ease of programming
Predictive Runtime Code Scheduling for Heterogeneous Architectures
Predictive Runtime Code Scheduling for Heterogeneous Architectures Víctor Jiménez, Lluís Vilanova, Isaac Gelado Marisa Gil, Grigori Fursin, Nacho Navarro HiPEAC 2009 January, 26th, 2009 1 Outline Motivation
Lift: a Functional Approach to Generating High Performance GPU Code using Rewrite Rules
Lift: a Functional Approach to Generating High Performance GPU Code using Rewrite Rules Toomas Remmelg Michel Steuwer Christophe Dubach The 4th South of England Regional Programming Language Seminar 27th
Accelerate Machine Learning on macos with Intel Integrated Graphics. Hisham Chowdhury May 23, 2018
Accelerate Machine Learning on macos with Intel Integrated Graphics Hisham Chowdhury May 23, 2018 Apple Machine Learning Stack Machine Learning Application 1 Machine Learning Application 2 Vision Natural
A NEW COMPUTING ERA JENSEN HUANG, FOUNDER & CEO GTC CHINA 2017
A NEW COMPUTING ERA JENSEN HUANG, FOUNDER & CEO GTC CHINA 2017 TWO FORCES DRIVING THE FUTURE OF COMPUTING 10 7 Transistors (thousands) 10 6 10 5 1.1X per year 10 4 10 3 10 2 1.5X per year Single-threaded
GPUfs: Integrating a file system with GPUs
GPUfs: Integrating a file system with GPUs Mark Silberstein (UT Austin/Technion) Bryan Ford (Yale), Idit Keidar (Technion) Emmett Witchel (UT Austin) 1 Traditional System Architecture Applications OS CPU
XPU A Programmable FPGA Accelerator for Diverse Workloads
XPU A Programmable FPGA Accelerator for Diverse Workloads Jian Ouyang, 1 (ouyangjian@baidu.com) Ephrem Wu, 2 Jing Wang, 1 Yupeng Li, 1 Hanlin Xie 1 1 Baidu, Inc. 2 Xilinx Outlines Background - FPGA for
Slalom: Fast, Verifiable and Private Execution of Neural Networks in Trusted Hardware
Slalom: Fast, Verifiable and Private Execution of Neural Networks in Trusted Hardware Florian Tramèr (joint work with Dan Boneh) Intel, Santa Clara August 30 th 2018 Trusted execution of ML: 3 motivating
Matchbox. automatic batching for imperative deep learning NVIDIA GTC, 2018/3/28. James Bradbury
Matchbox automatic batching for imperative deep learning James Bradbury NVIDIA GTC, 2018/3/28 Roadmap Imperative deep learning How manual batching works Other tools for automatic batching The Matchbox
COSC 6339 Accelerators in Big Data
COSC 6339 Accelerators in Big Data Edgar Gabriel Fall 2018 Motivation Programming models such as MapReduce and Spark provide a high-level view of parallelism not easy for all problems, e.g. recursive algorithms,
Why AI Frameworks Need (not only) RDMA?
Why AI Frameworks Need (not only) RDMA? With Design and Implementation Experience of Networking Support on TensorFlow GDR, Apache MXNet, WeChat Amber, and Tencent Angel Bairen Yi (byi@connect.ust.hk) Jingrong
Lecture 1 Introduction. I. Why Study Compilers? II.Example: Machine Learning III. Course Syllabus
Lecture 1 Introduction I. Why Study Compilers? II.Example: Machine Learning III. Course Syllabus Chapters 1.1-1.5, 8.4, 8.5, 9.1 CS243: Introduction 1 Why Study Compilers? CS243: Introduction 2 1 Reasons
Deploying Deep Learning Networks to Embedded GPUs and CPUs
Deploying Deep Learning Networks to Embedded GPUs and CPUs Rishu Gupta, PhD Senior Application Engineer, Computer Vision 2015 The MathWorks, Inc. 1 MATLAB Deep Learning Framework Access Data Design + Train
NVIDIA GTX200: TeraFLOPS Visual Computing. August 26, 2008 John Tynefield
NVIDIA GTX200: TeraFLOPS Visual Computing August 26, 2008 John Tynefield 2 Outline Execution Model Architecture Demo 3 Execution Model 4 Software Architecture Applications DX10 OpenGL OpenCL CUDA C Host
Deep Learning Inference in Facebook Data Centers: Characterization, Performance Optimizations, and Hardware Implications
Deep Learning Inference in Facebook Data Centers: Characterization, Performance Optimizations, and Hardware Implications Jongsoo Park Facebook AI System SW/HW Co-design Team Sep-21 2018 Team Introduction
MIXED PRECISION TRAINING: THEORY AND PRACTICE Paulius Micikevicius
MIXED PRECISION TRAINING: THEORY AND PRACTICE Paulius Micikevicius What is Mixed Precision Training? Reduced precision tensor math with FP32 accumulation, FP16 storage Successfully used to train a variety
VOLTA: PROGRAMMABILITY AND PERFORMANCE. Jack Choquette NVIDIA Hot Chips 2017
VOLTA: PROGRAMMABILITY AND PERFORMANCE Jack Choquette NVIDIA Hot Chips 2017 1 TESLA V100 21B transistors 815 mm 2 80 SM 5120 CUDA Cores 640 Tensor Cores 16 GB HBM2 900 GB/s HBM2 300 GB/s NVLink *full GV100
Toward Scalable Deep Learning
한국정보과학회 인공지능소사이어티 머신러닝연구회 두번째딥러닝워크샵 2015.10.16 Toward Scalable Deep Learning 윤성로 Electrical and Computer Engineering Seoul National University http://data.snu.ac.kr Breakthrough: Big Data + Machine Learning
Deep Learning Frameworks with Spark and GPUs
Deep Learning Frameworks with Spark and GPUs Abstract Spark is a powerful, scalable, real-time data analytics engine that is fast becoming the de facto hub for data science and big data. However, in parallel,
Lecture 9: Scheduling. CSE599G1: Spring 2017
Lecture 9: Scheduling CSE599G1: Spring 2017 Next Week Two Joint Sessions with Computer Architecture Class Different date, time and location, detail to be announced Wed: ASICs for deep learning Friday:
! Readings! ! Room-level, on-chip! vs.!
1! 2! Suggested Readings!! Readings!! H&P: Chapter 7 especially 7.1-7.8!! (Over next 2 weeks)!! Introduction to Parallel Computing!! https://computing.llnl.gov/tutorials/parallel_comp/!! POSIX Threads
GPU Programming with Ateji PX June 8 th Ateji All rights reserved.
GPU Programming with Ateji PX June 8 th 2010 Ateji All rights reserved. Goals Write once, run everywhere, even on a GPU Target heterogeneous architectures from Java GPU accelerators OpenCL standard Get
GPU FOR DEEP LEARNING. 周国峰 Wuhan University 2017/10/13
GPU FOR DEEP LEARNING chandlerz@nvidia.com 周国峰 Wuhan University 2017/10/13 Why Deep Learning Boost Today? Nvidia SDK for Deep Learning? Agenda CUDA 8.0 cudnn TensorRT (GIE) NCCL DIGITS 2 Why Deep Learning
Understanding GPGPU Vector Register File Usage
Understanding GPGPU Vector Register File Usage Mark Wyse AMD Research, Advanced Micro Devices, Inc. Paul G. Allen School of Computer Science & Engineering, University of Washington AGENDA GPU Architecture
High-Performance Data Loading and Augmentation for Deep Neural Network Training
High-Performance Data Loading and Augmentation for Deep Neural Network Training Trevor Gale tgale@ece.neu.edu Steven Eliuk steven.eliuk@gmail.com Cameron Upright c.upright@samsung.com Roadmap 1. The General-Purpose
Adaptable Intelligence The Next Computing Era
Adaptable Intelligence The Next Computing Era Hot Chips, August 21, 2018 Victor Peng, CEO, Xilinx Pervasive Intelligence from Cloud to Edge to Endpoints >> 1 Exponential Growth and Opportunities Data Explosion
SDA: Software-Defined Accelerator for Large- Scale DNN Systems
SDA: Software-Defined Accelerator for Large- Scale DNN Systems Jian Ouyang, 1 Shiding Lin, 1 Wei Qi, Yong Wang, Bo Yu, Song Jiang, 2 1 Baidu, Inc. 2 Wayne State University Introduction of Baidu A dominant
EXTENDING THE REACH OF PARALLEL COMPUTING WITH CUDA
EXTENDING THE REACH OF PARALLEL COMPUTING WITH CUDA Mark Harris, NVIDIA @harrism #NVSC14 EXTENDING THE REACH OF CUDA 1 Machine Learning 2 Higher Performance 3 New Platforms 4 New Languages 2 GPUS: THE
Deep Learning: Transforming Engineering and Science The MathWorks, Inc.
Deep Learning: Transforming Engineering and Science 1 2015 The MathWorks, Inc. DEEP LEARNING: TRANSFORMING ENGINEERING AND SCIENCE A THE NEW RISE ERA OF OF GPU COMPUTING 3 NVIDIA A IS NEW THE WORLD S ERA
Versal: The New Xilinx Adaptive Compute Acceleration Platform (ACAP) in 7nm
Engineering Director, Xilinx Silicon Architecture Group Versal: The New Xilinx Adaptive Compute Acceleration Platform (ACAP) in 7nm Presented By Kees Vissers Fellow February 25, FPGA 2019 Technology scaling
Integrating NVIDIA Deep Learning Accelerator (NVDLA) with RISC-V SoC on FireSim
Integrating NVIDIA Deep Learning Accelerator (NVDLA) with RISC-V SoC on FireSim Farzad Farshchi, Qijing Huang, Heechul Yun University of Kansas, University of California, Berkeley SiFive Internship Rocket
NVIDIA Think about Computing as Heterogeneous One Leo Liao, 1/29/2106, NTU
NVIDIA Think about Computing as Heterogeneous One Leo Liao, 1/29/2106, NTU GPGPU opens the door for co-design HPC, moreover middleware-support embedded system designs to harness the power of GPUaccelerated
Scaling Throughput Processors for Machine Intelligence
Scaling Throughput Processors for Machine Intelligence ScaledML Stanford 24-Mar-18 simon@graphcore.ai 1 MI The impact on humanity of harnessing machine intelligence will be greater than the impact of harnessing
HPVM: Heterogeneous Parallel Virtual Machine
HPVM: Heterogeneous Parallel Virtual Machine Maria Kotsifakou Department of Computer Science University of Illinois at Urbana-Champaign kotsifa2@illinois.edu Prakalp Srivastava Department of Computer Science
Deep learning prevalence. first neuroscience department. Spiking Neuron Operant conditioning First 1 Billion transistor processor
WELCOME TO Operant conditioning 1938 Spiking Neuron 1952 first neuroscience department 1964 Deep learning prevalence mid 2000s The Turing Machine 1936 Transistor 1947 First computer science department
Adaptable Computing The Future of FPGA Acceleration. Dan Gibbons, VP Software Development June 6, 2018
Adaptable Computing The Future of FPGA Acceleration Dan Gibbons, VP Software Development June 6, 2018 Adaptable Accelerated Computing Page 2 Three Big Trends The Evolution of Computing Trend to Heterogeneous
CSCI 402: Computer Architectures. Parallel Processors (2) Fengguang Song Department of Computer & Information Science IUPUI.
CSCI 402: Computer Architectures Parallel Processors (2) Fengguang Song Department of Computer & Information Science IUPUI 6.6 - End Today s Contents GPU Cluster and its network topology The Roofline performance
NumbaPro CUDA Python. Square matrix multiplication
NumbaPro Enables parallel programming in Python Support various entry points: Low-level (CUDA-C like) programming language High-level array oriented interface CUDA library bindings Also support multicore
OPERATIONALIZING MACHINE LEARNING USING GPU ACCELERATED, IN-DATABASE ANALYTICS
OPERATIONALIZING MACHINE LEARNING USING GPU ACCELERATED, IN-DATABASE ANALYTICS 1 Why GPUs? A Tale of Numbers 100x Performance Increase Infrastructure Cost Savings Performance 100x gains over traditional
Accelerating cublas/cudnn using Input-Aware Auto-Tuning
Accelerating cublas/cudnn using Input-Aware Auto-Tuning The ISAAC library Philippe Tillet Harvard University Introduction cublas does not always achieve peak performance: (M, N, K) 1 = (4096, 4096, 4096):
CafeGPI. Single-Sided Communication for Scalable Deep Learning
CafeGPI Single-Sided Communication for Scalable Deep Learning Janis Keuper itwm.fraunhofer.de/ml Competence Center High Performance Computing Fraunhofer ITWM, Kaiserslautern, Germany Deep Neural Networks
Red Fox: An Execution Environment for Relational Query Processing on GPUs
Red Fox: An Execution Environment for Relational Query Processing on GPUs Haicheng Wu 1, Gregory Diamos 2, Tim Sheard 3, Molham Aref 4, Sean Baxter 2, Michael Garland 2, Sudhakar Yalamanchili 1 1. Georgia
S WHAT THE PROFILER IS TELLING YOU: OPTIMIZING GPU KERNELS. Jakob Progsch, Mathias Wagner GTC 2018
S8630 - WHAT THE PROFILER IS TELLING YOU: OPTIMIZING GPU KERNELS Jakob Progsch, Mathias Wagner GTC 2018 1. Know your hardware BEFORE YOU START What are the target machines, how many nodes? Machine-specific
Characterization and Benchmarking of Deep Learning. Natalia Vassilieva, PhD Sr. Research Manager
Characterization and Benchmarking of Deep Learning Natalia Vassilieva, PhD Sr. Research Manager Deep learning applications Vision Speech Text Other Search & information extraction Security/Video surveillance
Introduction to CUDA CME343 / ME May James Balfour [ NVIDIA Research
Introduction to CUDA CME343 / ME339 18 May 2011 James Balfour [ jbalfour@nvidia.com] NVIDIA Research CUDA Programing system for machines with GPUs Programming Language Compilers Runtime Environments Drivers
Modern Processor Architectures. L25: Modern Compiler Design
Modern Processor Architectures L25: Modern Compiler Design The 1960s - 1970s Instructions took multiple cycles Only one instruction in flight at once Optimisation meant minimising the number of instructions
Lecture 12: Model Serving. CSE599W: Spring 2018
Lecture 12: Model Serving CSE599W: Spring 2018 Deep Learning Applications That drink will get you to 2800 calories for today I last saw your keys in the store room Remind Tom of the party You re on page
Copperhead: Data Parallel Python Bryan Catanzaro, NVIDIA Research 1/19
Copperhead: Data Parallel Python Bryan Catanzaro, NVIDIA Research 1/19 Motivation Not everyone wants to write C++ or Fortran How can we broaden the use of parallel computing? All of us want higher programmer