MCDNN: An Approximation-Based Execution Framework for Deep Stream Processing Under Resource Constraints

Transcription

1 MCDNN: An Appoximation-Based Execution Famewok fo Deep Steam Pocessing Unde Resouce Constaints Haichen Shen with Seungyeop Han, Matthai Philipose, Shaad Agawal, Alec Wolman, Avind Kishnamuthy Univesity of Washington Micosoft Reseach 1

2 Weaable computing à moe data??? 2

3 When compute vision meets weaable That dink will get you to 2800 caloies fo today I last saw you keys in the stoe oom Remind Tom of the paty You e on page 263 of this book Consume Manufactuing Public Safety 3

4 Deep leaning makes vision wok But... Recognition Task face scene* object* Accuacy 97% 88% 92% Compute/fame (FLOPs) 1.00G 30.9G 39.3G (FLOPS) 1-30G G 40G-1.2T Do we have enough esouces to un deep leaning? * top-5 accuacy is shown in the table 4

5 Resouce usage fo continuous vision Omnivision OV mW Tega K1 GPU = 34pJ/OP Qualcomm SD810 LTE >800mW Atheos a/g 47nJ/b Amazon EC2 CPU c4.lage 2x400GFLOPS $0.1/h GPU g2.2xlage 2.3TFLOPS $0.65/h Image Pocesso Radio Cloud Wokload Budget Compute powe Deep leaning 30GFLOPs/fame, 10fps Device powe Cloud cost 30% of 10Wh fo 10h = 300mW $10 peson/yea 9GFLOPS 3.5GFLOPS (GPU) / 8GFLOPS (CPU) Huge gap between wokload and budget 5

6 Neual netwok m c m c c c c classes + scoes s f f img R img G img B m f (c) convolution (f) fully connect (m) max pool () elu (s) softmax 6

7 Neual netwok matix multiplications m c classes + scoes s x x x c c f m c f Low ank appoximation (Y. Kim, et al. 2016) c img R img G img B m f (c) convolution (f) fully connect (m) max pool () elu (s) softmax c m c img R img G img B m c f classes + scoes s f Achitectual changes (J. Ba, et al. 2014) x Matix spasification (S. Han, et al. 2015) 7

8 Managing the appox. / esouce tade-off Detailed chaacteization of the appoximation / esouce tadeoff fo many optimizations Two new optimizations fo steaming, multi-application settings New scheduling poblem, Appoximate Model Scheduling, with a heuistic solution 8

9 Outline Detailed chaacteization of the appoximation / esouce tadeoff fo many optimizations Two new optimizations fo steaming, multi-application settings New scheduling poblem, Appoximate Model Scheduling, with a heuistic solution 9

10 Memoy / accuacy tade-off 10 3 memoy (0%) bM 6Fene )afe affuafy (%) 10

11 Memoy / accuacy tade-off 10 3 memoy (0%) 10 2 Substantially educe memoy use with gadual accuacy loss bM 6Fene )afe affuafy (%) 11

12 Enegy / accuacy tade-off Exceed enegy budget when execute locally Ej((xeF) 6Fene((xeF) )afe((xef) Can execute locally unde enegy budget Always execute locally enejy (J) affuafy (%) Nvidia Jetson TK1 Compute enegy budget (2.3J) LTE xmit cost (0.9J) Wifi xmit cost (0.5J) enegy budget = total enegy / total time(10h) / equests pe second(1 eq/sec) 12

13 Outline Detailed chaacteization of the appoximation / esouce tadeoff fo many optimizations Two new optimizations fo steaming, multi-application settings Specialization Model shaing New scheduling poblem, Appoximate Model Scheduling, with a heuistic solution 13

14 Exploiting steam locality by specialization Standad deep neual netwok ecognizes 4000 people Most of videos ae dominated by less than 10 faces ove minutes Timeline Poduce moe compact models fo skewed classes 14

15 Specialization untime class class full model check fo othe full model class input compact model input (with skewed distibution) 15

16 Bette esouce/accuacy tade-off 16

17

18 Outline Detailed chaacteization of the appoximation / esouce tadeoff fo many optimizations Two new optimizations fo steaming, multi-application settings New scheduling poblem, Appoximate Model Scheduling, with a heuistic solution 18

19 Appoximate model scheduling Model Pool Task 1 model 1 (90%) model 2 (80%) Task 2 model 3 (80%) model 4 (70%) memoy enegy cost Mobile device Enegy 14 Memoy 10 Cloud Cost 14 Accuacy Goal: maximize the oveall accuacy 19

20 Appoximate model scheduling Model Pool Task 1 model 1 (90%) model 2 (80%) Task 2 model 3 (80%) model 4 (70%) memoy enegy cost Mobile device model 4 model 4 Enegy 14 Memoy 10 Cloud Accuacy model 1 model 1 m Packing poblem Requests: 1. task 1 2. task 2 à device, model 4 3. task 1 à cloud, model 1 4. task 1 à cloud, model 2 5. task 2 à device, model 4 Cost 14 à cloud, model 1 20

21 Appoximate model scheduling Model Pool Task 1 model 1 (90%) model 2 (80%) Task 2 model 3 (80%) model 4 (70%) memoy enegy cost Mobile device model 4 model 4 model 4 Enegy 14 Memoy 10 model 2 Cloud Accuacy model 1 model 1 Cost m2 Requests: 1. task 1 à cloud, model 1 2. task 2 à device, model 4 3. task Paging 1 à cloud, poblem model 1 4. task 1 à cloud, model 2 5. task 2 à device, model 4 6. task 1 21

22 Appoximate model scheduling Packing poblem: pick vesions that satisfy enegy/cost budgets " e $ x $& &, E, " c $ x $& &,, C (x $&, x $& 0,1, x $& 3 x $& = 0) Paging poblem: pick vesions that fit in memoy Goal: maximize the accuacy 1 t T, " max A : $;< s $ x $& S, ) " " a $(x $& + x $& & $ No known optimal online algoithms 22

23 Heuistic schedule Estimate futue esouce use and compute the budget fo each equest Account fo paging cost to educe oscillations Use inceasingly moe accuate vesions of moe heavily used models 23

24 Tace-diven evaluation Lose connectivity Run out of battey 24

25 MCDNN famewok input type model schema taining/validation data development time compile tained model catalog un time input device untime schedule data oute classes input classes cloud untime schedule data oute pofile device apps cloud

26 MCDNN famewok input type model schema taining/validation data development time specialization time compile specialize tained model catalog un time specialized models stats input device untime schedule data oute classes input classes cloud untime schedule data oute pofile device apps cloud

27 Conclusion MCDNN makes efficient tade-offs between esouce use and accuacy Fomulate the appoximate model scheduling poblem and devise a heuistic algoithm Design a geneic appoximation-based execution famewok fo continuous mobile vision Thank you! Questions? 27

28 Backup Slides 28

29 Cloud cost / accuacy tade-off OatenFy (Ps) bj(COouG G38) 2bj(COouG C38) 6Fene(COouG G38) 6Fene(COouG C38) )afe(cooug G38) )afe(cooug C38) affuuafy (%) cloud CPU latency budget $10/y, 1 AWS c4.lage cloud GPU latency budget $10/y, 1 AWS g2.2xlage cloud GPU latency budget $10/y, 1 AWS g2.2xlage latency budget = cost budget / cost pe hou / #equests 29

30 Model shaing face ID ace age gende model-fagment cache input intemediate values input oute 30

31 Dynamically-sized caching scheme 31