Building an Operating System for AI

Transcription

1 Building an Operating System for AI How Microservices and Serverless Computing Enable the Next Generation of Machine Intelligence Diego Oppenheimer, CEO

2 About Me Diego Oppenheimer - Founder and CEO - Algorithmia Product developer, entrepreneur, extensive background in all things data. Microsoft: PowerPivot, PowerBI, Excel and SQL Server. Founder of algorithmic trading startup BS/MS Carnegie Mellon University

3 Make state-of-the-art algorithms discoverable and accessible to everyone.

4 Algorithmia.com AI/ML scalable infrastructure on demand + marketplace Function-as-a-service for Machine & Deep Learning Discoverable, live inventory of AI Monetizable Composable Every developer on earth can make their app intelligent 4

5 There s an algorithm for that! 70K+ DEVELOPERS 5K+ ALGORITHMS

6 How do we do it? ~5,000 algorithms (60k w/ different versions) Each algorithm: 1 to 1,000 calls a second, fluctuates, no devops ~15ms overhead latency Any runtime, any architecture 6

7 Characteristics of AI Two distinct phases: training and inference Lots of processing power Heterogenous hardware (CPUs, GPUs, TPUs, etc.) Limited by compute rather than bandwidth Tensorflow is open source, scaling it is not. - Kenny Daniel

8 TRAINING OWNER: Data Scientists Long compute cycle Fixed load (Inelastic) Stateful Single user 8

9 TRAINING OWNER: Data Scientists Long compute cycle Fixed load (Inelastic) Stateful Single user Analogous to dev tool chain. Building and iterating over a model is similar to building an app. 9

10 Use Case Jian Yang made an app to recognize food SeeFood. Fully trained. Works on his machine. All rights reserved HBO

11 Use Case He deployed his trained model to a GPU-enabled server? GPU-enabled Server

12 Use Case The app is a hit! SeeFood Productivity All rights reserved HBO

13 Use Case and now his server is overloaded.???? xn GPU-enabled Server

14 We ll be talking about Microservices & Serverless Computing MICROSERVICES: the design of a system as independently deployable, loosely coupled services. SERVERLESS: the encapsulation, starting, and stopping of singular functions per request, with a just-in-time-compute model. ADVANTAGES Maintainability Scalability Rolling deployments ADVANTAGES Cost / Efficiency Concurrency built-in Speed of development Improved latency

15 TRAINING OWNER: Data Scientists Long compute cycle Fixed load (Inelastic) Stateful Single user INFERENCE OWNER: DevOps Short compute bursts Elastic Stateless Multiple users Analogous to dev tool chain. Building and iterating over a model is similar to building an app. 15

16 TRAINING OWNER: Data Scientists Long compute cycle Fixed load (Inelastic) Stateful Single user INFERENCE OWNER: DevOps Short compute bursts Elastic Stateless Multiple users Analogous to dev tool chain. Building and iterating over a model is similar to building an app. Analogous to an OS. Running concurrent models requires task scheduling. 16

17 TRAINING OWNER: Data Scientists Long compute cycle Fixed load (Inelastic) Stateful Single user INFERENCE OWNER: DevOps Short compute bursts Elastic Stateless Multiple users Metal or VM Containers 17

18 TRAINING OWNER: Data Scientists Long compute cycle Fixed load (Inelastic) Stateful Single user INFERENCE OWNER: DevOps Short compute bursts Elastic Stateless Multiple users Metal or VM Containers Kubernetes 18

19 TRAINING OWNER: Data Scientists Long compute cycle Fixed load (Inelastic) Stateful Single user INFERENCE OWNER: DevOps Short compute bursts Elastic Stateless Multiple users Metal or VM Containers Kubernetes 19

20 Why Microservices? + + = Elastic Scalable Software agnostic Hardware agnostic 20

21 Why Serverless? Cost / Efficiency Concurrency built-in Improved latency 21

22 Calls per Second GPU Server Instances Why Serverless - Cost Efficiency Jian Yang s SeeFood is most active during lunchtime Max calls/s Avg calls/s 12 AM 02 AM 04 AM 06 AM 08 AM 10 AM 12 PM 02 PM 04 PM 06 PM 08 PM 10 PM

23 Calls per Second GPU Server Instances Traditional Architecture - Design for Maximum 40 machines 24 hours. $648 * 40 = $25,920 per month Max calls/s Avg calls/s 12 AM 02 AM 04 AM 06 AM 08 AM 10 AM 12 PM 02 PM 04 PM 06 PM 08 PM 10 PM

24 Calls per Second GPU Server Instances Autoscale Architecture - Design for Local Maximum 19 machines 24 hours. $648 * 40 = $12,312 per month Max calls/s Avg calls/s 12 AM 02 AM 04 AM 06 AM 08 AM 10 AM 12 PM 02 PM 04 PM 06 PM 08 PM 10 PM

25 Calls per Second GPU Server Instances Serverless Architecture - Design for Minimum Avg. of 21 calls / sec, or equivalent of 6 machines. $648 * 6 = $3,888 per month Max calls/s Avg calls/s 12 AM 02 AM 04 AM 06 AM 08 AM 10 AM 12 PM 02 PM 04 PM 06 PM 08 PM 10 PM

26 Load Balancer Why Serverless - Concurrency??? GPU-enabled Servers

27 Why Serverless - Improved Latency Portability = Low Latency

28 + + ALSO: GPU Memory Management, Job Scheduling, Cloud Abstraction, etc. 28

29 An Operating System for AI

30 Shell & Services Kernel Discoverability, Authentication, Instrumentation, etc. Elastic Scale Runtime Abstraction Cloud Abstraction Prioritize and automatically optimize execution of concurrent short-lived jobs. Support any programming language or framework, including interoperability between mixed stacks. Provide portability to algorithms, including public clouds or private clouds. 30

31 Kernel: Elastic Scale Web Load Balancer Web Servers User API Load Balancer API Servers Cloud Region #1 Cloud Region #2 Worker xn Worker xn Docker(algorithm#1).. Docker(algorithm#n) Docker(algorithm#1).. Docker(algorithm#n) 31

32 Composability Composability is critical for AI workflows because of data processing pipelines and ensembles. Fruit or Veggie Classifier Fruit Classifier Veggie Classifier 32

33 Kernel: Elastic Scale + Intelligent Orchestration FoodClassifier CPU util, GPU util, Memory util, IO util CPU util, GPU util, Memory util, IO util FruitClassifier VeggieClassifier CPU util, GPU util, Memory util, IO util 33

34 Kernel: Elastic Scale + Intelligent Orchestration Knowing that: Algorithm A always calls Algorithm B Algorithm A consumes X CPU, X Memory, etc Algorithm B consumes X CPU, X Memory, etc CPU util, GPU util, Memory util, IO util FoodClassifier Therefore we can slot them in a way that: Reduce network latency Increase cluster utilization Build dependency graphs CPU util, GPU util, Memory util, IO util FruitClassifier VeggieClassifier CPU util, GPU util, Memory util, IO util 34

35 Kernel: Runtime Abstraction FoodClassifier FruitClassifier VeggieClassifier 35

36 Kernel: Cloud Abstraction - Storage # No storage abstraction s3 = boto3.client("s3") obj = s3.get_object(bucket="bucket-name", Key="records.csv") data = obj["body"].read() # With storage abstraction data = Algorithmia().client.file("blob://records.csv").get() s3://foo/bar blob://foo/bar hdfs://foo/bar dropbox://foo/bar etc. 36

37 Kernel: Cloud Abstraction Compute EC2 CE VM Nova Autoscaling Autoscaling Group Autoscaler Scale Set Heat Scaling Policy Load Balancing Elastic Load Balancer Load Balancer Load Balancer LBaaS Remote Storage Elastic Block Store Persistent Disk File Storage Block Storage Partial Source: Sam Ghods, KubeConf

38 Summary - What makes an OS for AI? Stack-agnostic Composable Self-optimizing Auto-scaling Monitorable Discoverability 38

39 Unix Multi-tenancy, Composability GUI (Win/Mac) Accessibility Punched Cards 1970s DOS Hardware Abstraction ios/android Built-in App Store (Discoverability)

40 AI is here Punched Cards 1970s ios/android Built-in App Store (Discoverability)

41 FREE STUFF: Signup with code: NVIDIAGTC18 for $50 on us. Thank you! Diego Oppenheimer

42 more slides

43 The New Moats Source: Jerry Chen, Greylock Ventures 43

44 Punched Cards 1970s GitHub and Heroku Today

45 Kernel: Cloud Abstraction - Storage # init client = Algorithmia.client() # init client = Algorithmia.client() # get data (S3) s3 = boto3.client("s3") obj = s3.get_object(bucket="bucket-name", Key="records.csv") data = obj["body"].read() # get data (anything) data = client.file("blob://records.csv").get() # remove seasonality data = client.algo("ts/removeseasonality").pipe(data).result # remove seasonality data = client.algo("ts/removeseasonality").pipe(data).result # forecast time series data = client.algo("ts/forecastlstm").pipe(data).result # forecast time series data = client.algo("ts/forecastlstm").pipe(data).result 45

46 01 # MY_ALGORITHM.py client = Algorithmia.client() 04 data = client.file("blob://records.csv").get() # remove seasonality 07 data = client.algo("ts/removeseasonality").pipe(data).result # forecast time series 10 data = client.algo("ts/forecastlstm").pipe(data).result 46

47 Kernel: Elastic Scale + Intelligent Orchestration MyAlgorithm # MY_ALGORITHM.py CPU util, GPU util, Memory util, IO util client = Algorithmia.client() data = client.file("blob://records.csv").get() # remove seasonality data = client.algo("ts/removeseasonality").pipe(data).result # forecast time series data = client.algo("ts/forecastlstm").pipe(data).result RemoveSeasonality CPU util, GPU util, Memory util, IO util CPU util, GPU util, Memory util, IO util ForecastLSTM 47

48 Kernel: Elastic Scale + Intelligent Orchestration Knowing that: Algorithm A always calls Algorithm B Algorithm A consumes X CPU, X Memory, etc Algorithm B consumes X CPU, X Memory, etc CPU util, GPU util, Memory util, IO util MyAlgorithm Therefore we can slot them in a way that: Reduce network latency Increase cluster utilization Build dependency graphs RemoveSeasonality CPU util, GPU util, Memory util, IO util CPU util, GPU util, Memory util, IO util ForecastLSTM 48

49 Kernel: Runtime Abstraction MyAlgorithm # MY_ALGORITHM.py client = Algorithmia.client() data = client.file("blob://records.csv").get() # remove seasonality data = client.algo("ts/removeseasonality").pipe(data).result # forecast time series data = client.algo("ts/forecastlstm").pipe(data).result RemoveSeasonality ForecastLSTM 49

50 Challenges Machine learning CPU/GPU/Specialized hardware Multiple frameworks, languages, dependencies Called from different devices/architectures Snowflake environments Unique cloud hardware and services Uncharted territory Not a lot of literature, errors messages sometimes cryptic (can t just stackoverflow) 50