Core ML in Depth. System Frameworks #WWDC17. Krishna Sridhar, Core ML Zach Nation, Core ML

Transcription

1 System Frameworks #WWDC17 Core ML in Depth Krishna Sridhar, Core ML Zach Nation, Core ML 2017 Apple Inc. All rights reserved. Redistribution or public display not permitted without written permission from Apple.

2 The easiest way to integrate machine learning models into your app.

3 The easiest way to integrate machine learning models into your app.

4 The easiest way to integrate machine learning models into your app.

5 The easiest way to integrate machine learning models into your app.

6 The easiest way to integrate machine learning models into your app.

7 Think of models as code

8 Development Flow

9 Development Flow Xcode

10 Development Flow Xcode

11 Development Flow Xcode Your App

12 Development Flow Xcode Your App

13 Rose Confidence 95%

14 Rose Confidence 95% let model = FlowerClassifier() if let prediction = try? model.prediction(image: image) { return prediction.flowertype }

15 Rose Confidence 95% let model = FlowerClassifier() if let prediction = try? model.prediction(image: image) { return prediction.flowertype }

16 This Session Use Cases

17 This Session Use Cases Hardware Optimized Core ML Accelerate CPU MPS GPU

18 This Session Use Cases Hardware Optimized Obtaining Models Core ML Accelerate CPU MPS GPU

19 This Session Use Cases Hardware Optimized Obtaining Models Core ML Accelerate CPU MPS GPU

20

21

22

23

24

25 Sentiment Analysis Object Detection Personalization Style Transfer Music Tagging Gesture Recognition Summarization

26 Sentiment Analysis Object Detection Personalization Style Transfer Music Tagging Gesture Recognition Summarization

27 Sentiment Analysis Object Detection Personalization Style Transfer Music Tagging Gesture Recognition Summarization

28 Sentiment Analysis Object Detection Personalization Style Transfer Music Tagging Gesture Recognition Summarization

29 Sentiment Analysis Object Detection Personalization Style Transfer Music Tagging Gesture Recognition Summarization Generalized Linear Models Tree Ensembles Support Vector Machines Feedforward Neural Networks Convolution Neural Networks Recurrent Neural Networks

30 Feedforward Neural Networks Convolution Neural Networks Recurrent Neural Networks

31 Pipeline Vectorize Neural Network Map to String Rose

32 Sentiment Analysis Object Detection Personalization Style Transfer Music Tagging Gesture Recognition Summarization Focus on code, not models!

33 Models as Functions Numeric Categories Images Rose Arrays Dictionaries

34 Numeric and Categories developer.apple.com/machine-learning Numeric Categories Images Arrays Dictionaries Double, Int64 String, Int64 CVPixelBuffer MLMultiArray [String : Double], [Int64 : Double]

35 Images Numeric Categories Images Arrays Dictionaries Double, Int64 String, Int64 CVPixelBuffer MLMultiArray [String : Double], [Int64 : Double]

36 Multi-Dimensional Arrays Numeric Categories Images Arrays Dictionaries Double, Int64 String, Int64 CVPixelBuffer MLMultiArray [String : Double], [Int64 : Double]

37 Dictionaries Numeric Categories Images Arrays Dictionaries Double, Int64 String, Int64 CVPixelBuffer MLMultiArray [String : Double], [Int64 : Double]

38 Working with Text

39

40 CoreML is awesome. I love using it

41

42 Today s lunch was terrible and disappointing

43 Sentiment Analysis Application

44 Sentiment Analysis Application

45 Sentiment Analysis Application!!!

46 Sentiment Analysis Application!!!

47 Using Other Formats Word Counts { } "Core" : 1, "ML" : 1, "is" : 1, "awesome" : 1, "I" : 1, "love" : 1, "using" : 1, "it" : 1,

48 Using Other Formats Word Counts { } "Core" : 1, "ML" : 1, "is" : 1, "awesome" : 1, "I" : 1, "love" : 1, "using" : 1, "it" : 1,

49 Using Other Formats Word Counts { } "Core" : 1, "ML" : 1, "is" : 1, "awesome" : 1, "I" : 1, "love" : 1, "using" : 1, "it" : 1,!

50 Use NLP to Process Text Word Counts { } "Core" : 1, "ML" : 1, "is" : 1, "awesome" : 1, "I" : 1, "love" : 1, "using" : 1, "it" : 1,

51 Sentiment Analysis Application { } "Core" : 1, "ML" : 1, "is" : 1, "awesome" : 1, "I" : 1, "love" : 1, "using" : 1, "it" : 1,! Use NLP (NSLinguisticTagger) Use Core ML

52

53 Processing Text { } "Core" : 1, "ML" : 1, "is" : 1, "awesome" : 1, "I" : 1, "love" : 1, "using" : 1, "it" : 1, func tokenizeandcountwords(rawtxt: String) -> [String : Double] { // Tokenize using NSLinguisticTagger... } // Count occurrences of each token

54 Making Predictions { } "Core" : 1, "ML" : 1, "is" : 1, "awesome" : 1, "I" : 1, "love" : 1, "using" : 1, "it" : 1,! let model = SentimentAnalysis() if let prediction = try? model.prediction(wordcounts: counts) { print("sentiment: \(prediction.sentiment)") }

55 Text Gotchas! Most models don t work directly on raw text

56

57 Predictive Keyboard I m not sure Oliver will eat oysters, Love but he will

58 Predictive Keyboard I m not sure Oliver will eat oysters, Love but he will Task - Next word prediction

59

60 Language Models

61 Shakespeare Keyboard

62 Shakespeare Keyboard Current Word

63 Shakespeare Keyboard Current Word Next Word Choices State

64 Shakespeare Keyboard Next Word Current Word Next Word Choices State

65 Shakespeare Keyboard Next Word Current Word Next Word Choices State

66 Shakespeare Keyboard Next Word Current Word Next Word Choices State State

67 Sample Code if let output = try? model.prediction(input: input) { } // Send the best 3 words to the user displaytoppredictions(output.nextwordprob) // Update state for next prediction input.state = output.state input.currentword = getnextwordfromuser()

68 Sample Code if let output = try? model.prediction(input: input) { } // Send the best 3 words to the user displaytoppredictions(output.nextwordprob) // Update state for next prediction input.state = output.state input.currentword = getnextwordfromuser()

69 Sample Code if let output = try? model.prediction(input: input) { } // Send the best 3 words to the user displaytoppredictions(output.nextwordprob) // Update state for next prediction input.state = output.state input.currentword = getnextwordfromuser()

70 Sample Code if let output = try? model.prediction(input: input) { } // Send the best 3 words to the user displaytoppredictions(output.nextwordprob) // Update state for next prediction input.state = output.state input.currentword = getnextwordfromuser()

71 Come to the Labs Thursday 11:00 3:30 Friday 1:30 4:00

72 This Session Use Cases Hardware Optimized Obtaining Models Core ML Accelerate CPU MPS GPU

73

74

75 Built on Performance Primitives Your app Vision NLP Core ML ML Performance Primitives Accelerate MPS Hardware CPU GPU

76 Optimized for Hardware Runs on GPU MPS

77 Optimized for Hardware Runs on GPU Runs on CPU MPS Accelerate

78 Optimized for Hardware

79 Optimized for Hardware Runs on Core ML

80 Context Switching Runs on GPU Runs on CPU Dog playing in a soccer field

81 This Session Use Cases Hardware Optimized Obtaining Models Core ML Accelerate CPU MPS GPU

82 Deploying Core ML Models Xcode Your App

83 Deploying Core ML Models Xcode Your App Where do models come from?

84 Example Models Task specific Core ML models developer.apple.com/machine-learning

85 Tap into ML Community Popular ML libraries Many models Thriving communities

86 Tap into ML Community Popular ML libraries Many models Thriving communities

87 Core ML Tools

88 Core ML Tools Open Source

89 Conversion Workflow Xcode Your App

90 Conversion Workflow Model Source Xcode Your App e.g. Caffe

91 Conversion Workflow Model Source Xcode Your App e.g. Caffe

92 Getting Started ## Download and install python package > pip install coremltools

93 What Is coremltools? Converters Convert from other formats Core ML Bindings Converter Library Core ML Specification

94 What Is coremltools? Converters Convert from other formats Core ML Bindings Converter Library Build your own converter Core ML Specification

95 What Is coremltools? Converters Convert from other formats Core ML Bindings Converter Library Build your own converter Core ML Specification Write your own models

96 Compatible and Extensible Converters Compatible Core ML Bindings Converter Library Core ML Specification

97 Compatible and Extensible Converters Core ML Bindings Converter Library Core ML Specification Extensible

98 Core ML Model developer.apple.com/machine-learning Single document Converters Encapsulates Core ML Bindings Converter Library Functional description (inputs Trained parameters outputs) Core ML Specification Public format

99 Xcode Model View

100 Xcode Model View

101 Core ML Converters Model Source In Xcode Converters Core ML Bindings Converter Library Convert Core ML Specification e.g. Caffe Unified APIs to convert models from various formats to Core ML

102 Flower Classification in Caffe 74.caffemodel.prototxt

103 Flower Classification in Caffe 74 Rose.caffemodel.prototxt labels.txt

104 Flower Classification in Caffe 74 Rose.caffemodel.prototxt labels.txt Needed for conversion to Core ML format

105 Demo

106 Demo Summary import coremltools caffe_model = ('flowers.caffemodel','flowers.prototxt') model = coremltools.converters.caffe.convert( caffe_model, image_input_names = 'data', class_labels = 'labels.txt') model.save('flowerclassifier.mlmodel')

107 Demo Summary import coremltools keras_model = 'flowers.h5' model = coremltools.converters.keras.convert( keras_model, image_input_names = 'data', class_labels = 'labels.txt') model.save('flowerclassifier.mlmodel')

108 Supported Packages Neural Networks Pipelines Tree Ensembles Linear Models Support Vector Machines

109 Obtaining Models Summary Convert developer.apple.com/machine-learning

110 Summary

111 Summary Easy integration of ML models

112 Summary Easy integration of ML models Rich datatype support

113 Summary Easy integration of ML models Rich datatype support Hardware optimized

114 Summary Easy integration of ML models Rich datatype support Hardware optimized Compatible with popular formats

115 More Information

116 Related Sessions Introducing Core ML WWDC 2017 Vision Framework: Building on Core ML WWDC 2017 Natural Language Processing and your Apps WWDC 2017 Accelerate and Sparse Solvers Grand Ballroom A Thursday 10:00AM Using Metal 2 for Compute Grand Ballroom A Thursday 4:10PM

117 Labs Core ML and Natural Language Processing Lab Technology Lab D Thu 11:00AM-3:30PM Core ML and Natural Language Processing Lab Technology Lab D Fri 1:50PM-4:00PM

118