Tackling Big Data Using MATLAB

Transcription

1 Tackling Big Data Using MATLAB Alka Nair Application Engineer 2015 The MathWorks, Inc. 1

2 Building Machine Learning Models with Big Data Access Preprocess, Exploration & Model Development Scale up & Integrate with Production Systems 2

3 Case study: Predict Air Quality Factors Affecting Air Quality My Weather Page Temperature Pressure Relative Humidity Dew Point Wind speed Wind direction Ozone CO NO2 SO2 3

4 4

5 Building Machine Learning Models with Big Data Access Preprocess, Exploration & Model Development Scale up & Integrate with Production Systems 5

6 Challenges in Modeling and Deploying Big Data Applications Access Preprocess, Exploration & Model Development Scale up & Integrate with Production Systems Distributed Data Storage Different Data Sources & Types Preprocessing and Visualizing Big Data Parallelizing Jobs and Scaling up Computations to Cluster Enterprise level deployment Managing Different APIs for Data Sources and Data Formats Rewriting Algorithms to Use Big Data Platforms Parallelizing Code to Scale up to Use Cluster and Cloud Compute Overhead in Moving the Algorithm to Production 6

7 Wouldn t it be nice if you could: Easily access data however it is stored Prototype algorithms quickly using small data sets Scale up to big data sets running on large clusters Using the same intuitive MATLAB syntax you are used to 7

8 Building machine learning models with big data Access Preprocess, Exploration & Model Development Scale up & Integrate with Production Systems 8

9 Access and Manage Big Data Different Data Types Different Data Sources Different Applications Text Images Spreadsheet Custom File Formats Hadoop Distributed File System (HDFS) Amazon S3 Windows Azure Blob Storage Relational Database HDFS on Hortonworks or Cloudera MapReduce Image Segmentation Image Classification Denoising Images Predictive Maintenance Datastores 9

10 Datastore Single Machine Memory Single Machine Memory Process Cluster of Machines Memory Cluster of Machines Memory One or more files 10

11 Air Quality Data on Local Folder 11

12 Accessing and Processing different types of data TabularTextDatastore Text files containing column-oriented data, including CSV files Image Collection MDF Files ImageDatastore SpreadsheetDatastore MDFDatastore Custom Datastore Image files, including formats that are supported by imread such as JPEG and PNG Spreadsheet files with a supported Excel format such as.xlsx Datastore for collection of MDF files Datastore for custom or proprietary format 12

13 You have 1 TB of data you ve never seen before. How do you access this data? 13

14 Historical files are on HDFS and real time data are available through an API Temperature Pressure Relative Humidity Dew Point Wind Speed Wind Direction Ozone CO NO2 SO2 14

15 Access air quality data using datastore 15

16 Preview the data and adjust properties to best represent the data of interest 16

17 Access data from anywhere with minimal changes Local disk 17

18 Datastores enable big data workflows Deep Learning 18

19 Datastores enable big data workflows Predictive Maintenance 19

20 Datastores enable big data workflows Fleet Analytics 20

21 Datastores: Access Big Data with Minimal Changes Different Data Types Different Data Sources Different Applications Text Images Spreadsheet Custom File Formats Hadoop Distributed File System (HDFS) Amazon S3 Windows Azure Blob Storage Relational Database HDFS on Hortonworks or Cloudera MapReduce Image Segmentation Image Classification Denoising Images Predictive Maintenance 21

22 Building machine learning models with big data Access Preprocess, Exploration & Model Development Scale up & Integrate with Production Systems 22

23 You have 1TB of data you ve never seen before. How do you visualize and process the data? 23

24 Use tall arrays to work with the data like any MATLAB array 24

25 Introduction to Tall Arrays Tall Arrays for Big Data Visualization and Preprocessing Machine Learning for Big Data Using Tall Arrays 25

26 Tall arrays Single Machine Memory Data is in one or more files Files stacked vertically Typically tabular data Challenge Data doesn t fit into memory (even cluster memory) Takes a lot of time for even simple operations on data Cluster of Machines Memory 26

27 Tall arrays (new R2016b) Single Machine Memory tall array Process Single Machine Memory Create tall table from datastore ds = datastore('*.csv') tt = tall(ds) Datastore Operate on whole tall table just like ordinary table Cluster of Machines Memory summary(tt) max(tt.endtime tt.starttime) 27

28 tall arrays Single Machine Memory tall array Process Single Machine Memory With Parallel Computing Toolbox, process several chunks at once Process Single Machine Memory Can scale up to clusters with MATLAB Distributed Computing Server Cluster of Machines Memory Process Single Machine Memory Process Single Machine Memory 28

29 Use a Spark-enabled Hadoop cluster and MATLAB Support for many other platforms through reference architectures 29

30 It s easy to run MATLAB code on Spark + Hadoop Spark Connection Cluster Config for Spark Hadoop Access 30

31 MATLAB Documentation for 31

32 Summary for tall arrays Local disk, Shared folders, Databases Run on Compute Clusters or Spark + Hadoop (HDFS), for large scale analysis Process out-of-memory data on your Desktop to explore, analyze, gain insights and to develop analytics Use Parallel Computing Toolbox for increased performance MATLAB Distributed Computing Server, Spark+Hadoop Develop your code locally using Tall Arrays or MapReduce only once Use the same code to scale up to cluster 32

33 Create a tall array for each datastore ozone 33

34 Execution model makes operations more efficient on big data tt : tall array Deferred evaluation Commands are not executed right away Operations are added to a queue Execution triggers include: gather function summary function Machine learning models Plotting 34

35 Execution model makes operations more efficient on big data Unnecessary results are not computed 35

36 Introduction to Tall Arrays Tall Arrays for Big Data Visualization and Preprocessing Machine Learning for Big Data Using Tall Arrays 36

37 Explore Big Data with Tall Visualizations plot scatter binscatter histogram histogram2 ksdensity 37

38 Explore Big Data with Tall Visualizations 38

39 Get a summary of the data tt tall table 39

40 Use data types to best represent the data 40

41 Managing Big and Messy Time-stamped Data 41

42 Use the results of explorations to help make decisions - Synchronize to daily data - By location 42

43 Synchronize all data to daily times 43

44 Clean messy data using common preprocessing functions 44

45 Use familiar MATLAB functions on tall arrays Functions Supported with Tall Arrays 45

46 You don t need to leave MATLAB to monitor large jobs 46

47 Save preprocessed data 47

48 Introduction to Tall Arrays Tall Arrays for Big Data Visualization and Preprocessing Machine Learning for Big Data Using Tall Arrays 48

49 Predict air quality Air Quality Index Air Quality Label Regression Classification 49

50 How do you know which model to use? Try them all 50

51 Use apps for model exploration on a subset of data Air Quality Index Air Quality Label Regression Learner Classification Learner 51

52 Validate and Compare Machine Learning Models 52

53 Validate and Compare Machine Learning Models 53

54 Validate and Compare Machine Learning Models 54

55 Validate and Compare Machine Learning Models 55

56 Scale up with tall machine learning models Linear Regression (fitlm) Logistic & Generalized Linear Regression (fitglm) Discriminant Analysis Classification (fitcdiscr) K-means Clustering (kmeans) Principal Component Analysis (pca) Partition for Cross Validation (cvpartition) Linear Support Vector Machine (SVM) Classification (fitclinear) Naïve Bayes Classification (fitcnb) Random Forest Ensemble Classification (TreeBagger) Lasso Linear Regression (lasso) Linear Support Vector Machine (SVM) Regression (fitrlinear) Single Classification Decision Tree (fitctree) Linear SVM Classification with Random Kernel Expansion (fitckernel) Gaussian Kernel Regression (fitrkernel) 56

57 Training Machine Learning Model against Spark for Air Quality Classification 57

58 Train and validate with tall data for Air Quality Index Prediction 58

59 Select the most important features 59

60 Introduction to Tall Arrays Tall Arrays for Big Data Visualization and Preprocessing Machine Learning for Big Data Using Tall Arrays 61

61 Building machine learning models with big data Access Preprocess, Exploration & Model Development Scale up & Integrate with Production Systems 62

62 63

63 Predict air quality for given location Current Weather My Weather Page Your Weather Conditions Get weather conditions for your area. Location: MATLAB Runtime Temperature: 32F Humidity: 76% Wind: SSW 13 mph Use MATLAB model running on Spark in Python web framework 64

64 Integrate analytics with systems Embedded Hardware C, C++ HDL PLC GPU Enterprise Systems Standalone Application Excel Add-in Hadoop/ Spark C/C++ Java ++ Python.NET MATLAB Production Server MATLAB Runtime 65

65 Package and test MATLAB code 66

66 67

67 Package and test MATLAB code 68

68 Call MATLAB in production environment AirQual.ctf 69

69 MATLAB Production Server Server software Manages packaged MATLAB programs and worker pool MATLAB Runtime libraries Single server can use runtimes from different releases RESTful JSON interface Enterprise Application MPS Client Library Applications/ Database Servers RESTful JSON MATLAB Production Server Request Broker & Program Manager Lightweight client libraries C/C++,.NET, Python, and Java MATLAB Runtime 70

70 MATLAB for Modeling and Deploying Big Data Applications Access Preprocess, Exploration & Model Development Scale up & Integrate with Production Systems Distributed Data Storage Different Data Sources & Types Preprocessing and Visualizing Big Data Parallelizing Jobs and Scaling up Computations to Cluster Enterprise level deployment Easily Access Data however/wherever it is stored using Datastore Prototype and easily scale up algorithms to Big Data platforms using the familiar MATLAB Syntax with Tall Arrays Seamless integration with Enterprise level systems using MATLAB Production Server 71

71 How do you get started? Try Tall Array Based Processing on Your Own Set of Big Data Refer to the example mentioned below to get started: Other Resources mathworks.com/big-data mathworks.com/machine-learning ebook 72

72 MathWorks Training Offerings 73

73 Speaker Details LinkedIn: a/ Contact MathWorks India Products/Training Enquiry Booth Call: Share your experience with MATLAB & Simulink on Social Media Use #MATLABEXPO Share your session feedback: Please fill in your feedback for this session in the feedback form 74