Data Analytics with MATLAB

Transcription

1 Data Analytics with MATLAB Tackling the Challenges of Big Data Adrienne James, PhD MathWorks 7 th October The MathWorks, Inc. 1

2 Big Data in Industry ENERGY Asset Optimization FINANCE Market Risk, Regulatory AUTO Fleet Data Analysis AERO Maintenance, reliability Medical Devices Patient Outcomes 2

3 PROCESSING OPTIONS MATLAB RESTful interface to Cluster MATLAB Hadoop Streaming NoSQL connector (e.g. mongo) MATLAB / Java App accessing Cluster MATLAB Map-Reduce Components 3

4 Key takeaways New functions for analysing data that does not fit in memory on your desktop datastore mapreduce & that can scale for use with Hadoop Additional techniques for predictive modelling with large data Work with large data in memory on a cluster (spmd) Deploy predictive models Bring MATLAB analytics to the Web Share analytics with a wider community of users 4

5 How big is big? What characterises big data? Wikipedia Any collection of data sets so large and complex that it becomes difficult to process using traditional data processing applications. Volume : amount of data Velocity : speed at which data is generated or needs to be analysed Variety : range of data types/data sources 5

6 Considerations: Large Data Analytics Data Characteristics 1. Size & type of data? 2. Where is your data? 3. What hardware do you have access to? 4. Analysis Characteristics? 6

7 Example: Airline Delay Analysis Data BTS/RITA Airline On-Time Statistics 123.5M records, 29 fields Analysis Tasks Calculate delay patterns Visualize summaries Estimate & evaluate predictive models 7

8 Considerations: Large Data Analytics Airline Data Characteristics 1. Size & type of data? CSV Data 22 files 12GB 8

9 Considerations: Large Data Analytics Data Characteristics 1. Size & type of data? 2. Where is my data? Small subset available locally Entire data set stored elsewhere 9

10 Big Data Analysis with MATLAB start on the desktop Prototype Access Explore Share/Deploy Scale Work on your desktop Start simple Basic statistics Explore data 10

11 Demo: Exploring departure delays using datastore Explore approaches pre- & post- Start with a small subset What happens as the data size grows?. until eventually it does not fit in memory on your desktop machine datastore 11

12 Access & explore bigger data on the desktop more easily datastore Easily specify data set Single text file (or collection of text files) Database (using Database Toolbox) Preview data structure and format Customise data to import using column names Incrementally read subsets of the data airdata = datastore('*.csv'); airdata.selectedvariables = {'Distance', 'ArrDelay }; data = read(airdata); 12

13 Images datastore extends Data Access Landscape imread, SystemObjects streaming data ImageAdapter readtable pre- Binary files Databases Text files.mat files Import Tool fread, database post- load textscan memmapfile database. ODBCConnection textscan, readtable +programming matfile API datastore SMALL Increasing Data Size 13

14 Considerations: Large Data Analytics Data Characteristics 1. Size & type of data? 2. Where is your data? Small subset available locally Entire data set stored elsewhere 3. What hardware do you have access to? 4. Analysis Characteristics Initially, simple statistics & data exploration 14

15 Big Data Analysis with MATLAB Prototype Access Explore Share/Deploy Scale Start locally and then.. Scale to a cluster 15

16 What is? A Big Data Platform Datastore HDFS Node Data Map Reduce Node Data Map Reduce Node Data Map Reduce 16

17 A bit of audience participation mapreduce. 17

18 Introducing the mapreduce programming framework Example: Input files Intermediate files (local disk) Output files National popularity contest Steve 11% Emily 58% David 31% Newspaper pages For each page how many times do Steve, Emily and David get mentioned? Total mentions 18

19 mapreduce concept group counts Input files Intermediate files (local disk) Output files Map Reduce 19

20 Demo: Exploring mapreduce 20

21 Explore and Analyze Data on Hadoop Datastore HDFS MATLAB Distributed Computing Server Node Data Map Reduce Node Data Map Reduce MATLAB MapReduce Code Node Data Map Reduce Hadoop ds = datastore('hdfs://myserver:7867/data/file1.txt'); 21

22 Considerations: Large Data Analytics Data Characteristics 1. Size & type of data? 2. Where is your data? 3. What hardware do you have access to? Cluster 4. Analysis Characteristics Explore predictive modelling 22

23 Big Data Analysis with MATLAB Prototype Access Explore Share/Deploy Scale Options for more involved algorithms. may require all data in memory multiple iterations Scale to a cluster 23

24 Data Analytics Landscape COMPLEX More programming effort required iterative all data needed in memory at once Algorithm complexity easily partitioned; independent tasks SIMPLE vectorisation Built-in numerical & statistical algorithms gpuarray parfor spmd distributed arrays mapreduce SMALL Increasing Data Size 24

25 Instructions Reduced Data Working with more complex algorithms with data in memory on a cluster MDCS Client 25

26 Demo: Predictive Modelling Logistic Regression & Neural Networks 10 busiest airport origins & 7 largest airline carriers Explore & compare prediction quality of two models to predict flights delayed for more than 20 minutes Randomly partition data into test and training sets (cvpartition) Model #1: Logistic Regression Model #2: Neural Network Predictor Variables: DayOfWeek,Origin,Airline,DepTime,Distance 26

27 Single Program, Multiple Data Lab 1 Lab 2 Lab 3 Lab 4 >> mycode >> mycode >> mycode >> mycode 27

28 Single Program, Multiple Data Cluster Parallel Pool Lab 1 a = rand; Client spmd a = rand; end Lab 2 Lab 3 a = rand; a = rand; Lab 4 a = rand; 28

29 Explore Big Data Prototype Access Explore Share/Deploy Scale Subset data by filtering or variable selection and gain insight with visualization 29

30 Highlights: Airline Delay Analysis Start small Scale up Quick prototyping on large data Interactive exploration Interspersed visualizations Predictive modelling with large data 30

31 Deploy Prototype Access Explore Share/Deploy Scale Hadoop Desktop Enterprise Web 31

32 Web Analytics: Analysis of traffic around Paris 32

33 Predictive Data Analytics Load Demand Forecasting 33

34 Demo Station: 34

35 MATLAB on Hadoop Two modes of operation Execute mapreduce on Hadoop from your MATLAB desktop using MATLAB Distributed Computing Server Extends your desktop environment for use with Hadoop Execute algorithms within Hadoop MapReduce on data stored in HDFS Create standalone applications or libraries for deploying to production instances of Hadoop Locked down package for use in production environments Integration of MATLAB analytics with operational systems 35

36 Key takeaways New functions for analysing data that does not fit in memory on your desktop datastore mapreduce & that can scale for use with Hadoop Additional techniques for predictive modelling with large data Work with large data in memory on a cluster (spmd) Deploy predictive models Bring MATLAB analytics to the Web Share analytics with a wider community of users 36

37 New Big Data Capabilities in MATLAB Memory and Data Access 64-bit processors Memory Mapped Variables Disk Variables Databases Datastores Programming Constructs Streaming Block Processing Parallel-for loops GPU Arrays SPMD and Distributed Arrays MapReduce Platforms Desktop (Multicore, GPU) Clusters Cloud Computing (MDCS on EC2) Hadoop 37

38 Additional Resources MathWorks Web Site Big Data With MATAB: MapReduce & Hadoop: Machine Learning with MATLAB: A selection of user stories LiquidNet: Lean Data Analysis: The Awesome Data Dexterity of MATLAB Desktop Ruuki Metals: Steel Manufacturing Process Analytics CEESAR: Data Processing Framework Supporting Large Scale Driving Data Analysis Daimler AG: Analyzing Test Data from a Worldwide Fleet of Fuel Cell Vehicles 38

39 Thank You 39