BIG DATA: Data Analytics with MATLAB Christophe POUILLOT Senior Consultant MathWorks

Transcription

1 BIG DATA: Data Analytics with MATLAB Christophe POUILLOT Senior Consultant MathWorks 2014 The MathWorks, Inc. 1

2 Definition of Big Data Data so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. from wikipedia Volume Velocity Variety Name Symbol Value gigaoctet Go 10 9 téraoctet To pétaoctet Po exaoctet Eo Social network: xxto/day zettaoctet Zo Worldwide: 2,8Zo/year (2012) yottaoctet Yo Data structured or not from different sources: Web/Text/Image mining 2

3 Data containers Collection of files Databases Huge single file 3

4 Big Data Analytics with MATLAB Memory and Data Access 64-bit processors Memory Mapped Variables Disk Variables Databases Datastores Analysis Machine learning Analysis domain Statistics Programming Constructs Streaming Block Processing Parallel-for loops GPU Arrays SPMD and Distributed Arrays MapReduce Platforms Desktop (Multicore, GPU) Clusters Cloud Computing (MDCS on EC2) Hadoop 4

5 Agenda Machine Learning Datastore/MapReduce Integration with Hadoop Databases Huge file Deployment Key takeaways 5

6 Overview Machine Learning Type of Learning Categories of Algorithms Unsupervised Learning Clustering Machine Learning Group and interpret data based only on input data Classification Supervised Learning Develop predictive model based on both input and output data Regression 6

7 Demo: Machine Learning 7

8 Agenda Machine Learning Datastore/MapReduce Integration with Hadoop Databases Huge file Deployment Key takeaways 8

9 DataStore datastore Import text files & collections of text files that don t fit into memory ds = datastore('file1.mat'); ds = datastore('*.csv'); ds = datastore('/shared/data_repository/'); ds = datastore('hdfs://myserver:7867/data/file1.txt'); ds = datastore({'/shared01/','/shared02/'}); while hasdata(ds) T = read(ds); end 9

10 Demo mapreduce Data Store Map Reduce 1503 UA LAX PS BUR DL BOS DL SFO US BWI UA PS DL DL US UA 2356 UA 1867 UA 1365 PS PS SEA PS 176 PS PS LAX PS UA SFO UA 1867 DL US PS SJC UA MIA DL EWR DL DFW US UA DL DL DL 914 US 359 US

11 Demo: datastore/mapreduce 11

12 Agenda Machine Learning Datastore/MapReduce Integration with Hadoop Databases Huge file Deployment Key takeaways 12

13 Integration with Hadoop Datastore HDFS MATLAB Node Data Map Reduce Node Data Map Reduce map.m reduce.m main.m Node Data Map Reduce 13

14 Agenda Machine Learning Datastore/MapReduce Integration with Hadoop Databases Huge file Deployment Key takeaways 14

15 Databases Relational database (ODBC/JDBC-compliant) DatabaseDatastore (DataBase Toolbox) conn = database.odbcconnection('mysql','username','pwd'); dbds = datastore(conn, 'select * from producttable'); NOSQL database MATLAB calls external functions: C/C++ shared libraries JAVA libraries.net libraries COM Objects (ActiveX ) Python libraries WSDL Web Service 15

16 Agenda Machine Learning Datastore/MapReduce Integration with Hadoop Databases Huge file Deployment Key takeaways 16

17 Huge flat files: Mapping memory within MATLAB Read/write variables in files, without loading into memory matfile: mat files m = matfile('myfile.mat'); z = m.x(85:94,85:94); % read from disk m.x(81:100,81:100) = magic(20); % write on disk memmapfile: any files m = memmapfile('records.dat','offset',9000, 'Format','int32'); z = m.data(85:94,85:94); % read from disk m.data(81:100,81:100) = magic(20); % write on disk 17

18 Agenda Machine Learning Datastore/MapReduce Integration with Hadoop Databases Huge file Deployment Key takeaways 18

19 Deployment: Compiling MATLAB to go everywhere.lib.dll.exe Java MATLAB Java Analytic 19

20 Agenda Machine Learning Datastore/MapReduce Integration with Hadoop Databases Huge file Deployment Key takeaways 20

21 Key takeaways MATLAB is the framework for BIG DATA analytics MathWorks services can help you: Training services Consulting services 21

22 Questions? 22