Big Data Its All Around You

Transcription

1

2 Big Data Its All Around You Brian Macdonald Oracle Enterprise Architect

3 Big Data: Its All Around You Introduction What is Big Data What is Data Science Big Data Technologies Q&A 3

4 My Road to Big Data 5

5 My Road to Big Data Math and Computer Science Information Systems Analyze Data Implementation Sales 6

6 Who are Sales Engineers Engineers Mathematicians Programmers Business People Philosophers Biologist Any one who can solve problems and explain solutions! 7 Copyright 2012, Oracle and/or its affiliates. All rights reserved.

7 What is Big Data? 8

8 10 Copyright 2012, Oracle and/or its affiliates. All rights reserved.

9 Analyze Social Media Data 383+ Million Twitter accounts (255m+ tweeting) 1,280+ Million Facebook subscribers 200+ Million Instagram users 1 Billion YouTube users, 4 Billion Views/Day Billion Mobile Web users Over 6 million OnStar subscribers 11 Copyright 2012, Oracle and/or its affiliates. All rights reserved.

10 How Much Information was Produced in 2011? What do you call 1.8 Trillion Gigabytes? 1.8 Zettabytes* 1800 Exabytes 1.8 Million Petabytes 1.8 Billion Terabytes 1.8 Trillion Gigabytes Billion iphones (32GB model) Trillion Songs (6000/iPhone) 1,123,715,750 Years (3.5 Minutes/Song) *1.8 Zettabytes of information will be created and replicated in 2011 IDC Copyright 2012, Oracle and/or its affiliates. All rights reserved.

11 Twitter Demo 13

12 Twitter Exercise Search Twitter and analyze Sentiment Enter search terms and see if you get the results you would expect ID: bmacdona Pwd: bmacdona 14

13 How is Big Data Used Siri Google Maps Medicine Personalized Medicine Genomics Law Business Retail Finance Telecom Governments The Common Good Social Graphs Atrocity Watch Environmental 15

14 Public Data NYC Chicago Maps - Crime Pittsburgh - Data Sets - Interactive Maps 16

15 Some people are more certain of everything than I am of anything Robert Rubin In an Uncertain World 17

16 What Makes Big Data BIG DATA? SOCIAL BLOG SMART METER Volume Velocity Variety NACH Exa Business Development Team / NACH_Exa_Biz_Dev_Ca@oracle.com 18

17 The Internet of Things or the Sensor Revolution Source: The Economist 19

18 Big Data is Complex Structured Data Semi-Structured Data Unstructured Data <bib> <book year="1995"> <title> Database Systems </title> <author> <lastname> Date </lastname> </author> <publisher> Addison-Wesley </publisher> </book> <book year="1998"> <title> Foundation for Object/Relational Databases </title> <author> <lastname> Date </lastname> </author> <author> <lastname> Darwen </lastname> </author> <ISBN> <number> </number > </ISBN> </book> </bib> Oracle-SunGard Big Data Copyright Event 2014 Oracle and/or its affiliates. All rights reserved. 20

19 Visualization Look through Examples Important way to communicate information How to represent millions of data points is an art Many tools exist to help generate vizualization

20 What is Data Science? 22

21 When experts express uncertainty about their opinions, people find them more compelling. Harvard Business Review

22 Data Scientists The Most Awesome Job of the Future Create questions - Hypothesis Prepare Data Analyze large and small volumes of data Understand what the data means Visualize Data Tell Stories Create Innovative ways to use it 24 Copyright 2012, Oracle and/or its affiliates. All rights reserved.

23 What do you need to know? How to Think These are the things you re taught that you think you will never need As much as possible Math Science History Languages Art Be curious 25 Copyright 2012, Oracle and/or its affiliates. All rights reserved.

24 You must be able to think and solve problems! And observe Count the Number of F s in the following sentence. FINISHED FILES ARE THE RE- SULT OF YEARS OF SCIENTIF- IC STUDY COMBINED WITH THE EXPERIENCE OF YEARS. 26 Copyright 2012, Oracle and/or its affiliates. All rights reserved.

25 Leave your assumptions at the door! FINISHED FILES ARE THE RE- SULT OF YEARS OF SCIENTIF- IC STUDY COMBINED WITH THE EXPERIENCE OF YEARS. 27 Copyright 2012, Oracle and/or its affiliates. All rights reserved.

26 Lets Analyze some Data Correlation The degree to which two or more attributes or measurements on the same group of elements show a tendency to vary together Positive when values increase together Negative when values decrease together 28 Copyright 2012, Oracle and/or its affiliates. All rights reserved.

27 Does Ice Cream Consumption Cause Drowning? Obviously not Correlation does not imply Causation One may cause the other, but correlation just defines how they vary. There may be other reasons. i.e. Hot temperatures Be very cautious with Causation There are tests to determine causation 29 Copyright 2012, Oracle and/or its affiliates. All rights reserved.

28 How do I know if variables are correlated Understanding R and R 2 R = Correlation Coefficient Values between -1 & 1 Positive Correlation > 0 - As one variable increases, the other increases Perfect Correlation = 1 Negative Correlation < 0 - As one variable increases, the other decreases Perfect Negative Correlation = -1 0 = No correlation Can be shown with a trend line 30 Copyright 2012, Oracle and/or its affiliates. All rights reserved.

29 How do I know if variables are correlated Understanding R and R 2 R 2 = Coefficient of Determination Tells how likely one variable predicts the other variable Values between 0 & 1 If r 2 = 0.850, 85% of the total variation in y can be explained by the linear relationship between x and y R 2 is more commonly used 31 Copyright 2012, Oracle and/or its affiliates. All rights reserved.

30 Your Turn to Be a Data Scientist What does data about Pittsburgh tell you? - Find variables that have high Coefficient of Determination (R 2 ) Explore Data Create a hypothesis Who can find the best largest value of R 2? Experiment with multiple columns Create new variables Answer the Following Questions What variables are correlated? Are they Positively or Negatively correlated? What is R 2? Do you think they are causal? What does graph tell you? What additional data would you like to have? What would you do based on this information? 32 Copyright 2012, Oracle and/or its affiliates. All rights reserved.

31 Data Science can be used just for fun 33 Copyright 2012, Oracle and/or its affiliates. All rights reserved.

32 Predictive Analytics Where things get really interesting We looked at describing data What if you can predict what will happen Lots of algorithms exists to do this You just need the data And an interesting question 34 Copyright 2012, Oracle and/or its affiliates. All rights reserved.

33 Predictive Analytics Data Mining Algorithms Function Algorithms Applicability Classification Logistic Regression (GLM) Decision Trees Naïve Bayes Support Vector Machines (SVM) Classical statistical technique Popular / Rules / transparency Embedded app Wide / narrow data / text Regression Linear Regression (GLM) Support Vector Machine (SVM) Classical statistical technique Wide / narrow data / text Anomaly Detection One Class SVM Unknown fraud cases or anomalies Attribute Importance Association Rules A1 A2 A3 A4 A5 A6 A7 Minimum Description Length (MDL) Principal Components Analysis (PCA) Apriori Attribute reduction, Reduce data noise Market basket analysis / Next Best Offer Clustering Hierarchical k-means Hierarchical O-Cluster Expectation-Maximization Clustering (EM) Product grouping / Text mining Gene and protein analysis Feature Extraction F1 F2 F3 F4 Nonnegative Matrix Factorization (NMF) Singular Value Decomposition (SVD) Text analysis / Feature reduction

34 Data Mining Provides Better Information, Valuable Insights and Predictions Cell Phone Churners vs. Loyal Customers Segment #3: IF CUST_MO > 7 AND INCOME < $175K, THEN Prediction = Cell Phone Churner, Confidence = 83%, Support = 6/39 Insight & Prediction Segment #1: IF CUST_MO > 14 AND INCOME < $90K, THEN Prediction = Cell Phone Churner, Confidence = 100%, Support = 8/39 Customer Months Source: Inspired from Data Mining Techniques: For Marketing, Sales, and Customer Relationship Management by Michael J. A. Berry, Gordon S. Linoff

35 Decision Trees Decision Trees Classification Prediction Customer profiling Owns foreign car = yes Status Age >55 <55 Foreign car = no 45 > Age Problem: Find customers likely to buy a new car and their profiles <=45 Income Gender Num children <100K >100K F M <=4 >4 Car = 0 Car = 1 Car = 0 Car = 1 Car = 0 Car = 1 IF (Age >55) AND Owns foreign car=no AND (Income >100K ) THEN P(Buy Car=1) =.77 Support = 250

36 Big Data Technologies 38

37 Many Technologies Relational Databases Oracle, Microsoft SQL Server, MySQL, DB2, Teradata, Netezza, Postgres NoSQL Databases Oracle NoSQL Database, Cassandra, MongoDB, RIAK Hadoop R Visualization Tools Tableau, Spotfire, Excel Open Source is becoming more prominent 39

38 R Statistical Programming Language Open source language and environment Used for statistical computing and graphics Strength in easily producing publication-quality graphs Highly extensible

39 What to do with Big Data? Like DNA Data won t fit in spreadsheet Would take a long time to do the math Human Genome 3.2 billion base pairs (ATCG) Split problem into smaller pieces 41 Copyright 2012, Oracle and/or its affiliates. All rights reserved.

40 Hadoop The Apache Framework Hadoop for distributed software processing library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. Hadoop is designed to scale up from single Large Data Sets servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-availability, the library Clusters itself is of designed Computers to detect and handle failures at the application layer, so delivering a highly-available service on top of a cluster of computers, each of which may be prone to failures. Simple Computing Models Highly Available Service

41 Map Reduce

42 A MapReduce Analogy

43 A MapReduce Analogy Going From Estimates To Actuals

44 A MapReduce Analogy Sharing The Load Of Mapping Out The Raw Numbers Photo Credit: Lindsey Bauman/Hutchinson News

45 A MapReduce Analogy Reporting Back The Results Photo Credit: Lindsey Bauman/Hutchinson News Photo Credit: Renee Saff

46 A MapReduce Example Putting The Analogy Into Practice

47 A MapReduce Example The Input The data arrives into the system.

48 A MapReduce Example Splitting The Input Into Chunks The data is moved into the HDFS system, divided into blocks, each of which are copied multiple times for redundancy.

49 A MapReduce Example Mapping The Chunks The Mapper picks up a chunk for processing. The MR Framework ensures only one mapper will be assigned to a given chunk

50 A MapReduce Example Mapping The Chunks In this case, the Mapper emits a color with the number of times it was found.

51 A MapReduce Example A Shuffle Sort The Shuffler can do a rough sort of like items (optional)

52 A MapReduce Example Reducing The Emissions The Reducer combines the Mapper s output

53 A MapReduce Example The Output The job returns a count of colors found in the input

54 What would you like to predict? Let me know and I can sponsor a project. 56

55 Questions? 57