Business Analytics and Big Data: the process and the tools

Transcription

1 Business Analytics and Big Data: the process and the tools Mehmet Gençer Assoc.Prof., Organization Studies & Computer Engineering

2 How big?

3 1st Character of Big Data Volume A typical PC might have had 10 gigabytes of storage in Today, Facebook ingests 500 terabytes of new data every day. Boeing 737 will generate 240 terabytes of flight data during a single flight across the US. The smart phones, the data they create and consume; sensors embedded into everyday objects will soon result in billions of new, constantly-updated data feeds containing environmental, location, and other information, including video.

4 2nd Character of Big Data Velocity Clickstreams and ad impressions capture user behavior at millions of events per second high-frequency stock trading algorithms reflect market changes within microseconds machine to machine processes exchange data between billions of devices infrastructure and sensors generate massive log data in real-time on-line gaming systems support millions of concurrent users, each producing multiple inputs per second.

5 3rd Character of Big Data Variety Big Data isn't just numbers, dates, and strings. Big Data is also geospatial data, 3D data, audio and video, and unstructured text, including log files and social media. Traditional database systems were designed to address smaller volumes of structured data, fewer updates or a predictable, consistent data structure. Big Data analysis includes different types of data

6 The Structure of Big Data Structured Most traditional data sources Semi-structured Many sources of big data Unstructured Video data, audio data 6

7 A Application Of Big Data analytics Smarter Healthcare Homeland Security Traffic Control Manufacturing Multichannel sales Telecom Trading Analytics Search Quality

8 Types of tools used in Big-Data Where processing is hosted? Distributed Servers / Cloud (e.g. Amazon EC2) Where data is stored? Distributed Storage (e.g. Amazon S3) What is the programming model? Distributed Processing (e.g. MapReduce) How data is stored & indexed? High-performance schema-free databases (e.g. MongoDB) What operations are performed on data? Analytic / Semantic Processing Where is the processing performed? Web service (e.g. sense.io), desktop (R, SPSS, WEKA)

9 Leading Technology Vendors Example Vendors Commonality IBM Netezza EMC Greenplum Oracle Exadata Open source tools: R SPSS WEKA Sense.io sage MPP architectures Commodity Hardware RDBMS based Full SQL compliance

10 Statistics 101

11 Random Sample and Statistics Population: is used to refer to the set or universe of all entities under study. However, looking at the entire population may not be feasible, or may be too expensive. Instead, we draw a random sample from the population, and compute appropriate statistics from the sample, that give estimates of the corresponding population parameters of interest.

12 Statistic Let Si denote the random variable (e.g. age) corresponding to data point xi (e.g. a person in the sample), then a statistic θ is a function θ : (S1, S2,, Sn) R. If we use the value of a statistic to estimate a population parameter, this value is called a point estimate of the parameter, and the statistic is called as an estimator of the parameter.

13 Empirical Cumulative Distribution Function Where Inverse Cumulative Distribution Function

14 Example

15 Measures of Central Tendency (Mean) Population Mean: Sample Mean (Unbiased, not robust):

16 Measures of Central Tendency (Median) Population Median: or Sample Median:

17 Example

18 Measures of Dispersion (Range) Range: Sample Range: Not robust, sensitive to extreme values

19 Measures of Dispersion (Inter-Quartile Range) Inter-Quartile Range (IQR): Sample IQR: More robust

20 Measures of Dispersion (Variance and Standard Deviation) Variance: Standard Deviation:

21 Measures of Dispersion (Variance and Standard Deviation) Variance: Standard Deviation:

22 Univariate Normal Distribution

23 Multivariate Normal Distribution

24 OLAP (online analytical processing) and Data Mining

25 Warehouse Architecture Client Client Query & Analysis Metadata Warehouse Integration Source Source Source 25

26 Star Schemas A star schema is a common organization for data at a warehouse. It consists of: 1. Fact table : a very large accumulation of facts such as sales. Often insert-only. 2. Dimension tables : smaller, generally static information about the entities involved in the facts. 26

27 Terms Fact table Dimension tables product prodid name Measures price sale orderid date custid prodid storeid qty amt customer custid name address city store storeid city 27

28 Star product prodid p2 name price bolt 10 nut 5 sale oderid date o100 1/7/97 o102 2/7/ /8/97 customer custid custid name joe fred sally prodid p2 storeid c1 c1 c3 address 10 main 12 main 80 willow store storeid c1 c2 c3 qty amt city sfo sfo la 28 city nyc sfo la

29 Cube Fact table view: sale prodid p2 p2 storeid c1 c1 c3 c2 Multi-dimensional cube: amt p2 c c2 8 dimensions = 2 29 c3 50

30 3-D Cube Fact table view: sale prodid p2 p2 storeid c1 c1 c3 c2 c1 c2 Multi-dimensional cube: date amt day 2 day 1 p2 c1 12 p2 11 c1 44 c2 4 c2 8 dimensions = 3 30 c3 c3 50

31 ROLAP vs. MOLAP ROLAP: Relational On-Line Analytical Processing MOLAP: Multi-Dimensional On-Line Analytical Processing 31

32 Aggregates Add up amounts for day 1 In SQL: SELECT sum(amt) FROM SALE WHERE date = 1 sale prodid p2 p2 storeid c1 c1 c3 c2 c1 c2 date amt

33 Aggregates Add up amounts for day 1 In SQL: SELECT sum(amt) FROM SALE WHERE date = 1 sale prodid p2 p2 storeid c1 c1 c3 c2 c1 c2 date amt

34 Another Example Add up amounts by day, product In SQL: SELECT date, sum(amt) FROM SALE GROUP BY date, prodid sale prodid p2 p2 storeid c1 c1 c3 c2 c1 c2 date amt sale prodid p2 rollup drill-down 34 date amt

35 Aggregates Add up amounts for day 1 In SQL: SELECT sum(amt) FROM SALE WHERE date = 1 sale prodid p2 p2 storeid c1 c1 c3 c2 c1 c2 date amt

36 What is Data Mining? Discovery of useful, possibly unexpected, patterns in data Non-trivial extraction of implicit, previously unknown and potentially useful information from data Exploration & analysis, by automatic or semi-automatic means, of large quantities of data in order to discover meaningful patterns

37 Data Mining Tasks Classification [Predictive] Clustering [Descriptive] Association Rule Discovery [Descriptive] Sequential Pattern Discovery [Descriptive] Regression [Predictive] Deviation Detection [Predictive] Collaborative Filter [Predictive]

38 Regression Estimating the relationship between a dependent variable (Y) and one or more independent variables (predictors, X), represented as parameters (B) Linear: Y=BX+e Non-linear: no general form

39 Classification: Definition Given a collection of records (training set ) Each record contains a set of attributes, one of the attributes is the class. Find a model for class attribute as a function of the values of other attributes. Goal: previously unseen records should be assigned a class as accurately as possible. A test set is used to determine the accuracy of the model. Usually, the given data set is divided into training and test sets, with training set used to build the model and test set used to validate it.

40 Classification: Decision Trees Example: Conducted survey to see what customers were interested in new model car Want to select customers for advertising campaign training set 40

41 Classification: KNN K-nearest neighbours: key idea is that similar observations belong to similar classes. Thus, one simply has to look for the class designators of a certain number of the nearest neighbors and weigh their class numbers to assign a class number to the unknown. 41

42 Clustering income education age 42

43 K-Means Clustering 43

44 Association Rule Mining tio c sa n tra id sales records: n er m s to u c id ts c du ht o pr oug b market-basket data Trend: Products p5, p8 often bough together Trend: Customer 12 likes product p9 44

45 Association Rule Discovery Marketing and Sales Promotion: Let the rule discovered be {Bagels, } --> {Potato Chips} Potato Chips as consequent => Can be used to determine what should be done to boost its sales. Bagels in the antecedent => can be used to see which products would be affected if the store discontinues selling bagels. Bagels in antecedent and Potato chips in consequent => Can be used to see what products should be sold with Bagels to promote sale of Potato chips! Supermarket shelf management. Inventory Managemnt

46 Collaborative Filtering Goal: predict what movies/books/ a person may be interested in, on the basis of Past preferences of the person Other people with similar past preferences The preferences of such people for a new movie/book/ One approach based on repeated clustering Cluster people on the basis of preferences for movies Then cluster movies on the basis of being liked by the same clusters of people Again cluster people based on their preferences for (the newly created clusters of) movies Repeat above till equilibrium Above problem is an instance of collaborative filtering, where users collaborate in the task of filtering information to find information of interest 46

47 Other Types of Mining Text mining: application of data mining to textual documents cluster Web pages to find related pages cluster pages a user has visited to organize their visit history classify Web pages automatically into a Web directory Mine consumer or public opinion in Twitter messages Graph Mining: Deal with graph data Social Network Analysis 47

48 Data Streams What are Data Streams? Continuous streams Huge, Fast, and Changing Why Data Streams? The arriving speed of streams and the huge amount of data are beyond our capability to store them. Real-time processing Window Models Landscape window (Entire Data Stream) Sliding Window Damped Window Mining Data Stream 48

49 Model quality and comparison A statistical model has limited explanatory and/or predictive power One needs to use measures to compare alternative models and choose the best one Example measures: Regression analysis: Rsquare measure Classification: k-value of parameters Classification: Precision-recall

50 Know the alternative models "How [the] translation from subject-matter problem to statistical model is done is often the most critical part of an analysis" Sir David Cox, British Statistician unsupervised Hierarchical Clustering Association rules Text mining/topic analysis Recommender systems (Collaborative filter) Regression Decision tree supervised Parametric (results are easy to interpret) K-nearest neighbours Neural networks nonparametric

51 Hands on statistics with R Resources: See Appendix A sample session