CSC 261/461 Database Systems Lecture 26. Spring 2017 MW 3:25 pm 4:40 pm January 18 May 3 Dewey 1101

Transcription

1 CSC 261/461 Database Systems Lecture 26 Spring 2017 MW 3:25 pm 4:40 pm January 18 May 3 Dewey 1101

2 Announcements Poster Presentation on May 03 (During our usual lecture time) Mandatory for all Graduate students Class participation points for All attendees Also if you want a diverse audience, try the poster session organized by CIRC Registration deadline: April 28 th Poster requirements: High resolution images Proper citation Send the final version to Kamrul if you have made any changes Subject: Poster Team # xx

3 References ml

4 DATA MINING

5 Data Mining Finding interesting trends and patterns in large datasets to guide decisions about future activities Identify these patterns with minimal user input Gives data analyst useful and unexpected insight

6 Data Mining (cont.) Subarea of exploratory data analysis Closely related to knowledge discovery and machine learning Algorithms should be scalable

7 Knowledge Discovery Process Knowledge discovery and data mining (KDD) process has four steps: Data Selection: The target subset of data and the attributes of interest Data Cleaning: Noise and outliers are removed. Denormalization is often used Data Mining: Apply datamining algorithms to extract interesting patterns. Evaluation: The patterns are presented to end users in an understandable form Visualization

8 Counting Co-occurrences Market basket a collection of items purchased by a customer in a single customer transaction A common goal is to identify items that are purchased together. This information can be used to : improve the layout of goods in a store Layout of catalog pages Recommendations

9 Association rule learning Association rule learning is a rule-based machine learning method for discovering interesting relations between variables in large databases. In order to select interesting rules from the set of all possible rules, constraints on various measures of significance and interest are used. We will consider only frequent itemsets

10 Frequent Itemsets Itemset A set of items Support of an itemset The fraction of transactions in the database that contain all the items in the itemset Minsup (minimal support): We are interested in itemsets where support > minsup

11 A Priori Property Every subset of a frequent itemset is also a frequent itemset. First identify frequent itemsets with just one item In subsequent iteration, frequent itemsets in the previous iteration are extended with another item to create larger candidate set. Guarantees that the optimization is correct. Terminates when no new frequent itemsets are identified in an iteration.

12 Support, Confidence, and lift Support is an indication of how frequently the itemset appears in the dataset. Confidence is an indication of how often the rule has been found to be true. Lift is the ratio of the observed support to that expected if X and Y were

13 Frequent Pattern Mining - RDD-based API spark.mllib provides a parallel implementation of FPgrowth a popular algorithm to mining frequent itemsets.

14 Acknowledgement Spark slides and material: Jonathan Carroll-Nellenback (CIRC, UofR) Learning Spark Lightning-Fast Big Data Analysis By Holden Karau, Andy Konwinski, Patrick Wendell, Matei Zaharia Publisher: O'Reilly Media