Mining Association Rules in Large Databases

Transcription

1 Mining Association Rules in Large Databases Vladimir Estivill-Castro School of Computing and Information Technology With contributions fromj. Han 1 Association Rule Mining A typical example is market basket analysis analyzing customer buying habits Vladimir Estivill-Castro 2

2 Application of Rules Cross-marketing and attached mailing applications. customized and designed junk-mail catalog design, add-on sales. store layout customer segmentation based on buying patterns. Vladimir Estivill-Castro 3 Illustration Customer 1 Customer 2 Customer 3 Product A Product B Product C Product D X X X X X X X X ``90% of purchases that have bread and butter also include milk. It is a rule of the form A B (90%). A is the antecedent. B is the consequent. There is a confidence value associated to the rule. Vladimir Estivill-Castro 4

3 Usage of mining for rules Find all rules that have Diet Coke as the consequent. Help plan what should be done to boast the sales of Diet Coke. Find all the rules that have bagels in the antecedent. Determine what products may be impacted if the store discontinues selling bagels. Vladimir Estivill-Castro 5 Usage of mining rules Find all rules that have sausage in the antecedent and mustard in the consequent what items should be sold with sausages to make highly likely that mustard will also be sold. Find all rules relating items located in shelves A and B understand if the distance affects the sales of items from both shelves. Vladimir Estivill-Castro 6

4 Support and Confidence A rule of the form A B (support=20%, confidence=60%) In the database there are 20% of transactions where A B is true In the database, 60% of the transactions where A is true, B is also true. Rules are considered interesting if they satisfy a minimum support and a minimum confidence thresholds (called strong rules) Vladimir Estivill-Castro 7 Support and Confidence A B TRANSACTIONS D Support= Size of A B / size of D Confidence= Size of A B / size of A Vladimir Estivill-Castro 8

5 Basic concepts Names for the columns are items T={i 1,i 2, i n } A transaction T is a subset of T T T and has an identifier TID A transaction T contains a set A if and only if A T In an association A B, A and B are subsets of T and A B =φ Vladimir Estivill-Castro 9 Revising support and confidence In an association A B, support is the percentage of transactions T in D where A B T In an association A B, confidence is the percentage of transactions T with B T with respect to those containing A (A T) Support (A B) =P(A B) Confidence(A B)=P(B A) Care of the use of union Vladimir Estivill-Castro 10

6 Itemsets A set of identifiers of columns a subset of the columns If it has cardinality k it is called a k-itemset The number of transactions that containing an itemset is called support, occurrence, frequency, count If an itemset has support more than the minimum support it is called frequent (large) itemset The set of frequent k-itemsets is L k Vladimir Estivill-Castro 11 General strategy of the algorithms Find all frequent itemsets The algorithm complexity is this step Generate strong association rules from the frequent itemsets. Vladimir Estivill-Castro 12

7 Mining Single-Dimensional Boolean Association Rules The Apriori Algorithm Level-wise search Find L 1, then L 2, then L 3,, L k The Apriori property: If A is a frequent itemset, all its subsets are frequent itemsets If A is not a frequent itemset, all its supersets are NOT frequent Why? No more transactions if we require more items Vladimir Estivill-Castro 13 Apriori Algorithm Each level requires a pass through the data A sets of candidate items sets are verified to be frequent The L k frequent itemsets are used to make a small as possible list of candidates (explore L (k+1) ) but that include L (k+1) Vladimir Estivill-Castro 14

8 Exploring L (k+1) for L k Join Step Store itemsets in lexicographical order Compute C k =L k join L k All sets of size k+1 whose first k-1 item appear in two k-frequent sets A and B the k-th itemis in A/B The prune step the (k+1)-th item is in B/A Remove from C k any set who has a subset of size k that is not in L k Vladimir Estivill-Castro 15 Generating Association Rules from Frequent Itemsets Confidence (A B) P(B A) support_count(a B)/support_count(A) For each frequent itemset A, generate all non-empty subsets of A For every non-empty subset B of A, output the rule B A/B, if support_count(a)/support_count(b) min_conf Vladimir Estivill-Castro 16

9 Some necessary procedures subset(c k,t) Given a transaction t, find the set C k of all k- itemsets supported by t Used when scanning the database to obtain the id of itemsets whose count needs to be incremented has_infrequent_subset(c,l k ) given a k+1 itemset c, check if it has a subset of size k that is not in L k used to prune candidates resulting from L k join L k Vladimir Estivill-Castro 17 Improving the efficiency of Apriori Hash-based techniques When scanning the database, place subsets of size k supported by transactions in a hash table If a bucket does not have minimum support, none of the sets in that bucket can have minimum support So they can be pruned ahead of time Useful when going from 1-itemsets to 2-itemsets Transaction reduction remove transaction that do not have any frequent k-itemsets from the next round Partition Data Sampling Balance passes vs going p-levels of (rather than one) Vladimir Estivill-Castro 18

10 Mining Frequent Itemsets without Candidate Generation Problems with Apriori May generate many candidate sets If the frequent items are 10 4, the candidate 2- itemsets are (10 4 choose 2), that is 10 8 The FP-growth technique (frequent pattern growth The first step is the same as Apriori On database scan to find the frequent 1- itemsets Then sorted in order of frequency count Vladimir Estivill-Castro 19 Construction of the FP-tree It is a trie (classical data structure for word frequencies also called d-dimensional-tree) where the words are the transactions transactions are treated as words with the items in the lexicographic order given by the frequent 1-itemset counts (non-frequent items removed from consideration in the transaction) See Figure 6.8 in Chapter 6 of Han & Kamber This is the second (and last!) pass on the data Vladimir Estivill-Castro 20

11 The FP-tree Additionally, a linked list finds each frequent 1-itemset appearance in all branches of the trie. Vladimir Estivill-Castro 21 Extraction of the frequent itemsets Start from the least frequent 1-item set. Find its occurrences in the FP-tree The paths from the root of the FP-tree to the location of this letter in the tree are is called the conditional pattern base. Because the counts of letters are stored in the tree, the pattern base can be filtered for being frequent (minimum support). Paths that satisfy min_support are used to generate the frequent item sets Repeat for the next frequent 1-item set Vladimir Estivill-Castro 22

12 Mining Multilevel Association Rules from Transaction Databases Use a hierarchy to create new predicates Computer software desktop labtop educational accounting IBM Dell Microsoft PeopleSoft Vladimir Estivill-Castro 23 Change the granularity of the rules Rules of very specific products may have very little support setting the support value may give too many rules rather than having itemsets with {Microsoft Encarta, Dell 1100 labtop} have itemsets with more general categories {educational software, labtop} Vladimir Estivill-Castro 24

13 Approaches to Mining Multilevel Association rules Vary how the user specifies the minimum support in the hierarchy Using uniform minimum support Level 1 min_sup=5% Computer=[support=10%] Level21 min_sup=5% Laptopcomputer[support=6%] Desk top computer[support=4%] Vladimir Estivill-Castro 25 Reduced minimum support at lower levels Attempts to obtain desired effect of restricting to many rules of to specific item classes Level-by-level Level 1 min_sup=5% Computer=[support=10%] Level 2 min_sup=3% Laptopcomputer[support=6%] Desk top computer[support=4%] Vladimir Estivill-Castro 26

14 Reduced minimum support at lower levels Attempts to obtain desired effect of restricting to many rules of to specific item classes Level-cross-filtering by single item nodes are inspected only if parents have the minimum support at this level Level 1 min_sup=12% Computer=[support=10%] Level 2 min_sup=3% Laptopcomputer[support=6%] Desk top computer[support=4%] Vladimir Estivill-Castro 27 Reduced minimum support at lower levels Attempts to obtain desired effect of restricting to many rules of to specific item classes Level-cross filtering by k-itemset Level 1 min_sup=5% Computer [support=7%] And printer Level 2 min_sup=2% Laptopcomputer[support=1%] B/W-printer Desk top computer[support=2%] B/W-printer Desk top computer[support=1%] Laptopcomputer[support=3%] Color printer Color printer Vladimir Estivill-Castro 28

15 Other variants in multi-level level association rules Cross-level association rules (an overall support to the more specific level) Checking for redundant rules a more specific rules is eliminated in the presence of a more general rules with similar information (confidence/support) desk top computer => b/w printer (s=8%, c=70%) IBM desk top computer => b/w printer (s=2%, co= 72%) if 25% of desk top computers are IBM from the first rule we can infer the second one Vladimir Estivill-Castro 29 Multi-dimensional Association Rules Apply other tables, predicates or dimensions (of the data cube) buys(x, IBM desktop computer ) buys(x, b/w printer ) The only predicate involved is BUYS The next rule is multi-dimensional age(x, ) occupation (X, student ) buys(x, IBM desktop computer ) Vladimir Estivill-Castro 30

16 Multi-dimensional Rules need to handle Numerical attributes Called mining quantitative Association rules (as opposed to the previous mining of Boolean rules) Static Discretization Bins of equal length Bins with equal frequency Distance-based clustering Vladimir Estivill-Castro 31 ARCS - Association Rule Clustering System Pairs of quantitative attributes, and detect frequent 2-item sets AGE(X, ) INCOME(X, ) INCOME AGE Vladimir Estivill-Castro 32

17 ARCS - Association Rule Clustering System Cluster rules Age(X,35) Income(X,31-40) buys(x, TV ) Age(X,34) Income(X,31-40) buys(x, TV ) INCOME Age(X,35) Income(X,41-50) buys(x, TV ) Age(X,34) Income(X,41-50) buys(x, TV ) AGE Age(X, ) Income(X,31-50) buys(x, TV ) Vladimir Estivill-Castro 33 From Association Mining to Correlation Analysis Strong rules are not necessarily interesting Assume 10,000 transactions, 6,000 purchased computer games, 7,500 videos and 4,000 videos with computer games buys(x,computer game) buys(x,video)[s=40%,c=66%] Not interesting, 75% buy computer games, anyway In fact, computer games affects negatively the sales of videos Vladimir Estivill-Castro 34

18 Correlation between to Events Corr (A,B) = =Prob(A and B)/[Prob(A)Prob(B)] If A and B are itemsets we write A B for the event A and B Prob(A and B)/[Prob(A)Prob(B)] = Prob(B A)/P(B) The correlation is 1 if the events are independent It is also called the lift of the association Vladimir Estivill-Castro 35 Interesting rules Those with large lift The correlation departs from 1 Correlation rules require that the items in the itemset be correlated Tested by the Χ 2 statistic If an itemsets is correlated, then all supersets are correlated Vladimir Estivill-Castro 36

19 Constrain-Based Association Mining Knowledge type-constraints specify the type of knowledge ie association rules Data constraints specify task-relevant data Dimension/level constraints dimensions in the data (tables) or levels in concept hierarchies Interestingness constraints thresholds for statistical measures (support,confidence, etc) Rule constraints The specific form of the rules Vladimir Estivill-Castro 37 Metarule-Guided Mining of Association Rules Structure of the rule P 1 (X,Y) P 2 (X,W) buys(x, educational software ) P 1 and P 2 are predicate variables A rules complies with the Metarule AGE(X, ) Income(X, 30K- 41K ) buys(x, educational software ) Vladimir Estivill-Castro 38

20 Guiding the Mining Process P 1 P 2... P l Q 1 Q 2... Q r Letting p=l+r We need to find all frequent p-predicate sets Lp We must also have the support count for the l- predicate subsets of L p in order to compute the confidence of the rules derived from L p Note that in multidimensional association rules, itemsets are replaced by predicate sets One strategy is to materialize an n-data cube with n p with n attributes considered for instantiation for the predicate variables. Then scan p-cuboids and l-cuboids for the support counts Vladimir Estivill-Castro 39 Mining guided by Association Rule Constraints Dimensions sales(customer_name,item_name,transaction_id) lives(customer_name, region, city) item(item_name,category,price) transaction(transaction_id,day,month,year) QUERY (1) mine associations as (2) lives(c,_, Vancouver ) sales + (C,?{I},{S}) sales + (C,?{J},{T}) (3) from sales (4) where S.year=1999 and T.year=1999 and I.category=J.category (5) group by C,I.category (6) having sum(i.price) 100 and min(j.price) 500 (7) with support threshold=1% (8) with confidence threshold = 50% Vladimir Estivill-Castro 40

21 How the types of constraints affect the mining Anti-monotone constraints eliminates all supersets of an itemset sum(i.price) 500 If an k-itemset I already sums more than 500, none of its supersets will sum less than 500 Monotone constraints saves testing supersets of an itemset on a constraint sum(i.price) 100 If a k-itemset I already sums more than 100, all of its supersers will some more than 100 Succinct constraints they are computed without scanning the data sum(j.price) 500 Compute all subsets of items that sum more than 500 from the table of prices, not transactions Vladimir Estivill-Castro 41 Summary A popular area of application of association analysis is marketing and business management form market basket analysis Association rules consists of first finding frequent itemsets and then form strong rules Association rules have several categories Boolean vs quantitative single-dimensional vs multi-dimensional single level vs multi-level correlation analysis/maximal frequent patterns etc Vladimir Estivill-Castro 42

22 Summary Apriori algorithm finds all frequent k-items sets using the Apriori property and building candidates for frequent (k+1)-item sets from frequent k-itemsets Alternative methods have been suggested, like the Frequent Pattern Growth algorithm that treats transactions as words Or the Inverted Matrix Algorithm (El-Hajj & Zaiane) Vladimir Estivill-Castro 43