Unsupervised and Semi- Supervised Learning. Machine Learning, Fall 2010

Size: px

Start display at page:

Download "Unsupervised and Semi- Supervised Learning. Machine Learning, Fall 2010"

Noah Rafe Webb
5 years ago
Views:

1 Unsupervised and Semi- Supervised Learning Machine Learning, Fall

2 Announcement!"#$%&'($")*+,,-$")$").$-"/)0$).%'12'0+)*34$$5)-")6$&720+%)*3-+"3+) 8+'%)9:;)6*)#'3250<)'",=+%)04+,+)'"1)$04+%)>2+,($",?) I"(*%",#B&*5*D,*E,*D$'&#'E.*%F",,BJ* I"'E*&,*'&/C%%C,6%*F,//CK..%*B,,G*L,$J* I"'E*%",#B&*5*.M<.FE*L$,/*D$'&#'E.*%F",,BJ* N,O*&,*5*F",,%.*'*<$,D$'/J** N,O*&,*5*'<<B(J**!!"#$%&'()*+,-./0.$*1)*2343*! 56*7857*9,,/*4422*! 2

3 Administrativia This week continuing on unsupervised learning Some more of a different flavor clustering Semi-supervised learning Intersections between ensembles, active, un-sup and semisup learning May start Reinforcement Learning Reading Optional Chapter 17 of Manning, Raghavan, Schuetze Information Retrieval Book: IR-book/pdf/17hier.pdf 3

4 Hierarchical Clustering As opposed to k-means, which produces a flat clustering, here we produce a hierarchy of clusters From Ch17 of IR Book NYSE closing averages Hog prices tumble Oil prices slip Ag trade reform. Chrysler / Latin America Japanese prime minister / Mexico Fed holds interest rates steady Fed to keep interest rates steady Fed keeps interest rates steady Fed keeps interest rates steady Mexican markets British FTSE index War hero Colin Powell War hero Colin Powell Lloyd s CEO questioned Lloyd s chief / U.S. grilling Ohio Blue Cross Lawsuit against tobacco companies suits against tobacco firms Indiana tobacco lawsuit Viag stays positive Most active stocks CompuServe reports loss Sprint / Internet access service Planet Hollywood Trocadero: tripling of revenues Back!to!school spending is up German unions split Chains may raise prices Clinton signs law 4

5 Bottom-Up Clustering Initially each instance is in its own cluster Clusters are continually merged Will discuss the HAC Algorithm (Hierarchical Agglomerative Clustering) 5

6 HAC Input: { X 1, X 2,... Xn }, real-number vectors Initialize clusters: each cluster Iterate: X i becomes its own Find two most similar clusters c i and c j Replace c i and c j with c i c j 6

7 Another Look at the Dendrogram NYSE closing averages Hog prices tumble Oil prices slip Ag trade reform. Chrysler / Latin America Japanese prime minister / Mexico Fed holds interest rates steady Fed to keep interest rates steady Fed keeps interest rates steady Fed keeps interest rates steady Mexican markets British FTSE index War hero Colin Powell War hero Colin Powell Lloyd s CEO questioned Lloyd s chief / U.S. grilling Ohio Blue Cross Lawsuit against tobacco companies suits against tobacco firms Indiana tobacco lawsuit Viag stays positive Most active stocks CompuServe reports loss Sprint / Internet access service Planet Hollywood Trocadero: tripling of revenues Back!to!school spending is up German unions split Chains may raise prices Clinton signs law 7

8 Computing Similarity Between Clusters Single-link: Similarity between two clusters c i and c j is computed as similarity between their most similar members Complete-link: Computed as similarity between most dissimilar members Group-average clustering: similarity of two clusters computed as the average similarity over all possible pairs of instances in the clusters 8

9 Top-Down Clustering Also called Divisive Start with a single cluster containing all instances Use a flat clustering algorithm as a subroutine to split clusters 9

10 Semi-Supervised Learning Lots of different algorithms e.g. EM We ll discuss a classic: co-training (Blum & Mitchell 98) Awarded 10-Year best paper award in

11 Co-Training Algorithm Input: Set of labeled instances L Set of unlabeled instances U The attributes of each instance can be split into two views X 1 and X i i 2 that satisfy the following requirements X i Each is sufficient for classification The views are independent 11

12 Co-Training Algorithm Create a pool U consisting of u randomly chosen examples from U Loop for k iterations: Use L to train h 1 only on features Use L to train h 2 only on features Use h 1 to label p positive and n negative examples from U Use h 2 to label p positive and n negative examples from U Add self-labeled examples to L Add 2p + 2n examples from U to U X i 1 X 2 i 12

13 Co-Training Prediction Given a new instance X i = X i 1, X2 i, P (Y = y X i ) P h1 (Y = y X 1 i )P h 2 (Y = y X 2 i ) 13

14 Example: Document classification View 1: Text on document View 2: Anchor text in hyperlinks to document Task: classify page as course webpage or not 14

15 Co-Testing Co-training-like idea for active learning Loop for k iterations Use L to train h 1 only on features X i 1 Use L to train h 2 only on features Treat h 1 and h 2 as a committee of size 2 and request labels of unlabeled instances on which they disagree X i 2 15

Introduction to Information Retrieval

Introduction to Information Retrieval http://www.informationretrieval.org Hierarchical Clustering Institute for Natural Language Processing, Universität Stuttgart 2007.06.19 Hierarchical Clustering 1 /