Document clustering and its application to web search

Transcription

1 Document clustering and its application to web search Claudio Carpineto Fondazione Ugo Bordoni Research work in collaboration with: G. Romano, A. Bernardini, M. D Amico, S. Mizzaro, S. Osinski, D. Weiss

2 Overview Clustering methods: K-Means Quality-threshold clustering Hierarchical agglomerative clustering Concept lattices Why so many clustering algorithms? Web Clustering Engines: Issues Systems KeySRC Problems New Directions Mobile search

3 The goal of clustering is to maximize the inter-class similarity and minimize the intra-class similarity Keys to clustering are the object representation and the distance measure

4 K-means algorithm

5 Quality threshold (QT) clustering 1. Choose a maximum diameter for clusters. 2. Build a candidate cluster for each point by including the closest point, the next closest, and so on, until the diameter of the cluster surpasses the threshold. 3. Save the candidate cluster with the most points as the first true cluster, and remove all points in the cluster from further consideration. 4. Recurse with the reduced set of points.

6 K-means versus QT clustering Deterministic? Need to specify cluster number? Computationally intensive? Every object must be clustered? k-means no yes no yes QT yes no yes no

7 Hierarchical agglomerative clustering

8 Cluster similarity measures

9 Single-link clustering Complete-link clustering

10 breathes in water (a) can fly (b) has beak (c) has hands (d) has skeleton (e) has wings (f) livesin water (g) vivipar - ous (h) 1 Bat x x x x 2 Eagle x x x x 3 Monkey x x x 4 Parrot fish x x x x 5 Penguin x x x x 6 Shark x x x 7 Lantern fish x x x x e produce light (i) eg ce ef eh aeg 467 ceg 45 cef 25 bef 12 aegi 7 aceg 4 cefg 5 bcef 2 befh 1 deh 3 abcdefgh

11 Concept lattice (definition) Context = (G, M, I) gim or (g, m) I means object g has property m For A G, B M define: A = m M gim for all g A B = g G gim for all m B A concept of (G, M, I) is a pair (A, B) where A G, B M, A = B, B = A (A and B are called extent and intent) A subset X G is an extent if and only if X = X; the concept is (X, X ) A subset Y M is an extent if and only if Y = Y; the concept is (Y, Y ) (A 1, B 1 ) < (A 2, B 2 ) if A 1 A 2 (or B 1 B 2 ) C(G, M, I, <) is a complete lattice and.(the Basic Theorem of concept lattices)

12

13 Why so many clustering algorithms? Clusters and outliers are in the eye of the beholder The strategy of cluster analysis is structure seeking although its operation is structure-imposing (e.g., by a clustering criterion) The main distinction is in the underlying model (e.g., representatives, tree search, graph search) Clustering often translates into optimazion problem solved by approximate algorithms (e.g., k-means, HAC, fuzzy concept lattices) The clustering algorithm must be compatible with the data structure (e.g., k-means cannot find non-convex clusters)

14 Web search results clustering

15 Search engines vs clustering engines (broad query)

16 Search engines vs clustering engines (broad query)

17 Search engines vs clustering engines (ambiguous query)

18 Web Clustering Engines: Issues Systems Problems New Directions

19 Features of clustering engines Advantages: Fast subtopic retrieval Exploring unknown or dynamic domains Filtering out irrelevant results Mostly good when plain search engines fail

20 Search results clustering (post-retrieval) versus traditional document clustering (pre-retrieval)

21 Architecture of Web clustering engines

22 Search results acquisition

23 Preprocessing of search results 1. Language recognition 2. Tokenization 3. Shallow language preprocessing 4. Feature selection

24 Cluster construction and labeling Description comes first : from data-centric to description-centric clustering algorithms

25 Keyphrase-based Search Results Clustering (1)

26 Keyphrase-based Search Results Clustering (2)

27 1) Cat ate cheese 2) Mouse ate cheese too 3) Cat ate mouse too Generalized suffix tree (from Zamir and Etzioni, 1998)

28 KeySRC algorithm 1. Search results preprocessing 2. Construction of Generalized Suffix Tree (GST) 3. Extraction of keyphrases from GST (internal nodes of GST + < 4 words + POS tagging) 4. Keyphrase clustering 5. Label assignment 6. Cluster ranking

29 Search results for query zebra

30 GST of zebra results software routing software genuus equus zebra equus mammal... equus mussel mollusk name $5 distributed.. $4 $6 $7 $6 $7 $6 zebra..$1 $5 distributed.. $4 mussel $6 $7 open.. $5 common.. $6 species.. $2 mammals.. $6 originated.. $3 name.. $2 $1 originated.. $3 name.. $2 $1

31

32 sim=1 thr=0.8 sim=1 thr=0.8 sim=1 thr=0.8 sim=0,76 thr=0.8 sim=0,65 thr=0.72 sim=0,873 thr=0.8 sim=1 thr=0.8 software routing software geuus equus equus mammal genus equus Mussel zebra mussel mollusk name Final clusters

33 Keyword-based visualization

34 Folder-based visualization

35 Nesting & zooming visualization

36 Graph-based visualization

37 Web Clustering Engines: Issues Systems Problems New Directions

38 Research prototypes Web clustering engines

39 Commercial Web clustering engines

40 Web Clustering Engines: Issues Systems Problems New Directions

41 Response times To improve efficiency: Client-side processing, incremental processing, pretokenized docs

42 Theoretical limitations of clustering engines Quality and usability of clusters is still unsatisfactory: - incompleteness of clusters, - lack of intra- and inter-cluster consistency, - label expressiveness, - different cluster granularity

43 Evaluation of retrieval performance still an open issues

44 Subtopic reach time

45 Label-driven Subtopic Reach Time

46 The AMBIENT (AMBIguous ENTries) test collection

47 Web Clustering Engines: Issues Systems Problems New Directions

48 Research directions More powerful search results indexing Multiple clustering methods combination Generation of more informative cluster labels Personalized clustering creation/reorganization Integration with ontologies

49 Potentials of mobile Web clustering engines Reduction of scrolling Reduction of typing for query refinement Reduction of downloaded data Extending mobile usage patterns to domain exploration

50 Mobile CREDO Architecture CREDINO (PDAs) SmartCREDO (cellphones) User s Commands XHTML pages Mobile CREDO server XML XML CREDO server Built on CREDO Bandwidth saving GUI for small screen

51 Credino (for PDAs)

52 SmartCREDO (for cellphones)

53 KeySRC on iphone

54 Web search results clustering at Fondazione Ugo Bordoni: papers and relevant URLs C. Carpineto, S. Osinski, G. Romano, G. Weiss (to appear). A Survey of Web Clustering Engines. To appear in ACM Computing Surveys. Carpineto, S. Mizzaro, G. Romano, M. Snidero (2009). Mobile Information Retrieval with Search Results Clustering: Prototypes and Evaluations. JASIST, 60(5), A.Bernardini, C. Carpineto, M. D Amico (submitted). Full-Subtopic Retrieval with Keyphrase-based Search Results Clustering. Submitted. Carpineto, C., Della Pietra, A., Mizzaro, S., and Romano, G. (2006). Mobile Clustering Engine. Proceedings of ECIR Carpineto, C., Romano, G.. (2004). Concept Data Analysis: Theory and Applications. John Wiley & Sons. CREDO Credino SmartCREDO AMBIENT KeySRC