Data Mining for Business Intelligence: from Relational to Graph Representation

Transcription

1 Data Mining for Business Intelligence: from Relational to Graph Representation Marie-ude UFURE Ecole Centrale Paris cademic Chair in Business Intelligence

2 «TRDITIONL» DT MINING Knowledge extraction from large databases Input: data in a tabular form Mainly ignore relations between objects descriptive or predictive data mining Data Methods -Clustering -Similarity inside a cluster maximized -Similarity between clusters minimized -Overlapping clusters (hierarchical, FC) -ssociation rules: -Find associations between objects - Other methods

3 Concept Lattices Context table nimal Preying Mammal Flying Bird Lion X X Finch X X Eagle X X X Hare X Ostrich X Concept table Concept Intension Extension Top Ø Lion, Finch, Eagle, Hare, Ostrich 1 Bird Finch, Eagle, Ostrich 2 Preying Lion, Eagle 3 Mammal Lion, Hare 4 Flying, Bird Finch, Eagle 5 Preying, Mammal Lion 6 Bird, Flying, Preying Eagle Bottom Bird, Flying, Preying, Mammal Ø Concept lattice Lion, Finch, Eagle, Hare, Ostrich Bird Preying Mammal Finch, Eagle, Ostrich Lion, Eagle Lion, Hare Flying, Bird Preying, Mammal Finch, Eagle Lion Bird, Flying, Preying Eagle Bird, Flying, Preying, Mammal dvantages and drawbacks: + Groups objects into concepts according to their common properties +Keeps the semantics of data -Complexity -Need for understanding the way objects are clustered

4 Conceptual classification for e-reputation

5 Improving visualization: Trees as Lattice lternatives Idea: using known lattices measures to select best parent for each node stability, support, confidence, similarity and topological features Original lattice tree from the lattice Tree visualizations

6 What s new?? Data everywhere Big Data phenomenon Data are mainly unstructured 80% of data manipulated in an enterprise are unstructured Data are produced in real time and distributed Data come from heterogeneous sources in an unpredictable way Mobile phone, sensors, computers, TV, etc. Big Data phenomenon is considered as the main computer science challenge for the next decade Da

7 Graphs everywhere -Social networks -Web -Enterprise databases -Biology -Etc. Graphs can be seen as a way of managing structured and semi-structured data, as well as unstructured information. 7

8 Graphs: what can we do with? Traversing linked information, finding shortest path, doing (semantic) partition Recommendation and discovery of potentially interesting linked information Exploit the graph structure of large repositories Web environment Digital documents repositories Databases/Data Warehouses with metadata 8

9 Variety of graphs From simple graphs (basic mathematic definition): No information about nodes (all nodes have the same semantics, no attributes) Mainly focus on the relations between objects To labeled and attributed graphs dd semantic information to nodes nd more complex structures like Hypergraphs and Hypernodes allowing nested structures (complex attributes and/or relations) 9

10 (1)Complex-node creation (2)Relations Identification

11 The corresponding instance Graph

12 Graph transformation according to the user s point of view set of transformation patterns is applied to identify nodes and relations, and to extract new relations Director_thesis_Weber Jean Dir-id 38 Lab_id Laboratory_1 Thesis Thesis_1 Thesis_3 Director_thesis_Lochan Norman Student_Mohsen li St_id IS- Country Dir-id Thesis 03 Foreign_Student Egypt Director_thesis_2 Thesis_1 Same_Director_thesis St_id IS- Country Dir-id Same Laboratory Student_Yen Yang 12 Dir-id 27 Lab_id Laboratory_1 Thesis Foreign_Student China Director_thesis_2 Thesis_2 Student_Jack Pierre St_id 05 Dir-id Director_thesis_1 Thesis Thesis_2 Thesis Thesis_3 12

13 Graph ggregation: SNP & k-snp Tian, Hankins and Patel (SIGMOD 2008) Summarization based on user-selected node attributes and relationships. Provide drill-down and roll-up abilities to navigate multi-resolution summaries. Produce meaningful summaries for real applications (and multiple points of view) Efficient and scalable for very large graphs

14 SNP Operation ttributes first C C C B B B

15 SNP Operation Then, relations C C C B B B

16 ttributes and relationships ttributes and relationships together, but attributes first! For example: ll students in the blue group have the same gender and are in the same dept Every student in the blue group has: at least one friend in the green group at least one classmate in the purple group at least one friend in the orange group at least one classmate in the orange group

17 Graph ggregation: example Initial graph with selected nodes and relations: Nodes: Thesis-Director ttribute: grade Relations: Same_Laboratory and Same_Student Director_thesis_Weber Jean Dir-id 38 Lab_id Laboratory_1 Thesis Thesis_1 Thesis_3 Director_thesis_Lochan Norman Initial Graph Excerpt of the graph Student_Mohsen li St_id IS- Country Dir-id Thesis 03 Foreign_Student Egypt Director_thesis_2 Thesis_1 Same_Director_thesis St_id IS- Country Dir-id Same Laboratory Student_Yen Yang Thesis 12 Dir-id 27 Lab_id Laboratory_1 Thesis Foreign_Student China Director_thesis_2 Thesis_3 Thesis_2 Student_Jack Pierre St_id 05 Dir-id Director_thesis_1 Thesis Thesis_2 17

18 Graph ggregation 1 st iteration 2 nd iteration K-snap generates a summary formed by 3 groups (-compatible grouping): HDR, co-supervisor, prof (modalities of the attribute grade) 1 st iteration: subdivision of the HDR group into 2 subgroups according to the relation Same_Student: HDR_1: HDRs supervising a student with at least one professor or co-supervisor, HDR_2: HDR supervising students having only as director HDRs 2 nd iteration: subdivision of the Prof group into 2 subgroups according to the relation Same_Laboratory 18

19 Conclusion/Open problems/challenges Graphs: towards a unified view of structured data and unstructured content? Models: many existing models choose the most appropriate one! find communities that not only takes into account links between individuals, but also their similarities based on their own attributes Combine graphs algorithms with data mining methods dd semantics matching with a semantic layer Summarization ggregation Manage the consistency of the graph llow users to easily analyze the resulting graph 19

20 QUESTIONS? 20