Mining Social and Semantic Network Data on the Web

Transcription

1 Mining Social and Semantic Network Data on the Web Markus Schatten, PhD University of Zagreb Faculty of Organization and Informatics May 4, 2011

2 Introduction Web 2.0, Semantic Web, Web 3.0 Network science Data collection Semistructured data XPath Regular expressions Ajax APIs Data analysis Network matrices Folksonomy model Logic programming Case study - CROSBI 2 of 54

3 Outline Introduction Web 2.0, Semantic Web, Web 3.0 Network science Data collection Data analysis Case study - CROSBI 3 of 54

4 Questions Why mining social and semantic network data? How to collect these data? What are the common obstacles and how to solve them? How to analyze these data? 4 of 54

5 Outline Introduction Web 2.0, Semantic Web, Web 3.0 Network science Data collection Data analysis Case study - CROSBI 5 of 54

6 Web 2.0, Semantic Web, Web 3.0 Why should one bother with these buzzwords? 6 of 54

7 Why mining (social) web data? Millions of people, 7 of 54

8 Why mining (social) web data? Millions of people, on thousands of sites, 7 of 54

9 Why mining (social) web data? Millions of people, on thousands of sites, across hundreds of diverse cultures, 7 of 54

10 Why mining (social) web data? Millions of people, on thousands of sites, across hundreds of diverse cultures, leave trails 7 of 54

11 Why mining (social) web data? Millions of people, on thousands of sites, across hundreds of diverse cultures, leave trails about various aspects of their lives... 7 of 54

12 Why mining (social) web data? Millions of people, on thousands of sites, across hundreds of diverse cultures, leave trails about various aspects of their lives... Social systems are systems of communication (Niklas Luhman) 7 of 54

13 Why mining (social) web data? Millions of people, on thousands of sites, across hundreds of diverse cultures, leave trails about various aspects of their lives... Social systems are systems of communication (Niklas Luhman) By analyzing the trails (the results of social systems structural coupling) we can analyze the social system itself: 7 of 54

14 Why mining (social) web data? Millions of people, on thousands of sites, across hundreds of diverse cultures, leave trails about various aspects of their lives... Social systems are systems of communication (Niklas Luhman) By analyzing the trails (the results of social systems structural coupling) we can analyze the social system itself:politics, culture, media, business, technology, science... 7 of 54

15 Outline Introduction Web 2.0, Semantic Web, Web 3.0 Network science Data collection Data analysis Case study - CROSBI 8 of 54

16 The new science of networks source: The New York Times (April 3rd, 1933., p. 17). 9 of 54 source: An Attraction Network in a Fourth Grade Class (Moreno, Who shall survive?, 1934).

17 Political and financial networks Mark Lombardi (1980s & 1990s) 10 of 54

18 Terrorist networks 11 of 54

19 Board of directors membership networks source: 12 of 54

20 On-line social networks 13 of 54

21 Internet 14 of 54 source: Bill Cheswick

22 Air traffic networks source: Northwest Airlines WorldTraveler Magazine 15 of 54

23 Railway networks source: TRTA, March Tokyo rail map 16 of 54

24 Semantic networks source: 17 of 54

25 Gene networks source: 18 of 54

26 Food chain networks 19 of 54 source:

27 On-line social and semantic networks Where can we find networks? social networking sites 20 of 54

28 On-line social and semantic networks Where can we find networks? social networking sites forums, buletinboards, discussion groups, mailing lists 20 of 54

29 On-line social and semantic networks Where can we find networks? social networking sites forums, buletinboards, discussion groups, mailing lists blogs, microblogs, vlogs 20 of 54

30 On-line social and semantic networks Where can we find networks? social networking sites forums, buletinboards, discussion groups, mailing lists blogs, microblogs, vlogs wikis, semantic wikis 20 of 54

31 On-line social and semantic networks Where can we find networks? social networking sites forums, buletinboards, discussion groups, mailing lists blogs, microblogs, vlogs wikis, semantic wikis social tagging 20 of 54

32 On-line social and semantic networks Where can we find networks? social networking sites forums, buletinboards, discussion groups, mailing lists blogs, microblogs, vlogs wikis, semantic wikis social tagging RSS feeds, RDF repositories, metadata collections 20 of 54

33 On-line social and semantic networks Where can we find networks? social networking sites forums, buletinboards, discussion groups, mailing lists blogs, microblogs, vlogs wikis, semantic wikis social tagging RSS feeds, RDF repositories, metadata collections of 54

34 On-line social and semantic networks Where can we find networks? social networking sites forums, buletinboards, discussion groups, mailing lists blogs, microblogs, vlogs wikis, semantic wikis social tagging RSS feeds, RDF repositories, metadata collections... A social network can be found on any application where users communicate. 20 of 54

35 On-line social and semantic networks Where can we find networks? social networking sites forums, buletinboards, discussion groups, mailing lists blogs, microblogs, vlogs wikis, semantic wikis social tagging RSS feeds, RDF repositories, metadata collections... A social network can be found on any application where users communicate. A semantic network can be found in any specification where concepts are meaningfully connected. 20 of 54

36 Outline Introduction Web 2.0, Semantic Web, Web 3.0 Network science Data collection Semistructured data XPath Regular expressions Ajax APIs Data analysis Case 21 of 54study - CROSBI

37 Common obstacles in data collection Semi-structured data (HTML, XHTML, XML, RDF, OWL...) 22 of 54

38 Common obstacles in data collection Semi-structured data (HTML, XHTML, XML, RDF, OWL...) The need for pattern matching 22 of 54

39 Common obstacles in data collection Semi-structured data (HTML, XHTML, XML, RDF, OWL...) Client-side content generation The need for pattern matching 22 of 54

40 Semistructured data and XPath query language for semistructured data part of the W3C standard uses path expressions to navigate through documents has a standard library with over 100 useful functions Example: /html/body/div[@id= background ]/div/div[@id= mainbar ]/div/p[last()-1]/ 23 of 54

41 XPather (Firefox add-on) 24 of 54

42 Regular expressions Useful for extracting various formatted data and searching for keywords Example (date matching) [0-9]{1,2}[0-9]{1,2}[1-2][0-9]{3} 25 of 54

43 Client side generated content Some sites use Ajax to generate data on page load or user interaction (e.g. mouse click, mouse over etc.) 26 of 54

44 Client side generated content Some sites use Ajax to generate data on page load or user interaction (e.g. mouse click, mouse over etc.) In combination with obfuscated JavaScript code this is hard to scrape! Current solutions: Browser scripting (can be slow and cumbersome) - e.g. with selenium 26 of 54

45 Client side generated content Some sites use Ajax to generate data on page load or user interaction (e.g. mouse click, mouse over etc.) In combination with obfuscated JavaScript code this is hard to scrape! Current solutions: Browser scripting (can be slow and cumbersome) - e.g. with selenium Headless browser - e.g. mechanize, spidermonkey, HtmlUnit 26 of 54

46 Client side generated content Some sites use Ajax to generate data on page load or user interaction (e.g. mouse click, mouse over etc.) In combination with obfuscated JavaScript code this is hard to scrape! Current solutions: Browser scripting (can be slow and cumbersome) - e.g. with selenium Headless browser - e.g. mechanize, spidermonkey, HtmlUnit 26 of 54

47 Open APIs Most popular social web sites have an open API that allow you to connect your application to their site (Facebook, Twitter, Wikipedia,...) 27 of 54

48 Open APIs Most popular social web sites have an open API that allow you to connect your application to their site (Facebook, Twitter, Wikipedia,...) Still such APIs often have drawbacks for network data collection: 27 of 54

49 Open APIs Most popular social web sites have an open API that allow you to connect your application to their site (Facebook, Twitter, Wikipedia,...) Still such APIs often have drawbacks for network data collection: Limited access rights (e.g. Facebook) 27 of 54

50 Open APIs Most popular social web sites have an open API that allow you to connect your application to their site (Facebook, Twitter, Wikipedia,...) Still such APIs often have drawbacks for network data collection: Limited access rights (e.g. Facebook) Limited queries per time interval (e.g. Twitter) etc. 27 of 54

51 Outline Introduction Web 2.0, Semantic Web, Web 3.0 Network science Data collection Data analysis Network matrices Folksonomy model Logic programming Case study - CROSBI 28 of 54

52 Networks are most often analyzed using graph theory: Definition A graph G is the pair (N, E) whereby N represents the set of verticles or nodes, and E N N the set of edges connecting pairs from N. 29 of 54

53 Networks are most often analyzed using graph theory: Definition A graph G is the pair (N, E) whereby N represents the set of verticles or nodes, and E N N the set of edges connecting pairs from N. Definition A directed graph or digraph G is the pair (N, A), whereby N represents the set of nodes, and A N N the set of ordered pairs of elements from N that represent the set of graph arcs. 29 of 54

54 Networks are most often analyzed using graph theory: Definition A graph G is the pair (N, E) whereby N represents the set of verticles or nodes, and E N N the set of edges connecting pairs from N. Definition A directed graph or digraph G is the pair (N, A), whereby N represents the set of nodes, and A N N the set of ordered pairs of elements from N that represent the set of graph arcs. Definition A valued or weighted graph G V is the triple (N, A, V) whereby N represents the set of nodes or verticles, A N N the set of pairs of elements from N that represent the set of graph arcs, and V : N R a function that attaches values or weights to nodes. 29 of 54

55 Definition A bipartite graph G = (N, E) is a graph for which set of nodes there exists an partition N = {X, Y }, such that every arc has one end in X, a and the other in Y. 30 of 54

56 Definition A bipartite graph G = (N, E) is a graph for which set of nodes there exists an partition N = {X, Y }, such that every arc has one end in X, a and the other in Y. Definition A multigraph G = (N, E) is a graph in which E is a multiset, e.g. there can exist multiple edges between two nodes. 30 of 54

57 Matrix representation Definition Let G be a graph defined with the set of nodes {n 1, n 2,..., n m } and edges {e 1, e 2,..., e l }. For every i, j (1 i m and 1 j m) we define { 1, if there is an edge between nodes n i and n j a ij = 0, otherwise Matrix A = [a ij ] is then the adjacency matrix of graph G. The matrix i symmetric since if there is an edge between nodes n i and n j then clearly there is also an edge between n j and n i. Thus A = [a ij ] = [a ji ]. 31 of 54

58 Simple graph 32 of 54

59 Digraph 33 of 54

60 Digraph 34 of 54

61 Weighted graph 35 of 54

62 Bipartite graph 36 of 54

63 Multigraph 37 of 54

64 Matrix algebra and interpretation Matrices can be transposed, multiplied of 54

65 Matrix algebra and interpretation Matrices can be transposed, multiplied... Examples: P - process flow matrix (p p, p - number of processes) 38 of 54

66 Matrix algebra and interpretation Matrices can be transposed, multiplied... Examples: P - process flow matrix (p p, p - number of processes) W - co-worker matrix (w w, w - number of workers) 38 of 54

67 Matrix algebra and interpretation Matrices can be transposed, multiplied... Examples: P - process flow matrix (p p, p - number of processes) W - co-worker matrix (w w, w - number of workers) O - process responsibility matrix (p z) 38 of 54

68 Matrix algebra and interpretation Matrices can be transposed, multiplied... Examples: P - process flow matrix (p p, p - number of processes) W - co-worker matrix (w w, w - number of workers) O - process responsibility matrix (p z) WW - co-workers of co-workers 38 of 54

69 Matrix algebra and interpretation Matrices can be transposed, multiplied... Examples: P - process flow matrix (p p, p - number of processes) W - co-worker matrix (w w, w - number of workers) O - process responsibility matrix (p z) WW - co-workers of co-workers P T - inversed process flow 38 of 54

70 Matrix algebra and interpretation Matrices can be transposed, multiplied... Examples: P - process flow matrix (p p, p - number of processes) W - co-worker matrix (w w, w - number of workers) O - process responsibility matrix (p z) WW - co-workers of co-workers P T - inversed process flow O T O - matrix of workers who are responsible for the same processes 38 of 54

71 Matrix algebra and interpretation Matrices can be transposed, multiplied... Examples: P - process flow matrix (p p, p - number of processes) W - co-worker matrix (w w, w - number of workers) O - process responsibility matrix (p z) WW - co-workers of co-workers P T - inversed process flow O T O - matrix of workers who are responsible for the same processes PO - matrix of workers who are responsible for the forthcoming processes 38 of 54

72 Matrix algebra and interpretation Matrices can be transposed, multiplied... Examples: P - process flow matrix (p p, p - number of processes) W - co-worker matrix (w w, w - number of workers) O - process responsibility matrix (p z) WW - co-workers of co-workers P T - inversed process flow O T O - matrix of workers who are responsible for the same processes PO - matrix of workers who are responsible for the forthcoming processes P T O - matrix of workers who are responsible for the previous processes 38 of 54

73 Matrix algebra and interpretation Matrices can be transposed, multiplied... Examples: P - process flow matrix (p p, p - number of processes) W - co-worker matrix (w w, w - number of workers) O - process responsibility matrix (p z) WW - co-workers of co-workers P T - inversed process flow O T O - matrix of workers who are responsible for the same processes PO - matrix of workers who are responsible for the forthcoming processes P T O - matrix of workers who are responsible for the previous processes of 54

74 ACI model (Mika 2007) A - actors 39 of 54

75 ACI model (Mika 2007) A - actors C - concepts 39 of 54

76 ACI model (Mika 2007) A - actors C - concepts I - instances 39 of 54

77 ACI model (Mika 2007) A - actors C - concepts I - instances T A C I - folksonomy 39 of 54

78 ACI model (Mika 2007) A - actors C - concepts I - instances T A C I - folksonomy A tripartite hypergraph which can be reduced into three bipartite graphs: 39 of 54

79 ACI model (Mika 2007) A - actors C - concepts I - instances T A C I - folksonomy A tripartite hypergraph which can be reduced into three bipartite graphs: AC - network of actors and concepts 39 of 54

80 ACI model (Mika 2007) A - actors C - concepts I - instances T A C I - folksonomy A tripartite hypergraph which can be reduced into three bipartite graphs: AC - network of actors and concepts AI - network of actors and instances 39 of 54

81 ACI model (Mika 2007) A - actors C - concepts I - instances T A C I - folksonomy A tripartite hypergraph which can be reduced into three bipartite graphs: AC - network of actors and concepts AI - network of actors and instances CI - network of concepts and instances 39 of 54

82 Delicious - analysis 40 of 54

83 Problems with matrix representation 41 of 54

84 Problems with matrix representation Only the 1 -s contain information 41 of 54

85 Sparse matrix 42 of 54

86 Problems with sparse matrix If A is connected to B, then B is connected to A (undirected graphs) 43 of 54

87 Problems with sparse matrix If A is connected to B, then B is connected to A (undirected graphs) How to find friends of friends? 43 of 54

88 Problems with sparse matrix If A is connected to B, then B is connected to A (undirected graphs) How to find friends of friends? Is there a path from node A to node B? Which path is that? 43 of 54

89 Problems with sparse matrix If A is connected to B, then B is connected to A (undirected graphs) How to find friends of friends? Is there a path from node A to node B? Which path is that? How to compute the previously outlined graphs? 43 of 54

90 Problems with sparse matrix If A is connected to B, then B is connected to A (undirected graphs) How to find friends of friends? Is there a path from node A to node B? Which path is that? How to compute the previously outlined graphs? 43 of 54

91 Logic programming as a solution Deductive languages like Prolog, F LORA-2 44 of 54

92 Logic programming as a solution Deductive languages like Prolog, F LORA-2 Advantages: Declarative problem solving ( on a higher level ) Very appropriate for expressing mathematical structures 44 of 54

93 Logic programming as a solution Deductive languages like Prolog, F LORA-2 Advantages: Declarative problem solving ( on a higher level ) Very appropriate for expressing mathematical structures Disadvantages: Combinatoric explosion (use heuristics and predicate tabling) 44 of 54

94 Examples If A is connected to B, then B is connected to A link( A, B ) :- link( B, A ). 45 of 54

95 Examples Friend of a friend ff( A, B ) :- link( A, C ), link( C, B ). 46 of 54

96 Examples Is there a path from node A to node B? path( A, B ) :- link( A, B ). path( A, B ) :- link( A, C ), path( C, B ). 47 of 54

97 Examples Which path is that? whichpath( X, Y, [ X, Y ] ) :- link( X, Y ). whichpath( X, Y, [ X, Z T ] ) :- link( X, Z ), whichpath( Z, Y, [ Z T ] ). 48 of 54

98 Outline Introduction Web 2.0, Semantic Web, Web 3.0 Network science Data collection Data analysis Case study - CROSBI 49 of 54

99 Project outline Croatian Scientific Bibliography ( 50 of 54

100 Project outline Croatian Scientific Bibliography ( Scientists enter data about their research activities by them selves which makes it a social application 50 of 54

101 Project outline Croatian Scientific Bibliography ( Scientists enter data about their research activities by them selves which makes it a social application Research hypotheses: 50 of 54

102 Project outline Croatian Scientific Bibliography ( Scientists enter data about their research activities by them selves which makes it a social application Research hypotheses: 1. analyze two social systems (the Croatian scientific community and the Croatian public) and see how the most important research concepts (as viewed by the scientific community) are interpreted by the public (does the public understand most important concepts from Croatian science in the last decade?). 50 of 54

103 Project outline Croatian Scientific Bibliography ( Scientists enter data about their research activities by them selves which makes it a social application Research hypotheses: 1. analyze two social systems (the Croatian scientific community and the Croatian public) and see how the most important research concepts (as viewed by the scientific community) are interpreted by the public (does the public understand most important concepts from Croatian science in the last decade?). 2. analyze how various scientific fields are conceptually and socially interrelated (do scientists do interdisciplinary research together if they have conceptually connected fields?) 50 of 54

104 Methodology Social systems are represented by their trails : CROSBI as the Croatian scientific community, Croatian Wikipedia as the Croatian public One system can interpret concepts from another if the concept exists within its conceptual network (e.g. keyword or wiki page) 51 of 54

105 Data collection CROSBI data has been collected using Scrapy in November 2010 (a total of 285,234 entries has been collected) Wikipedia data has been collected using the Wikipedia API + Scrapy for parsing 52 of 54

106 Results 53 of 54

107 Bibliography 1. Adamic, L.: Why networks are interesting to study, University of Michigan, School of Information, 2. Barratm M., Barthlemy, M., Vespignani, A.: Dynamical Processes on Complex Networks, Cambridge University Press, Carley, K.M.: Dynamic Network Analysis, Summary of the NRC workshop on social network modeling and analysis, Committee on Human Factors, National Research Council, , Eds: Breiger, R., Carley, K.M., and Pattison, P., Divjak, B., Lovrenčić, A.: Diskretna matematika s teorijom grafova, TIVA & FOI, Krackhardt, David, and Carley, Kathleen M.: A PCANS Model of Structure in Organization, Proceedings of the 1998 International Symposium on Command and Control Research and Technology, Evidence Based Research, Vienna, VA, June Mika, Peter: Ontologies are us: A unified model of social networks and semantics, Web Semantics: Science, Services and Agents on the World Wide Web 5(1), volume 5, 515, March Newman, M., Barabsi, A.-L., Watts, D. j.: The Structure and Dynamics of Networks, Princeton University Press, Various web sites (images, graphs) 54 of 54