Prof. Dr. Christian Bizer

Transcription

1 STI Summit July 6 th, 2011, Riga, Latvia Global Data Integration and Global Data Mining Prof. Dr. Christian Bizer Freie Universität ität Berlin Germany

2 Outline 1. Topology of the Web of Data What data is out there? 2. Global Data Integration How to split the integration effort 3. Global Data Mining The logical next step

3 Linked Data Deployment on the Web Year Datasets Triples Growth % % %

4 Uptake in the Government Domain The EU is starting to publish Linked Data (LOD2, LATC) Various other national efforts W3C egovernment Interest Group

5 Uptake in the Libraries Community Institutions publishing Linked Data Library of Congress (subject headings) German National Library (PND dataset and subject headings) Swedish National Library (Libris - catalog) Hungarian National Library (OPAC and Digital Library) Europeana project just released data about 4 million artifacts t Growth of Library Linked Data ( ): 1000% W3C Library Linked Data Incubator Group Goals: 1. Integrate Library Catalogs on global scale. 2. Interconnect resources between repositories (by topic, by location, by historical period, by...).

6 LOD data set statistics as of November 2010 Domain Data Sets Triples Percent RDF Links Percent Cross domain 20 1,999,085, ,105, Geographic 16 5,904,980, ,589, Government 25 11,613,525, ,658, Media 26 2,453,898, ,374, Librariesi 67 2,237,435, ,951, Life sciences 42 2,664,119, ,417, User Content 7 57,463, ,402, ,930,509, ,499,896 LOD Cloud Data Catalog on CKAN net/group/lodcloud More statistics

7 What are the big players doing?

8 Structured Data becomes a SEO Topic Data Snippets Query Answer

9 Result: Further growth usage g of RDFa has increased 510% between March, 2009 and October, million webpages contain RDFa Source: Yahoo

10 The Structural Continuum The Web of Data is interwoven with the classic Web. Unstructured text: HTML Structured data: RDFa embed into HTML (Open Graph) Microdata embed into HTML (Schema.org) Microformats embed into HTML Linked data: RDF/XML

11 Topology of the Web of Data

12 How to get the data? Download the Billion Triples Challenge Dataset 2 billion triples (20GB gzipped) crawled from the public Web of Linked Data in May/June Download the Sindice Dump 12 billion triples (164GB gzipped, ~1,16TB 16TB uncompressed) crawled from the public Web of Linked Data and includes RDFa, Microformat, and wrapped API data

13 2. Global Data Integration Applications hate heterogeneity! The wild wild west My little world

14 The Dataspace Vision Alternative to classic data integration systems in order to cope with growing number of data sources. Properties of dataspaces no upfront investment into a global schema rely on pay-as-you-go data integration ti give best effort answers to queries Franklin, M., Halevy, A., and Maier, D.: From Databases to Dataspaces A new Abstraction for Information Management, SIGMOD Rec Madhavan, J., et al.: Web-scale Data Integration: You Can Only Afford to Pay As You Go, CIDR 2007

15 Linked Data relies on Pay-as-You-Go Idea for Identity Management for Schema/Vocabulary Management

16 Publish Identity Links on the Web Identity Link < owl:sameas < You publish links pointing at other data sources. Somebody else publishes links pointing at your data source.

17 Effort Distribution between Publisher and Consumer Consumer data mines identity links Effort Distribution Publishers or third parties provides identity links

18 Vocabularies on the Web of Data Everyone can use whatever vocabularies she likes to publish Data on the Web. Or invest effort and reuse Common Vocabularies Friend-of-a-Friend for describing people and their social network SIOC for describing forums and blogs SKOS for representing topic taxonomies Organization Ontology for describing the structure of organizations GoodRelations provides terms for describing products and business entities Music Ontology for describing artists, albums, and performances Review Vocabulary provides terms for representing reviews Many Linked Data Source use mixture of common and proprietary vocabulary terms.

19 Publish Vocabulary Links on the Web < owl:equivalentclass < Vocabulary Link Simple Mappings: RDFS, OWL rdfs:subclassof, rdfs:subpropertyof owl:equivalentclass, owl:equivalentproperty Complex Mappings: R2R provides value transformation functions structural transformations

20 Deployment of Vocabulary Links Source: Linked Open Vocabularies,

21 Effort Distribution between Publisher and Consumer Consumer defines or data mines mappings Effort Distribution Publisher reuses vocabularies Publisher or third party publishes mappings

22 Somebody-Pays-As-You-Go The overall data integration effort is split between the data publisher, the data consumer and third parties. Data Publisher publishes data as RDF sets identity links reuses terms or publishes mappings Fix Overall Data Integration Effort Third Parties set identity links pointing at your data publish mappings to the Web Publisher s Effort Third Party Effort Data Consumer has to do the rest using record linkage and schema matching techniques Consumer s Effort

23 Research Directions 1. More research on pay-as-you-go data integration is needed. 2. More research on data mining mappings and identity resolution heuristics is needed. Identity links make it easier to mine vocabulary links. Vocabulary links make it easier to mine identity links. 3. More research on SPAM detection and data quality assessment is needed.

24 LDIF Linked Data Integration Framework Combines vocabulary normalization and identity resolution Currently only in-memory implementation ti Next release: Hadoop-based implementation i /bi /ldif/ Normalize vocabularies Identity Resolution

25 What can we do afterwards build better entity search engines

26 3. Global Data Mining

27

28 Think about interesting questions that you can answer based on the Web of Data that require aggregation summarization classification association rule mining combined with text mining sediment analysis

29 Everybody has the tools to find the answers

30 Research Directions 1. More research on data space profiling is needed. 2. More research on global l data mining i is needed. d Google, Yahoo, Microsoft, Facebook will get there soon.

31 Semantic Web Challenge Submission Statistics Year Open Track Billion Triple Track Do something interesting with the Billion Triple Data and submit your results to the challenge until October 1 st present your results at the 10th International Semantic Web Conference (ISWC2011), October 2011, Koblenz, Germany

32 Conclusions The Web of Data is there Linked Data, Microdata, RDFa, Microformats Upcoming research topics pay-as-you-go data integration mapping discovery, schema clustering identity resolution heuristics discovery probabilistic data integration data quality assessment data space profiling global data mining

33 Thanks! References Textbook: Tom Heath, Christian Bizer: Linked Data: Evolving the Web into a Global Data Space. Christian Bizer, Tom Heath, Tim Berners-Lee: Linked Data The Story So Far