Introduction Data Integration Summary. Data Integration. COCS 6421 Advanced Database Systems. Przemyslaw Pawluk. CSE, York University.

Transcription

1 COCS 6421 Advanced Database Systems CSE, York University March 20, 2008

2 Agenda 1 Problem description Problems 2 3 Open questions and future work Conclusion Bibliography

3 Problem description Problems

4 Why to do it? Problem description Problems Many data sources and applications in one organization

5 Why to do it? Problem description Problems Many data sources and applications in one organization The need of fast access and understanding of information from different sources.

6 Why to do it? Problem description Problems Many data sources and applications in one organization The need of fast access and understanding of information from different sources. Elimination of bottlenecks in the data-flow (many points where data is transformed).

7 Goals of data integration Problem description Problems To answer queries in most efficient way. (When we know exactly the queries we want to answer and what data are available) To discover the knowledge. (When don t know the queries in advance or we don t know all data sources in advance) In both we need to analyze data and present results according to requirements of users.

8 Goals of data integration Problem description Problems To answer queries in most efficient way. (When we know exactly the queries we want to answer and what data are available) To discover the knowledge. (When don t know the queries in advance or we don t know all data sources in advance) In both we need to analyze data and present results according to requirements of users.

9 Goals of data integration Problem description Problems To answer queries in most efficient way. (When we know exactly the queries we want to answer and what data are available) To discover the knowledge. (When don t know the queries in advance or we don t know all data sources in advance) In both we need to analyze data and present results according to requirements of users.

10 Problems in data integration Problem description Problems Identification of the best data source How to get data from the source? Interface to integrate schemes Formulation of queries to different data sources Problems with data incomplete and incorrect data inconsistent data incomprehensible data duplicates different formats Security issues Quality of Data Quality of Service reliability efficiency...

11 Problems with data Example Problem description Problems Many data formats i.e. John Brown, Brown J., Brown John

12 Problems with data Example Problem description Problems Many data formats i.e. John Brown, Brown J., Brown John Two different phone no. or addresses where integrity constraint allows only one

13 Problems with data Example Problem description Problems Many data formats i.e. John Brown, Brown J., Brown John Two different phone no. or addresses where integrity constraint allows only one Zip codes, phone numbers, SIN etc. with or without

14 Problems with data Example Problem description Problems Many data formats i.e. John Brown, Brown J., Brown John Two different phone no. or addresses where integrity constraint allows only one Zip codes, phone numbers, SIN etc. with or without Bad news! AI cannot solve all problems. Knowledge of meaning of the data is required!

15

16 Integration Process

17 Elements of the integration process Understanding meta-data analysis discover inconsistencies and dependencies

18 Elements of the integration process Understanding meta-data analysis discover inconsistencies and dependencies Standardization target schema data repair (to achieve consistent data) data identification constraints (How to identify object?)

19 Elements of the integration process Understanding meta-data analysis discover inconsistencies and dependencies Standardization target schema data repair (to achieve consistent data) data identification constraints (How to identify object?) Specification build mapping(s)

20 Elements of the integration process Understanding meta-data analysis discover inconsistencies and dependencies Standardization target schema data repair (to achieve consistent data) data identification constraints (How to identify object?) Specification build mapping(s) Execution materialization (ETL) federation (data exchange or query reformulation) indexing

21 Techniques

22 Materialization DS1 DS2 ODS DS3 The information pipeline[1]

23 Materialization DS1 DS2 ODS ETL DS3 The information pipeline[1]

24 Materialization DS1 DS2 ODS ETL CDB DS3 The information pipeline[1]

25 Materialization DS1 DM1 DS2 ODS ETL CDB DM2 DS3 DM3 The information pipeline[1]

26 Materialization DS1 DM1 DS2 ODS ETL CDB DM2 Present. DS3 DM3 The information pipeline[1]

27 Federation DS1 DS2 DS3

28 Federation Query processor DS1 DS2 DS3

29 Federation Presentation Query processor DS1 DS2 DS3

30 Federation Presentation Query processor Do not materialize data Usually use query reformulation or unfolding DS1 DS2 DS3

31 Indexing XML1 XML2 XML3

32 Indexing Index XML1 XML2 XML3

33 Indexing Presentation Query processor Index XML1 XML2 XML3

34 Indexing Presentation Query processor Index Do not materialize data Use index to find proper document XML1 XML2 XML3

35

36 Views

37 Views in data integration Architecture based on global schema (GS) and set of sources Data in the data sources (DS) Global schema as virtual view of DS Each DS and GS can be expressed in a different language The goal is to define some mappings between DSs and GS Two approaches: Global as view Local as view

38 Global as view Global schema expressed in terms of data sources. Each element of the global schema is defined as a view over the sources.

39 Global as view Global schema expressed in terms of data sources. Each element of the global schema is defined as a view over the sources. Requirement! Stable set of data sources!

40 GAV benefits The benefits of GAV: Easier to understand Mapping explicitly tells how to retrieve data from DSs

41 Local as view Global schema defined independently from data-sources, and the relationships between the global schema and the sources are expressed by defining every source as a view over the global schema.

42 Local as view Global schema defined independently from data-sources, and the relationships between the global schema and the sources are expressed by defining every source as a view over the global schema. Requirement! Stable global schema!

43 LAV benefits The immediate benefits of LAV: Easier describing sources(doesn t require any knowledge about other DS) New DS can be easily added Describing precise constraints on the contents of DS and describing sources that have different relational structure than the GS is easier

44 GAV vs. LAV The LAV system can be transformed into a GAV one and vice-versa.

45 GAV vs. LAV The LAV system can be transformed into a GAV one and vice-versa. LAV to GAV can be used to derive the procedural specification from declarative one.

46 GAV vs. LAV The LAV system can be transformed into a GAV one and vice-versa. LAV to GAV can be used to derive the procedural specification from declarative one. GAV to LAV can be useful to derive a declarative characterization of the content of the sources starting from procedural specification.

47 Answering queries using views Depends on chosen technique GAV gives simple solution how to get the data. Just unfold! LAV uses view-based query rewriting or view-based query answering. More complicated than GAV, uses reasoning in the presence of incomplete information

48 Schema mapping

49 Schema mapping Data exchange How to move data from one schema to the other one? Data integration How to integrate data from many different schemes?

50 Schema mapping in data exchange problem

51 LAV and GAV Example In general elements of the set Σ can have a form φ(x) yψ(x, y) For example (Student(s) Enrolls(s, c)) t g(teaches(t, c) Grade(s, c, g)) In LAV there is only one element on the left side P(x) yψ(x, y) In GAV there is only one element on the right side φ(x) R(x)

52 Schema composing

53 Schema inverting in schema evolution

54 Schema mapping Example of changes in schemas Why evolution? new atribute add/remove constraint... Example new information about student or course required changes in the grading system...

55

56 Example banking system Our point of view: CRM system Questions: The purpose of the system The source/sources of data for the system?...

57 Example Banking systems Customers DB CRM System

58 Example Banking systems Customers DB CRM System Cards DB WebSys. Data Account DB Contracts

59 Example Banking systems Different formats and accuracy of data Different Database Management Systems Different owners (security levels)...

60 What is Master Data? Facts describing core business entities Provides business context Does not imply a particular style of usage Not everything is Master Data...

61 What Is Master Data Management? An approach that decouples master information from individual applications and unifies it. Central application- and process-neutral resource Ensures consistent, up-to-date master information across business processes, and systems Simplifies ongoing integration tasks and new application development

62 Problems...

63 Proposed solution Master Data Management brakes walls between the systems and provides fast access to the consistent information.

64 Open questions and future work Conclusion Bibliography

65 Open questions Open questions and future work Conclusion Bibliography Automatic schema mapping Reference reconciling Model management Peer-to-Peer Data Management

66 Conclusion Open questions and future work Conclusion Bibliography Data integration theoretical as well as business problem Some commercial solutions are available but not sufficient Many issues without solution = reach research field

67 Open questions and future work Conclusion Bibliography Bibliography

68 Open questions and future work Conclusion Bibliography Brazhnik, O., and Jones, J. F. Anatomy of data integration. J. of Biomedical Informatics 40, 3 (2007), Fagin, R., Kolaitis, P. G., Tan, W.-C., and Popa, L. Composing schema mappings: second-order dependencies to the rescue. In PODS 04: Proceedings of the twenty-third ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems (New York, NY, USA, 2004), ACM, pp Haas, L. M. Beauty and the beast: The theory and practice of information integration. In ICDT (2007), T. Schwentick and D. Suciu, Eds., vol of Lecture Notes in Computer Science, Springer, pp

69 Open questions and future work Conclusion Bibliography Halevy, A., Rajaraman, A., and Ordille, J. Data integration: the teenage years. In VLDB 06: Proceedings of the 32nd international conference on Very large data bases (2006), VLDB Endowment, pp Halevy, A. Y. Answering queries using views: A survey. The VLDB Journal 10, 4 (2001), Kolaitis, P. G. Schema mappings, data exchange, and metadata management. In PODS 05: Proceedings of the twenty-fourth ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems (New York, NY, USA, 2005), ACM, pp Lenzerini, M.

70 Open questions and future work Conclusion Bibliography Data integration: a theoretical perspective. In PODS 02: Proceedings of the twenty-first ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems (New York, NY, USA, 2002), ACM, pp Pankowski, T. Integracja danych w teorii i praktyce przeglad problemow i rozwiazan. In Bazy Danych: Nowe Technologie (2007), WK, pp Ra, Y.-G., and Rundensteiner, E. A. A transparent schema-evolution system based on object-oriented view technology. IEEE Transactions on Knowledge and Data Engineering 9, 4 (1997), Shahri, H. H., and Shahri, S. H.

71 Open questions and future work Conclusion Bibliography Eliminating duplicates in information integration: An adaptive, extensible framework. IEEE Intelligent Systems 21, 5 (2006),