Distributed Data Management

Transcription

1 Lecture Data Management Chapter 1: Erik Buchmann IPD, Forschungsbereich Systeme der Informationsverwaltung

2 of this Chapter are databases and distributed data management not completely different concepts, even contradictory? what is distributed? classification. transactions.. Query processing. Outline of this course. Erik Buchmann DDM: 2

3 IPD, Forschungsbereich Systeme der Informationsverwaltung

4 Situation without Databases (1) Access to data stored in files. Respective functionality is part of applications. Book New Entrant Book File Lending Reader File Reminder Lending File Erik Buchmann DDM: 4

5 Situation without Databases (2) Redundancy. Book New Entrant Lending Reminder Book File Book File Reader File Lending File Reader File Lending File Erik Buchmann DDM: 5

6 Situation without Databases (3) Other challenges: concurrency transactional guarantees (atomarity, consistency, isolation, durability) physical / logical data representation independence data privacy, data security no standard approach for the management of huge amounts of data Erik Buchmann DDM: 6

7 From Files to Databases Access to data stored in files. Respective functionality is part of applications. (Taking physical issues into account, concurrency, access control, consistency). Databases factor out this functionality. Book New Entrant Lending Reminder Book New Entrant Lending Reminder Book File Reader File Lending File Erik Buchmann DDM: 7

8 (Centralized) Databases Data integration, all applications use the same database Physical/logical data independence Efficiency, databases handle large volumes of data Integrity constraints, consistency even if many parallel users execute different transactions on the same data Declarative query languages, SQL Automatic query optimization When considering all these benefits, why bothering with distributed data management? Erik Buchmann DDM: 8

9 Why Data Management, After All? Data might be distributed to minimize communication costs Data might be distributed to equalize the workload among multiple nodes Data might be kept at the site of the creator in order to allow cheap updates Data might be replicated at multiple sites to improve availability, throughput and response times Some scenarios are distributed by nature, e.g., the IT infrastructure of a global company Erik Buchmann DDM: 9

10 Example: A Large Retailer 1 central enterprise resource planning system 100 stores In each store 1 manufacturing information system 3 points of sale Different data, queries, updates on each node Different hardware used Erik Buchmann DDM: 10

11 Databases vs. Data Management Databases and distributed data management approaches that seem to exclude each other. Databases: Application does not do the data management itself any more, data management is centralized. Data Management: Many different nodes manage different data at different locations. Erik Buchmann DDM: 11

12 Databases (1) Several databases, together with coordination layer. App. 1 App. 2 App. 3 Coordination Layer Again, one single point of access transparent to application Erik Buchmann DDM: 12

13 Databases (2) Do we end up with the deficiencies of the situation without databases again? has control over redundant data. Technology discussed here avoids inconsistencies. Generic functionality remains to be factored out. is transparent (at least, this is objective). User/application programmer has the illusion that he deals with a centralized DB. this objective is not always realistic. we will learn in the next chapters when compromises are unavoidable Erik Buchmann DDM: 13

14 IPD, Forschungsbereich Systeme der Informationsverwaltung

15 Systems, Computing system (of computers): Set of autonomous processors which are connected by a network, and which cooperate in fulfilling the tasks assigned to them. What is distributed? application logic, function, data, control. Erik Buchmann DDM: 15

16 Classification of Systems (1) Degree of coupling ratio of volume of data exchanged and extent of local processing; typical situations: communication via computer network weak coupling, shared components (main memory, secondary storage) strong coupling. of the connection alternatives: point-to-point, shared connection (bus). Erik Buchmann DDM: 16

17 Classification of Systems (2) Degree of independence of the components frequency of information exchange continuously or only at the beginning and the end (task, result), Synchronization of components synchronous asynchronous. Erik Buchmann DDM: 17

18 Why in the First Place? (1) Bottom up. Top Down. Erik Buchmann DDM: 18

19 Why in the First Place? (2) Bottom up: Corresponds to structure of organization, many modern information systems are distributed by nature (multimedia applications, ERPs, web-based information systems and kiosks), Erik Buchmann DDM: 19

20 Why in the First Place? (3) Top down: higher reliability, no single point of failure. Better performance and lower response times, data locality, divide&conquer for complex problems more computing power becomes available, software development less complex and therefore cheaper. Erik Buchmann DDM: 20

21 Performance Impact of distribution on performance: Less CPU- and I/O-contention. Data Locality Locality reduces network delays, less communication overhead, for physical reasons, this reduction has natural bounds in WANs. Inter- and intra-query parallelism. Systems that do caching: Intra-query parallelism may even yield superlinear speedup. Erik Buchmann DDM: 21

22 Read-Only Performance vs. Update Performance Mirroring the entire database only works in read-only environments, Approaches in database products: Multiplexing of the database, i.e., production database and query database. better query performance (if query is not part of a transaction that also contains updates) Time multiplexing: Batching of updates, only queries; vice versa later on. Erik Buchmann DDM: 22

23 Fragmentation vs. Replication NYC Employees, London Employees, NYC Projects NYC Employees, London Employees, NYC Projects, London Projects New York City Tokio Tokio Employees, Tokio Projects, London Projects Hong Kong London HK Employees, HK Projects Example of distributed application. Erik Buchmann DDM: 23

24 Example points out necessity of different kinds of transparency (see following slides): data independence, network transparency, replication transparency, fragmentation transparency. Erik Buchmann DDM: 24

25 Data Independence Data independence: Immunity of application against changes of data organization, Type of transparency that also exists in non-distributed context. Data Definition: Schema description, Physical data description. Correspondingly, two kinds of data independence: Logical data independence (e.g., attribute is added), physical data independence (e.g., index is changed). Erik Buchmann DDM: 25

26 Physical Data Independence Illustration NAME FIRSTNAME STREET AGE Böhm Klemens Nordstrasse 28 Buchmann Erik Breiter Weg 26 Duckstein Ralf Goethestrasse 25 Saake Gunter Waldweg 43 FIRSTNAME NAME STREET AGE Erik Buchmann Breiter Weg 26 Gunter Saake Waldweg 43 Klemens Böhm Nordstrasse 28 Ralf Duckstein Goethestrasse 25 Query, e.g., select NAME from PERSON where FIRSTNAME == 'Ralf' works for any representation. Erik Buchmann DDM: 26

27 Logical Data Independence Illustration Underlying relation: ESD(Employee, Salary, Department) create view highincomeemp as select Employee, Salary from ESD where Salary > 20 View can be used like normal relation, e.g.: select * from highincomeemp where Salary < 50 insert into highincomeemp values ('Klemens', 35000) Modifications of base relation might not affect view. Erik Buchmann DDM: 27

28 Network Existence of network (at least certain details on a technical level) should be hidden from application programmer. Two complementary variants: Location transparency: Command independent of location of the data and the system where command is executed. Naming transparency: Each object has unique name. (Otherwise: Application must insert location name as part of the object name.) Popular examples of location/naming transparency? Erik Buchmann DDM: 28

29 Network for Flexibility Requirements regarding performance become higher add new node, instead of replacement of entire system. Scale-out. Analogously, node may want to leave the (distributed) system. We could call it number-of-nodes transparency. Erik Buchmann DDM: 29

30 Replication In principle, replication is advantageous: Higher locality of reference, reliability and availability. Replication transparency: User does not see that replicas exist. (Thus, replication transparency is stronger than network transparency.) Important concern of this course: Show that mechanisms for replication transparency are very elaborate (in the presence of updates and failures), Present mechanisms for replication transparency. Erik Buchmann DDM: 30

31 Fragmentation Global query fragment queries NYC Employees, London Employees, NYC Projects New York City NYC Employees, London Employees, NYC Projects, London Projects London Tokio Tokio Employees, Tokio Projects, London Projects Hong Kong HK Employees, HK Projects Erik Buchmann DDM: 31

32 Transactions (1) Important topic of the course: distributed transactions. Transaction: Sequence of operations s.t. the system gives certain guarantees for its execution. Transaction transition from one consistent database state to another one. Terminology: Atomicity (failure atomicity) Consistency Isolation (concurrency transparency) Durability Erik Buchmann DDM: 32

33 Atomicity, Isolation Transactional guarantees in particular, atomicity and isolation. Atomicity Example, bank scenario : Bank Person Balance Sparkasse Klemens 5000 Deutsche Bank Gunter 200 Which distribution/replication scenarios are possible? Money transfer two elementary operations. debit(klemens, 500), credit(gunter, 500). Isolation can be explained with this example, too Erik Buchmann DDM: 33

34 Transactions (2) Several aspects: Failure atomicity (cf. previous example). transactions for replicated databases one objective of distributed data management: higher reliability. failure other parts of the distributed database shall remain accessible, operation goes on. Assumption: site failures and communication failures are always possible. Erik Buchmann DDM: 34

35 IPD, Forschungsbereich Systeme der Informationsverwaltung

36 What is a (1)? database = collection of several databases with logical relationships between them, distributed over a network of computers. = software administering the distributed database and hiding distribution from the users. we are not talking about simple shared resources scenarios Erik Buchmann DDM: 36

37 What is a (2)? Not a distributed DB: Collection of files on different nodes of a network Several databases on the same machine on multi-processor One centralized database in a network Reason: no common structure among the files; no common interface for applications Network is not the only common component. No message exchange necessary. Same database problems as before. Erik Buchmann DDM: 37

38 Federated vs. Federated a number of loosely coupled, indepentent different database schemas, query languages, transaction models, programming interfaces wrapper needed, virtual integration (Integrated) tightly coupled with logical relationships between them uniform view on all resources no wrapper, physical integration Erik Buchmann DDM: 38

39 Advantages of Advantages, as compared to centralized. local autonomy local control degree of autonomy not as high as with federated s higher performance smaller number of transactions in local, but: sometimes data from different nodes. higher reliability and availability failures do not affect the entire system extensibility new nodes cost effectiveness smaller computers, sharing of resources Erik Buchmann DDM: 39

40 Disadvantages (1) Design of a distributed is more complex than the one of a centralized. Replication of data objects Choice of copy to be read, Guaranteeing that update takes place on each copy. Site- and/or communication failure Effects of updates must go to all nodes in time. Synchronization of transactions more difficult in presence of several sites, as compared to centralized case. Erik Buchmann DDM: 40

41 Disadvantages (2) Complexity all intricacies known from centralized plus additional problems, see rest of this course. Costs not only hardware; but also software, communication, staff Control is decentralized coordination? Data security more components that require protection, and network in addition. Erik Buchmann DDM: 41

42 Query Processing (1) Query Processing not the same as read operations. What differences in the distributed case? Replication, fragmentation etc. query processing must be extended Such extensions are either straightforward, are interesting, e.g., semi-joins, but impact tends to be low. r A r C,r D r A,r B r C,r D Erik Buchmann DDM: 42

43 Query Processing (2) Slightly different perspective: sources are around, e.g., in enterprises. Queries over several/many of such sources. Solutions: Federated database systems, mediator-based architectures. Scenario behind more recent developments: Sources are highly distributed, e.g., WWW, queries over several/many such sources. r A r C,r D r A,r B r C,r D Erik Buchmann DDM: 43

44 Query Processing (3) Which changes? Availability of sources not clear, may change at any time. In general, semantics of data rather unclear to users. Cost issues frequencies of data values are less obvious; construction of histograms not feasible in general. Topics in this course: Continuous QP, Online QP. r A r C,r D r A,r B r C,r D Erik Buchmann DDM: 44

45 of the Course IPD, Forschungsbereich Systeme der Informationsverwaltung

46 of this Course (1) motivation, architecture of distributed s, brief intro to concurrency control, transactions atomic commit protocols, 2PC vs. 3PC; optimizations, e.g., presumed abort; transactions in federated databases, tickets, Replication transactional guarantees in distributed systems with replication, replication in presence of site failures and communication failures, lazy replication, epidemic protocols for update propagation, Erik Buchmann DDM: 46

47 of this Course (2) Caching semantic caching, cache consistency and cache coherency, distributed caching, prefetching, Query processing in distributed environments (with a focus on open, unreliable settings such as the Internet) continuous query processing, first-few queries and user-adaptive query evaluation. Data management in sensor networks query processing in unreliable scenarios where many independent nodes process sensor data; optimization of communication processes Erik Buchmann DDM: 47