Tecnologie DB2 LUW per la distribuzione dei dati

Transcription

1 Tecnologie DB2 LUW per la distribuzione dei dati Una panoramica Mariangela Fumagalli Senior IT Specialist Software Group - IBM Italia mariangela_fumagalli@it.ibm.com Michele Benedetti Senior IT Specialist Software Group - IBM Italia michele_benedetti@it.ibm.com Francesco Airoldi Executive Architect ets Team - IBM Italia airfranz@it.ibm.com

2 Agenda Introduction Distributed Access Data Federation Data Replication 2 DB2 LUW overview

3 Agenda Introduction Distributed Access Data Federation Data Replication 3 DB2 LUW overview

4 Distributed data: summary DATA MOVE: NO connect DBMS DB connect DBMS DB connect DBMS DB connect Fed Srv DBMS DBMS DB DB Basic (single db) Distributed Access Federation DATA MOVE: YES DBMS DB Repl Srv Replication DBMS DB DBMS DB EP Srv Event Publishing DBMS DB ETL srv DBMS DB Extract Trasform & Load DBMS DW 4 4 DB2 LUW overview

5 Basic (single db) connect DBMS DBMS engine based SQL is used Insert ( DB) Select (DB ) Logging yes, concurrency control yes DB Utilities DBMS engine bypassed SQL not used Load (external data DB) Unload (DB external data) Load DBMS Unload For large and very large data volumes Logging may be disabled DB DB objects (e.g. tables) may be locked while the utility runs Various external data format supported Special techniques to handle anomalous data (e.g. duplicate keys) 5

6 Distributed Access connect DBMS DB connect DBMS DB Based on the DRDA standard DRDA = Distributed Relational Database Architecture Proposed by IBM, now adopted as a database interoperability standard from The Open Group Implemented in all IBM products belonging to the DB2 Family, and by several non-ibm products SQL based Key points: How many SQL statements per Unit of Work (UOW)? How many databases per UOW? How many databases referenced in a single SQL statement? Read-only access or not? DBMS belonging to the same family (homogeneous) or not (heterogeneous scenarios)? 6

7 Federation connect Fed Srv DBMS DBMS DB DB Extension of the Distributed Access model to heterogeneous environments ications connect to a single virtual database Can be extended to allow access to non-relational data sources Key points: Query optimization Differences in DBMS (SQL dialects, data types, semantics ) Two-phase commit required? How to handle the non-relational data sources? Performances 7

8 Replication Data physically copied from a source system to a target system DBMS DB Repl Srv Replication DBMS DB Often between heterogeneous DBMS Many topological variants: One-to-one One-to-many Many-to-one Fan-out One-way or bidirectional Key points: Total replica (bulk) or delta replicas (change capture) Batch (e.g. once a day) or real-time (continuous replication) Table-based or transaction-based Change capture: triggers or log-based Performance impact on running applications Conflict detection and resolution (bidirectional, many-to-one) Data replicated as-is or transformed in flight Transport mechanism between source and target systems 8

9 Event Publishing Event Publishing is a variant of Replication DBMS DB EP Srv Data that change in a DBMS when certain events occur are sent to external applications New rows inserted Existing rows deleted Existing rows updated.. Key points: Data format (e.g. xml) Event published in real time or not May be a component of EAI (Enterprise ication Integration) scenarios (e.g. when the target application is an ESB system) 9

10 Extract Transform & Load (ETL) Theoretically, ETL is an extension of the Replication model DBMS DBMS DB DB ETL srv DBMS In practice, ETL is the key technology for feeding data into a Data Warehouse system: Extract data from operational data sources Transform data in a format suitable to be used in the DW environment Load data into the DW dbms DW Occasionally, some variants may be used: ELT (Extract Load Transform) TEL (Transform Extract Load) Key points: Availability of connectors for different data sources (also non-relational) Data volumes to be handled per unit of time (performances) Batch or near real-time Complexity of data transformation required Further data transformation within the DW are possible Data quality System management and overall governance 10

11 Agenda Introduction Distributed Access Data Federation Data Replication 11 DB2 LUW overview

12 DRDA Distributed Relational Database Architecture (DRDA) is a database interoperability standard from The Open Group. DRDA describes the architecture for distributed data. It defines the rules for accessing the distributed data, but it does not provide the actual application programming interfaces (APIs) to perform the access. It was first used in DB High level architecture connect AR AS SQL ication Support Protocol Database Support Protocol DS Database Support Protocol DS ication Requester (AR) The AR accepts SQL requests from an application and sends them to the appropriate application servers for processing. Using this function, application programs can access remote data. ication Server (AS) The AS receives requests from application requesters and processes them. The AS acts upon the portions that can be processed and forwards the remainder to database servers for subsequent processing. The AR and the AS communicate through a protocol called the ication Support Protocol which handles data representation conversion. Database Server (DS) The DS receives requests from AS or other DS servers. The DS supports distributed requests and will forward parts of the request to collaborating DS in order to fulfill the request. The AS and the DS among themselves communicate through a protocol called the Database Support Protocol. 12

13 DRDA Implementation levels and capabilities DRDA Level 0: Remote Request SQL AR connect One DBMS One Unit Of Work (UOW) One SQL request SQL example connect to REM_DB insert into REM_TAB1... commit Implementation example AS DB2 client connect DRDA Level 1: Remote Unit Of Work (RUOW) SQL example DB2 LUW server SQL AR connect One DBMS One Unit Of Work (UOW) Multiple SQL requests connect to REM_DB insert into REM_TAB1... update REM_TAB2 select from REM_TAB3.. commit REM_DB AS Switch from one to another dbms is possible, but you need to close the UOW and disconnect from the first dbms, then connect to the second dbms and open another UOW 13

14 DRDA Implementation levels and capabilities DRDA Level 2: Distributed Unit Of Work (DUOW) connect AS SQL AR connect AS Multiple DBMS One Unit Of Work (UOW) Two-Phase commit (2PC) required Transaction Manager required Multiple SQL requests Each SQL request limited to a single DBMS SQL example connect to REM1_DB connect to REM2_DB select. from REM1_TAB1 insert into REM2_TAB2 delete from REM1_TAB1.. commit Implementation example REM1_DB connect DB2 LUW server DB2 client connect DB2 z/os server REM2_DB The 2PC functionality is compliant with the Open Group XA specification for distributed transaction processing, where the two basic roles are Transaction Manager TM) Resource Manager (RM) The Transaction Manager role may be fulfilled by a DRDA AR, a DRDA AS, or even by an external component. The Resource Manager role may be fulfilled by a DRDA AS or a DRDA DS Acts as Trx Mgr 14

15 DRDA Implementation levels and capabilities SQL AR connect AS DS DRDA Level 3: Distributed Request (DR) Multiple DBMS One Unit Of Work (UOW) Two-Phase commit (2PC) required Transaction Manager required Multiple SQL requests One SQL request may refer to objects managed by different DBMS (e.g. distributed join) SQL example connect to FED_DB select... from NICK_LOC, NICK_REM... update NICK_REM... insert into NICK_LOC. commit Implementation example connect DB2 client The AR connects to an AS that owns local data and forwards part of the SQL request to a DS There is no explicit connection form the AR to the DS: the latter is connected under the hood by the AS The AR sees only the AS, which acts as federator over its local data and the data managed by the second DBMS (the DS) Database object are referenced through nicknames LOC_DB DB2 LUW server Acts as Federation Server and Trx Mgr DB2 LUW server REM_DB A natural evolution of this capability is the Data Federation scenario, where the AS does not own local data and the underlying DBMS may be heterogenous 15

16 DRDA Mapping DRDA roles on some IBM software products AR DRDA Req DRDA Server DRDA Database Server XA Trx Mgr AS DS DB2 Client Y N N N DS DB2 Connect Y N N Y DB2 LUW Y Y Y Y DB2 z/os Y Y Y Y InfoSphere Federation Server Y Y (N) Y 16

17 Agenda Introduction Distributed Access Data Federation Data Replication 17 DB2 LUW overview

18 Different Integration Techniques Meet Different Requirements Federation Consolidation Web ications Analytical & Reporting Tools Data Warehouse Federation Consolidation Product Performance Real-time Inventory Level Region 1 Product Performance Region 2 Product Performance Headquarters Replication Replication Stores Event Publishing Capture & Publish Database Primary Data Center Replication Backup Data Center EAI Repl ETL RYO 18

19 Federation- How does it work? Federation Consolidation Web ications Analytical & Reporting Tools Data Warehouse Federation Consolidation Product Performance Real-time Inventory Level Region 1 Product Performance Region 2 Product Performance Headquarters Replication Replication Stores Event Publishing Capture & Publish Database Primary Data Center Replication Backup Data Center EAI Repl ETL RYO 19

20 Federation Data Sources SQL InfoSphere Federation Server Relational databases Web Other Read - Write DB2 for iseries DB2 for z/os DB2 for LUW Informix Oracle Sybase Teradata Microsoft SQL Server ODBC JDBC MQ UDF Excel Table-structured files Web services OLE DB Scripts Custom-built Read only 20

21 FS Basic Concepts DB2 Catalog Federated Server Stores information about: Wrappers,servers, nicknames Server attributes Nickname attributes Remote functions Server Nickname Wrapper Data Source Client Nickname Nickname Nickname Server Server Wrapper Data Source Client Federated server: a DB2 database enabled for federation. Wrapper: a library allowing access to a particular class of data sources or protocols (Net8, DRDA, CTLIB...). Contains information about data source characteristics Server: represents a specific data source Nickname: a local alias to data on a remote server (mapped to rows and columns); appears as a DB2 table 21 Orders Customers

22 Query Optimizer Flow for Federated Queries Push Down Analysis (PDA) is a component of query compilation PDA determines whether or not an operation can be pushed down to the data source Just because processing can be pushed down, does not mean it will be If an operation can be pushed down, the optimizer still has the final say on whether or not the operation is pushed down. 22

23 Agenda Introduction Distributed Access Data Federation Data Replication 23 DB2 LUW overview

24 Different Integration Techniques Meet Different Requirements Federation Consolidation Web ications Analytical & Reporting Tools Data Warehouse Federation Consolidation Product Performance Real-time Inventory Level Region 1 Product Performance Region 2 Product Performance Headquarters Replication Replication Stores Event Publishing Capture & Publish Database Primary Data Center Replication Backup Data Center EAI Repl ETL RYO 24

25 Many Topologies Possible SOURCE CD1 SOURCE CD1 SOURCE CD1 TARGET TARGET TARGET SOURCE CD1 SOURCE CD1 STAGING CD1 STAGING CD1 Data Distribution (1:many) Peer-to-Peer SOURCE CD1 TARGET SOURCE CD1 SOURCE CD1 Data Consolidation (many:1) Conflict Detection/Resolution PRIMARY CD1 SECONDARY Bi-directional TARGET TARGET TARGET Multi-Tier Staging TARGET 25

26 Changed Data Replication ications make changes to a database (the source) Changes are then: Read, captured, from the database log Copied to other systems (the targets) ied to tables Diagram shows one-way, or unidirectional, replication Source SOURCE2 SOURCE1 Target TARGET 1 Capture y TARGET N TARGET TARGET 2 Log Subsets Transformations History Tables 26

27 Q Capture Process Flow SOURCE2 SOURCE1 Q-SUBS Q-PUBS Restart Queue DB2 Log TX1: INSERT S1 TX2: INSERT S2 TX3: DELETE S1 TX1: UPDATE S1 TX1: COMMIT TX3: ROLLBACK CAPTURE In-Memory-Transactions TX3: DELETE S1 TX3: ROLLBACK TX1: INSERT S1 TX1: UPDATE S1 TX1: COMMIT TX2: INSERT S2 Zapped at Abort Never makes it to send queue Send Queue MQ Put when Commit record is found Transaction is still in-flight Nothing inserted yet. 27

28 Q Replication - The BIG Picture SOURCE SOURCE2 METADATA SOURCE1 METADATA Q y Browser DB2 Log Q Capture y Agent y Agent y Agent Subsets Transformations History Tables ADMINISTRATION Replication Center Replication Monitor TGT1 TGT2 TGT3 ications make changes to a database source Changes are then: Read, captured, from the database log Copied to other systems (the targets) ied to tables TARGET 28

29 Data Event Publishing Conceptually, data replication without the apply Change data is made available to consuming applications Examples, InfoSphere DataStage or a message broker One common delivery mechanism is WebSphere MQ Target Source SOURCE2 SOA/User ication SOURCE1 Capture WBI Event Broker User ication TARGET DB Log InfoSphere DataStage TARGET TARGET 29

30 Essential links Distributed Access DB2 for Linux Unix and Windows home page DB2 for LUW and DB2 Connect 9.7 InfoCenter (search for DRDA) Data Federation and Data Replication InfoSphere Federation Server home page InfoSphere Replication Server home page InfoSphere Event Publisher home page InfoSphere Information Server V8.7 InfoCenter Overview > Introduction to InfoSphere Information Server > Companion products 30

31 Questions? 31 DB2 LUW overview

32 Grazie 32 DB2 LUW overview

33 Additional slides on Data Federation InfoSphere Federation Server Customer Requirements and Scenarios How it works Performance considerations InfoSphere Federation Server 33

34 Problems Delivering Data to the End User Multiple sources for the same entity Heterogeneous data sources: DB2, Oracle, Microsoft SQL Server, XML files, spreadsheets, etc. Employees spend significant amount of their time (70%) searching for information Lack of an integrated view of information Time-consuming and costly aggregation 34

35 InfoSphere Federation Server Transparent Appears to be one source Independent of how and where data is stored ications continue to work despite of any change in how data is stored Heterogeneous Accesses data from diverse sources: relational, structured, messages Extensible Bring together almost any data source Wrapper Development Toolkit Access and integrate heterogeneous information across multiple sources as if they were a single source Extend value of existing analytical applications by providing real-time access to integrated information High Function Full query support against all data Capabilities of sources as well Autonomous Non-disruptive to data sources, existing applications, systems InfoSphere Federation Server High Performance Optimization of distributed queries Excel Oracle Web Services SQL Server. 35

36 Customer-Data Integration Customer Challenge: Providing a holistic view of information for customer-facing or customer supporting applications High development and maintenance costs to access diverse data sources Maximizing value of customer data for customer satisfaction/retention and increasing sales Customer value: Reduce coding and skills requirements when integrating two or more sources Reduce redundant data by consolidating only frequently accessed data Reduce application maintenance costs Extend customer data with document and other content data ication Developer ication Development effort to handle: Unique interfaces InfoSphere for each data type Federation Joining data from varied sources Transformations Correlating data Server Nontraditional data Nonrelational data RDBMS 36

37 Access to Regionally Distributed Data Requirements Solution Several regional databases with similar logical data models, but unique data ication needs to see the data as one large database with a single schema Impractical to physically consolidate data Access relevant remote tables via FS nicknames Linux Connect matching nicknames from different ORACLE sources via a UNION ALL view Can optionally cache common data at the FS or create local aggregates Windows SQL Server InfoSphere Federation Client Linux ORACLE Seattle Linux ORACLE InfoSphere Federation Server Phoenix San Jose 37 Linux ORACLE

38 Speeding Portal ication Development Customer Challenge: Integrating multiple data sources in a single application is complex and costly Accessing non-traditional sources is too impractical to leverage their benefit Time pressure to deploy new applications Scarcity of skills who can work with legacy, non-traditional data sources Extending built-in search to new domains Customer value: Reduce amount of integration coding by 40-65% Use existing SQL tools to access all data Give applications access to all the relevant data sources Reduce application maintenance costs Deploy existing skills over wider range of integration projects ication Developer Federated ication Connection to Federated server DB2 Portal ication Connection to all individual data Development effort sources to handle: Federation Non-Federated Multiple portlets, one Server ication for each source InfoSphere Unique interfaces for Federation each data type Server Oracle Joining data from varied sources Transformations Correlating data Excel/ODBC local DB2 for "scratch" temp tables Nontraditional data Nonrelational data Legacy data 38

39 Example: SELECT What the application sees SELECT cust_nation, sum(o_totalprice) FROM Customer, Orders WHERE c_custkey = o_custkey and o_orderstatus = 'OPEN and c_mktsegment = 'BUILDING GROUP BY cust_nation SELECT o_custkey FROM Orders WHERE o_orderstatus = 'OPEN What FS does: 1. Join rows from both sources 2. Sort them by cust_nation 3. Sum up total order price for each nation 4. Return result to application SELECT c_custkey, cust_nation FROM Customer WHERE c_mktsegment = 'BUILDING Orders Customer 39

40 Federated Cost-Based Query Optimization Physical Properties: Federated system configuration Query Properties: Optimization class, data distribution, operators used, query type, cost models, FIRST N ROWS? Statistics: Table Statistics Column statistics Index statistics SERVER Characteristics Cpu/io ratio, Commrate Capabilities Type/version Non-Relational Wrapper Wrapper Plans Cost Models 40

41 Federated Database System (Global) Catalog Contains information both about local objects and remote objects Global because it contains information about all the objects in the federated database Table information is found in the following SYSCAT tables SYSCAT.TABLES SYSCAT.NICKNAMES SYSCAT.TABOPTIONS SYSCAT.INDEXES SYSCAT.INDEXOPTIONS SYSCAT.COLUMNS SYSCAT.COLOPTIONS Global catalog also contains other information about remote sources including: Connection, authorizations, etc. 41

42 Global Catalog information Data type mappings describe the relationship between the data source data type and the FS data type Can override defaults by altering local nickname column types if appropriate Function mappings tell FS that a remote function is semantically equivalent to a local function (need: compatible arguments + types) Increases the opportunity for pushing down the function to the data source Without a valid mapping, data has to be retrieved and function applied at FS Can also tell FS about remote functions that have no local equivalent using function templates Statistics are used by DBMS s to describe the logical and physical structure of the data Helps the optimizer generate optimal access strategies FS retrieves statistics from remote-source catalog and populates DB2 catalog at CREATE NICKNAME time There are no actual "local indexes" on nicknames. Information on remote indexes is kept in the FS catalog Normally, information about remote indexes is picked up during nickname creation (including index specification and statistics) 42

43 Actual pushdown is cost-based Just because processing can be pushed down doesn't mean it will be. Decision influenced by estimates of rows processed/returned. Consider a join of two nicknames ORA.T1 and ORA.T2 on a single remote source that is "nearly" a Cartesian product. May be better to do the join at the InfoSphere Federation Server to avoid retrieval of many rows. Retrieving (10, ) rows to do a local join is probably faster than retrieving (10,000 * 25) = 250,000 row remote join result SELECT... from ORA.T1, ORA.T2 where T1.a = T2.b ORA.T1 ORA.T2 25 rows 10,000 rows Single remote Oracle source 43

44 'Pushdown' of Query Operations FS decides whether some or all parts of a query can be "pushed-down", i.e. processed at the remote data source(s). Pushdown-ability depends on Availability of needed functionality at remote source Server options (example: is collating sequence at FS and remote source the same?) Typically faster than processing the query at FS because of less data movement from the data source to FS Example: A remote source that can handle an equality predicate, but not count(*)... Compensation SELECT count(*) FROM t1 WHERE col = 27 SELECT count(*) FROM... SELECT '1' FROM t1 ication Federation Server non-db2 data 44

45 Sort Order Varies in some cases for different collating sequences Data consists of combinations of letters and numeric characters Data contains both uppercase and lowercase letters Data contains special characters, e.g. # Affects how data is sorted in a query with an ORDER BY Affects how character comparisons are made E.g., SELECT WHERE Column3 > Aa3@ Two data source of the same type (wrapper) can use different collating sequences E.g., in DB2 the collating sequence is specified when the database is created Different databases can use difference collating sequences The collating sequence of the data source can be specified when the Server is defined 45

46 Server options: Collating Sequence Differences EBCDIC Sequence... ab yz... AB YZ ASCII Sequence AB YZ... ab yz... LEXICAL Sequence AaBb YyZz... 46

47 Server options: Collating Sequence Differences ORDER BY COLM2 Different order WHERE COLM2 > TT3 Different results EBCDIC ASCII/LEXICAL EBCDIC ASCII/LEXICAL COLM2 V1G Y2W 7AB COLM2 7AB V1G Y2W COLM2 TW4 X72 39G COLM2 TW4 X72 47

48 Server Options: Case Insensitive Collating Sequences...WHERE NAME = MARIANGELA' Assume that the data source column contains: MariAngela TRUE Databases using the Insensitive Collate option (an optional parameter for MS.SQL Server, Sybase and Informix) FALSE Databases not using the Insensitive Collate option 48

49 Server Options: COLLATING_SEQUENCE option You must inform Federation about Data Source collating COLLATING_SEQUENCE= 'Y' indicates that FS and the remote data source sort the same all char sort and comparison operations can be pushed down COLLATING_SEQUENCE = 'N' (Es. DB2/390) indicates that FS and the remote data source sort differently char sort and most char comparison operations can not be pushed down only char = comparisons can be pushed down COLLATING_SEQUENCE= 'I' (Es. SQL Server) indicates that the remote data source uses insensitive collating sequence no char sort or char comparison operations can be pushed down Set COLLATING_SEQUENCE as a Server Option or on a nickname as the NUMERIC_STRINGS Column Option 49

50 Server Options: VARCHAR comparison semantics Column FIRSTNAME is VARCHAR(25) Actual contents are MARYb / DB2 (and all other major RDBMSs) Oracle SELECT *... WHERE FIRSTNAME = MARY SELECT *... WHERE FIRSTNAME = MARY TESTS TRUE TESTS FALSE Forces COL1= MARY' to be pushed down as RTRIM(COL1) = MARY' resulting in a relational scan by ORACLE Mitigated with VARCHAR_NO_TRAILING_BLANKS = 'Y' > but know the data! 50

51 More Federation Server Features Ability to define informational constraints over nicknames Ability to refer to and execute remote stored procedures for DB2, Oracle, Sybase, and MSSQL data sources Error Tolerant Nested Table Expression UNION ALL Connection error Remote 1 + Remote 3 Remote 1 Remote 2 Remote 3 51

52 WebSphere Federation Server w/ F2PC Update Money Transfer Example 1)Connect 2)Withdraw 3)Commit SQL Server Client CHECKING_ACCOUNT 4)Connect 5)Deposit 6)Commit Oracle SAVING_ACCOUNT 52

53 WebSphere Federation Server w/ F2PC Update Money Transfer Example 2)Withdraw 4)PREPARE 5)COMMIT SQL Server Client 1)Connect InfoSphere Federation Server CHECKING_ACCOUNT I am the TM_DATABASE 3)Deposit 4)PREPARE 5)COMMIT Oracle SAVING_ACCOUNT 53