DB2 Information Integrator

Transcription

1 IBM Software Group Daniel Ling IBM Software Solution 2002 IBM Corporation

2 Today s World: Complex and Costly Heterogeneous, distributed data Applications create and maintain caches and replicas of data Proliferation of copies 30%- 50% design expenses go to copy management No guaranteed quality of service nor feedback App. App. App. App. Invalidate Cache Cache Refresh Cache Replicas Copies Sysplex Data 2

3 Enabling our Client s Information Infrastructure integrating diverse sources of information to provide new insight, greater efficiency and improve their competitive advantage Transaction systems Business intelligence Content management partner-enabled solutions Information Infrastructure Search and Access Information Services Industry Specific Extensions Analysis Information Integration Content Control higher value information services Databases and Files Information Assets Data Warehouses Content Repositories information assets 3

4 What is Information Integration? A category of middleware which lets applications access diverse and distributed enterprise data as if it were a single source, whether or not it is It enables the integration of structured and unstructured data to provide real-time read and write access, to transform data for business analysis and data interchange, and to manage data placement for performance, currency, and availability. 4

5 Information Management Vision Any data -- Multiple access paradigms -- Multiple integration disciplines Data virtualization App. App. App. App. SQL XQuery Content Search Request & QoS Find Federate Transform Place Placement Manager Data Virtualization Shared metadata and access foundation Publish Data Policy Response & Delivered QoS Databases and Files Information Assets Data Warehouses Content Repositories 5

6 Integrating diverse business information across and beyond the enterprise Information Integration Data federation Extensible read/write access across diverse data and content sources Database programming model Content programming model Data placement Caching and replication over heterogeneous information Free-form search ( II OminiFind ) Fast, relevant results Data transformation SQL, XML, Web services Metadata management Part of a complete integration solution XML publishing, consumption, and interchange Business integration Open platform based on industry standards 6

7 IBM DB2 UDB VSAM IMS Software AG Adabas CA-Datacom DB2 II Classic Federation O D B C Integrated SQL View SQL, SQL/XML Federation Engine Wrappers and functions DB2 Family Sybase Informix SQL Server CA-IDMS Oracle Teradata Biological Data and Algorithms Text XML Excel WebSphere MQ IBM Extended Search WWW, , ODBC 7 IBM Confidential 2002 until IBM GA Corporation

8 Data Federation Concepts Global optimizer Leverages knowledge of: Source statistics Source indexes Source functions Server and network speed Locally persisted data: Global catalog Materialized query tables User data Nickname Nickname Nickname Server Server Wrapper Oracle MSSQL Wrapper: a library allowing access to a particular class of data sources or protocols (eg; Net8 for Oracle.). Contains information about data source characteristics Server: represents a specific data source (eg oracle1, oracle2) Nickname: a local alias to data on a remote server (mapped to rows and columns) IBM Confidential until GA 8

9 A Federated Data Server Query distributed data as if it were a single source Define integrated view across diverse and distributed data Wide range of data and content sources Extensible to virtually any data source Query as if a single source Use standard SQL query and SQL expressions Include text semantics in the search Surface specialized functions into SQL Leverage query optimization and caching Compose XML documents Combine diverse sources Validate against DTDs or schema Publish results to a message queue Familiar DB programming model Single source, relational updates Integrated SQL View DB2, Informix, Oracle, SQL Server, Sybase, Teradata, OLE DB, ODBC, Excel, XML, message queues, Web services, flat files, document repositories, content repositories, LDAP directories, WWW, databases, and more. 9

10 II Federation Capabilities Powerful query processing engine in federated server Decomposes, rewrites and distributes queries Cost-based optimizer chooses query plan with pushdown as appropriate Query execution engine drives wrappers, combines results Compensates for missing function in data source Invokes functions at remote sources as needed Incorporate data sources using wrappers High-function relational wrappers from IBM Clean, simple interface for building additional wrappers to access custom data sources Wrapper SDK provides class library (C++ today, Java SDK under development) Written by IBM, third parties, or customers R/O in R1 Extend additional access via connectivity functions (UDFs) Integrated part of object relational architecture Consideration for application requiring OLTP-like throughput Masking complexity though still distributed transaction 10

11 A Replication Server Manage placement for performance and availability Distribute data among relational databases DB2, Informix, Microsoft, Oracle, Sybase, Teradata Support flexible topologies Distribution: One to many Consolidation: Many to one Column and row subsetting Multiple flavors, e.g. history tables, key changes Match data movement modes to usage requirements Table-at-a-time for warehouse loading during batch window Transaction-consistent for online data Choose latency characteristics Scheduled, interval-based, event-driven, continuous Apply transformations in-line Standard SQL expressions or stored procedure execution. Microsoft DB2 Microsoft 11

12 Replication Architecture 12

13 Data Placement: Heterogeneous Caching Improve query performance and availability Administrator defines Materialized Query Table Precomputed or frequently used values Any data from the federated system Application indicates ability to use cache Implicit or explicit use Developer enables cache use If enabled, reads are handled from the cache, writes passed through to the source If not, reads and writes passed through to source Cache refresh managed: Manually By replication 13

14 Bridge to the XML World Information Integration Integration of existing relational with non-relational data Federated access to existing XML documents and join with other data sources Compose XML documents from federated data Validate XML documents against schema or DTD Store documents intact or shredded into relational tables Support new SQL standard for publishing XML documents XML world 14

15 Plug into a Service-Oriented Architecture All the power of the EII software through a Web Service Any SQL operation Over diverse, distributed data Extend data reach and transformation richness with Web services Integrated SQL statements and Web service invocations Any Web Service becomes a transformation function or data source SQL Query: Invoke single conversion web service select t.supplier, t.part_num, t.qty, to_dollars(currency,price) as dollars from supplier_ops s, table(get_quote (s.supplier,s.url,'abc',20,current date)) t where s.operation = 'GET_QUOTE'; Invoke many quote web services 15

16 Increase Messaging System Value Simplify integration between database and messaging systems Brings message queue access into a familiar paradigm for database programmers Analyze message queue data with standard analytical software Virtual or physical queue snapshots can be analyzed using standard SQL-based analytical tools Insert into PENDING_ORDERS select t.msg from table(mqreceiveall()); Compose and publish XML messages A single query can access diverse and distributed data, compose a complex document, validate the document against a DTD or XML schema, and publish it to a message queue 16

17 User Defined Functions Extend Connectivity Provide additional logic which is invoked as part of SQL processing Can return either scalar, row, or table results Can be used to compose standard views Simple to develop and configure Can exploit parallelism Built-in UDFs for WebSphere MQ, OLE DB, Web Services SELECT MQSend(followup.service,a.custid ' ' a.ordid) FROM account a WHERE a.status = 'overdue' 17

18 Classic Federation SQL-based read and write access to mainframe data sources Metadata-driven for easy configuration and maintenance No mainframe programming required Multi-threaded with native drivers for scalable performance Designed for your existing mainframe, tools, and application infrastructure Classic Federation for z/os DB2 UDB for z/os VSAM IMS Software AG Adabas CA Datacom CA IDMS 18

19 Emerging Content and Data Integration Requirements Regulatory compliance combining loan applications, credit rating, loan default records, etc. to assess risk and justify actions Law enforcement combing diverse information (criminal records, immigration records, travel records, DNA records ) to assist criminal investigations Pharmaceutical research research collaboration across biological and chemical research domains IBM VeniceBridge Claims adjustment combining current claim forms and images with customer account detail and historical trends re: the claimant, the automobile, etc. Finance analyzing news feeds and other unstructured data sources for specific kinds of information about companies that affect stock values. Call Center analyzing call center and other problem records for early detection of problem trends Documentum, FileNet Content Services, FileNet Image Services, FileNet Report Manager, FileNet eprocess, FileNet P8 BPM, FileNet P8 Content Mgr, Google, IBM DB2 Content Manager, IBM DB2 CM OnDemand, IBM ImagePlus, IBM WebSphere MQ Workflow, Interwoven TeamSite, Lotus Domino.Doc, Lotus Notes, Microsoft Index Server, Microsoft Site Server, Open Text Livelink, Stellent, Verity K2, UNIX and Windows File Systems 19

20 Customer Scenario : Extending Business Integration Customer Data Integration Feed the ODS from diverse data sources Augment customer views with additional attributes that are seldom needed, must-be-current, or very large Business Performance Management Access and integrate KPIs, planned targets, historical trends, and current values Application Integration for Data Consistency Replicate data between application databases Support consolidation and distribution between HQ and branch/store locations Portal Development Access disparate sources through unified views for Web deployment 20

21 Taikang Life Insurance Uniting the business with Information Integration Business Challenge Provide a clear picture of customer s and associated services Capitalize on new business and growth opportunities with an up-to-the minute view of the organization Solution Single data architecture providing a unified federated view of corporate data across the company Integrate information from DB2 UDB, Informix, Oracle, SQL Server, XML, , CRM and Portal applications Key software used: DB2 Information Integrator, WebSphere MQ, IBM iseries Server Business Benefits: Comprehensive status reports delivered to corporate executives in real-time Improved strategic decision making and business analysis Increased employee productivity Technology Benefits: Faster development of new applications Enhanced customer service and business development opportunities 21

22 Taikang Information Platform Prior to SQL Server Client Data Client Data Client Data SQL Server SQL Server Channels not effectively integrated, client data dispersed and not effectively shared Multiple application systems, multiple application development tools Client Data DB2/400 Client Data Informix Client Data Oracle 22

23 Taikang Integrated Information Platform Architecture Channels Application Platform Data Service XML SQL Web Services Information Integration Platform Integrated Information Mapping (nicknames) cache ODS Core Systems Informix DB2/400 Oracle 23

24 Crystal Decisions Vision As a world-leading information infrastructure company, Crystal Decisions helps businesses make better decisions by bringing together their people and their information. Challenge Improve response time for complex queries over distributed heterogeneous data sources. Solution Provides transparent, globally optimized access to heterogeneous, distributed data. Crystal Reports accesses the distributed data as if it were a single database. Response time improvement of up to 98% seen in house. Business Value "Users of Crystal Reports and Crystal Enterprise, with DB2 Information Integrator, can discover new ways to meet the information needs of their organization." Janet Wood, Vice President of Business Development, Crystal Decisions. Competitive Value provides Crystal Reports with exceptionally fast and efficient federated querying capability. Trevor Smith, Program Manager, Business Development Group, Crystal Decisions 24

25 Ministry of Railways (China) Vision Ministry of railways manages the Chinese the national railroad system and is responsible for defining and implementing railway transportation plans Want to provide an organization wide intelligence platform to drive decision making that will increase freight and passenger rail travel Challenge Data needed for effective decision-making is spread across multiple databases Begin solution rollout with limited skill and resource Solution Give business managers access to summarized historical data in a DB2 based warehouse with federated access to real-time freight transportation operational data in DB2 and Oracle and passenger data in Sybase and SQL Server. Value Improve the quality of railway transportation management Respond more quickly to market demand by improving freight forecasting Optimize carrying resource assignment "With its real time federation capability significantly enhances the power of the data warehouse supporting our intelligent transportation management systems and other enterprise applications. The XML and unstructured data support of this product will help us fulfill the reporting requirements to the states administrative bureaus." Mr. Sun Yuan-Yun, Chief Engineer, Ministry of Railways, People s Republic of China 25

26 Customer Scenario : Data Base Migration of a local customer App 1 App 2 App 3 App 4 App 5 Database A Database B Database C Database D Causes : Business Merging & Acquisition, Database Standardization and migration Offer : Phase by phase approach of seamless migration transparence to the end users Previously, there were only the big band approach for database migration 26

27 Proposed Migration Infrastructure by End of Phase 1 NewApp 1 NewApp 5 App 1 App 2 App 3 App 4 App 5 II Federated Databases Database A Database B Database C Database D New applications developed using the new DB interface New applications access legacy database through Information Integrator Parallel run (i.e., App1 and NewApp1) is possible New application can be deployed one at a time (no big-bang migration required) Old applications can be phased out gradually after users adopted new applications Reduced risk factor 27

28 Proposed Migration Infrastructure by Phase 2 NewApp 1 NewApp 2 NewApp 3 NewApp 4 NewApp 5 Information Integrator Federated Databases Database A Database B Database C Database D Old applications phased out All applications use II interface Data migration phase to migrate legacy data to new database 28

29 Quad Graphics Vision It is Quad Graphics mission to be the very best at putting ink on paper based on achieving the best fit with client needs Challenge Incrementally migrate from legacy Sybase infrastructure to DB2 infrastructure Solution Provide transparent access to Sybase and DB2 databases for migrating applications using DB2 Information Integrator Value Enable incremental application migration Works seamlessly with established PowerBuilder tools Anticipated strategic value in accessing heterogeneous data environment 29

30 IBM Competitive Advantage More open: Offer more choices than Oracle and Microsoft It is also worth noting a major difference in approach that differentiates IBM from Oracle. IBM takes the view that it should support the ability to integrate and federate information rather than centralize it. This is in direct contrast with Oracle, which is pushing centralization. Bloor Research, Databases, an evaluation and comparison, Jan 2002 More scalable: 3 rd generation query optimization; Integrated caching Building on existing strengths, IBM is readying its EII offering for 1Q03 to counter BEA's Liquid Data (OEMed Enosys). We believe IBM's strong optimization will persuade technical buyers. Meta Group, Dec 2002 Rising Demand for Distributed Data on Demand More function: Structured, semi-structured, unstructured data, XML, web services IBM will raise the bar on enterprise information integration. It has so much more to offer than small companies, and it has broadened the category to include unstructured data. More deployable Leverages wide-spread skills and tools infrastructure Philip Russom, Giga Information Group, Jan 2003 CRN, IBM: Xperanto Rollout to Start In Early

31 Customer Value Choose your data access strategy to optimize IT business value Centralize information for availability or performance needs On-demand, distributed access to data that must remain in place Integrate diverse and distributed data, without moving the data, or changing the platforms Access diverse and distributed data as though it were a single source, no matter where it resides. Gain greater return on existing assets Make more progress, more quickly, and at lower cost Increase developer productivity for integrating diverse, distributed, and real-time data Deploy current skills over a greater range of project requirements, Complete integration projects faster Reduce storage costs Reduce ongoing maintenance costs 31

32 Customer Value Rely on proven IBM technology Trust 25 years of data management research and development for a scalable infrastructure. Integrates with complementary WebSphere integration technologies for a complete business integration infrastructure Protect your current and future IT investments Reduce the need to modify and replace systems to make them work together. Base investments on industry standards Create a strategic and open information integration platform. 32

33 IBM Software Group Thanks You 2002 IBM Corporation

34 Business Drivers for Integration Grow the business Integrate cross-divisional sales information with demographics to mine buying patterns and increase revenues Accelerate time to market Leverage more comprehensive and timely business analysis Deploy solutions more quickly with scalable infrastructure Automate business processes Enhance business performance management integrating real-time and historical data Deliver self service portals for client access to account and market information Adapt to new business or organizational models Reduce the risk and cost of executing mergers and acquisitions through effective integration of incompatible systems and infrastructures Forge links for better collaboration Integrate cross-enterprise manufacturing availability and shipping capability with order processing for fastest delivery 34

35 - Federation Capabilities Transparency: hides differences among sources Appears to be one source Supports a high level query language Functional compensation and passthru High Function One query integrates data from multiple sources Capabilities of sources as well Performance Optimization of distributed queries Extensibility Development wrapper toolkit Access wide range of data sources Heterogeneity: integrates data from diverse sources Relational, XML, flat files, spreadsheet, messages, content repositories, Web, Autonomy Non-disruptive to data sources, existing applications, systems. Functions 35

36 Complementary Information Integration Approaches Consolidate data for local access Data warehouses Operational data stores Production applications Best performance and availability Typically managed by ETL (Extract/Transform/Load) or replication technologies Integrated access to distributed sources Real time data, e.g., stock quotes Mixed format data, e.g., combining ODS with related contract documents in separate repository No latency and standard access interface Represents an emerging category of technology Enterprise Information Integration (EII) 36

37 Wrappers: Integration Point for Data Sources Connection and transaction management Query Planning Provide optimizer with cost and cardinality information Provide II with hints about data source capabilities Pushes down as much work to data source as possible Allows II to compensate for missing function Data modeling Map data model to relational data model (tables with rows and columns) Map functions into SQL operations Query Execution and data retrieval Execute parts of a user s query for a specific data source 37

38 Application Connectivity Scenarios Data event notification Messages representing interesting events are automatically generated by DB2 triggers An inventory management system notifying low inventory A financial institution notifying applications of new research report Information forwarding Changes to important information on DB2 may automatically be forwarded or distributed to downstream systems A banking application inserts a new customer and a message containing the information is forwarded to a CRM application After a batch load of new risk information a financial institution may forward high risk changes to risk management applications 38

39 Application Connectivity Scenarios Information aggregation Messages automatically combined to maintain an accurate view of activities A corporate wide roll-up of financial transactions An insurance company feeding a data warehouse from externally provided data Query integration Developers incorporate messaging operations into SQL queries from any environment A developer wishes to generate a report containing a union of processed and unprocessed sales data 39

40 Cellway Vision Cellway has become one of Germany's leading wireless service providers with almost 3 million customers. Their goal is to win new customers without losing their installed base in the very competitive market Challenge Customer retention is a critical issue in the wireless market and the central pillar of their business model. It depends on achieving a profound understanding of our customers Solution Cellway leveraged DB2 Information Integration technologies to pull together data from several heterogeneous sources into their data warehouse Business Value Call centers, marketing, sales agents are now able to track complete customer information which will boost customer satisfaction and loyalty Competitive Value "By analyzing subscriber data stored in our data warehouse, our marketing department develops targeted sales tactics to improve customer retention. Our IBM information integration solution will greatly improve our ability to manage our relationships with our subscribers. We are a very satisfied IBM customer." Eric Nagel, Data Warehouse Manager, Cellway Kommunikationsdienste GmbH 40

41 Verizon Vision Verizon Communications is one of the world's leading providers of communications services. Their core purpose is to bring the benefits of communications to everybody. Challenge Make their application infrastructure more agile and accelerate project development. Solution Build a virtual data access layer (VDAL) for all information access Value This solution will accelerate application development while reducing overall project cost "We believe VDAL will minimize or eliminate impact of changing or upgrading database products on one or many places in the application code, reduce the cost of application development/maintenance and compress implementation downtimes in some cases to zero. This project will herald a new era in managing and accessing data at Verizon". Ghassan Ghaida Manager Wholesale Systems Data Engineering, Verizon 41