Information Management System z 2008 Information Management System z Louis Holla Information Management zseries Software
Agenda Information Management 2008 Why Data Warehousing on System z Data Warehousing Market Directions Target Segments for Data Warehousing on System z What exists today for Data Warehousing on System z What is coming for Data Warehousing on System z 2
Understanding the terminology Business Intelligence (BI) and Data Warehousing (DW) are sometimes used interchangeably Typically BI includes end user tools for query, reporting, analysis, dashboarding etc. Both concepts depend on each other BI almost always assumes a Warehouse (WH), Operational Data Store (ODS) or Data Mart (DM) exists with timely, trusted information A DW depends on end user tools that turn data into information. Both terms (DW and BI) address desire for timely, accurate, available data delivered when, where and how the end users want it 3
What is Data Warehousing? Data Warehousing provides the underlying data storage facilities required to support any BI Solution. An Operational Data Store (ODS) is a subject oriented database organized by business area. It is up to date (vs. historical) and detailed (vs. summarized). A Data Warehouse (DW or sometimes WH) is a multi-subject oriented database populated from operational systems and/or ODS s. It is historical (vs. point-in-time) in nature and typically contains detailed data. It is often looked upon as the single source of corporate truth. A Data Mart (DM) is a database designed to support the analysis of a particular business subject area. Data has usually been transformed and aggregated from the source DW or operational system. Data Marts can be relational, multidimensional (OLAP) or statistical in nature. 4
How customers use WH/BI to manage their business better Financial intelligence and business performance management The ability to provide accurate information to shareholders and regulators The ability to monitor and correct business performance as quickly as possible Marketing intelligence Insights about their customers A single view of their customers Market and competitive intelligence Information on the profitability of their customers Operations intelligence The ability to monitor business activities Insights into their suppliers A single view of their suppliers Insights on their production processes that enable greater efficiencies Human resources intelligence Insights into their workforce IT intelligence The ability to analyze information systems and performance 5
Why is Warehousing important to IBM and to you Warehousing/Business Intelligence/Analytics is a high growth (6-8%), lightly penetrated opportunity Estimated 85 percent of users who need some form of reporting/analytic capability do NOT have it yet GTS survey results: Information Management is largest and fastest growing middleware services area Business Intelligence Services was #2 priority for surveyed customers (Information Management was #1) 35% rated highly important/top budget item A top priority for Data Services Segment Grow share among SAP clients Grow volume of Dynamic Warehousing solutions Extend Data Warehousing value to Information as a Service Grow Dynamic Warehousing reach and share via channel partners A top priority for STG: Focus on new workloads Business Intelligence Mission critical enterprise management, Business integration 6
Why Talk about Data Warehousing on z/os in 2007 System z customers are asking for increasing support and capabilities for DWH on z Protects their investment in System z Keep up or lead in BI and DWH technologies Significant amount of enterprise data is on System z Many z customers have a DWH on DB2 z/os (?) System z can capture DW consolidation and modernization opportunities Demand for real-time access to operational data for BI analytics is growing The BI operational needs for reliability, availability and security are growing The z/os platform is a secured system second to none. Significant DW capabilities have been added to DB2 V8 with even more DB2 9 The new ziip processor provides TCO and IT optimization Particularly useful for BI queries (star schema) and parallel query processing Newly acquired data integration and analytic components combined with partner capabilities make best of breed integrated solution. 7
A Single View of the Enterprise Drives the requirement for Data Warehousing Data volumes rise Amount of historical Information grows Number of users grows Questions are more numerous and sophisticated Schema complexity grows Workload mixture changes Development time shrinks 8
DW Compliance System z Supports Compliance Single version of the truth Data confined to one location regardless of where the application is running Single process to manage compliance Fewer people to manage compliance process DB2 Audit Management Expert Tivoli Compliance Insight Manager Single place for auditors to look for both operational and warehouse data Industry specific data models assist DW compliance 9
Factors that Influence Data Warehouse on System z Platform Selection Existing infrastructure Platform preferences and skills in-house Budget/Cost Server Consolidation Market trends in warehousing solutions and vendors Complexity Competitive influences Mergers and acquisitions or consolidations Data or application sources Business processes required and business drivers Reliability, Extremely High Availability, Security 10
When is System z the Preferred Platform? Need true real-time Operational Data Store (ODS) Operational data is on System z Data must virtually be in sync with the operational data Availability, Security and Resiliency needs are high Meet auditable data warehouse requirements Specialty engine allows for IT optimization ziips, zaaps, IFLs. Want to leverage & optimize existing z skills and investment Can balance workload based on business policies/needs (WLM), separate OLTP and DW workloads Consolidation of distributed marts or DW to an existing System z Data Serving platform And when NOT.. 11
Commercial Wednesday April 23th, 2008: DB2 9 for zos and DB2 tools Seminar DB2 9 for zos and tools workshop IBM - Forum Bourgetlaan, 42 B-1130 Brussel This one-day free seminar provides a unique opportunity for technical professionals, Phone : +32 2 655 54 40 architects and managers to gain insight into DB2 9 for z/os and optimizing its use by getting the latest information on the IBM DB2 tools. We also provide you with the latest industry trends and hot topics like Information on demand, Information integration and Data governance on System Z. You'll hear a brief overview of the IBM's Information as a Service initiative, learn about the latest IBM acquisitions (Princeton Softech, Datamirror and Cognos) and see some upcoming releases of key DB2 tools (DB2 9 for z/os Value Unit Edition, IBM Data Studio, IBM Information Server for System Z, DB2 Optimization Expert, DB2 Optim solution, IBM Dataquant, DB2 Audit Management Expert, DB2 Recovery Expert, DB2 Cloning tool, DB2 Accessories Suite). You can register on : https://www-926.ibm.com/events/wwe/swgp/swgpbe.nsf/agenda?openform&seminar=kwo98djnwes&lang=be 12
Market Directions 13
Dynamic Warehousing A New Approach to Leveraging Information Information On Demand to Optimize Real-Time Processes Dynamic Warehousing OLAP & Data Mining to Understand Why and Recommend Future Action Traditional Data Warehousing Query & Reporting to Understand What Happened 14
Warehousing strategic pillars Guiding principles for innovation Simplicity Easy to deploy and integrate Easy to use Easy to manage Easy to start and grow as needed Reliability & Performance Reliable access to information Highly available Real-time performance Maximized resource efficiency Extended Insight Beyond traditional capabilities Further leverage information Extended business insight Support broader usage 15
IBM DB2 Warehouse software A complete, integrated platform Embedded analytics Modeling and design Data mining and visualization Data partitioning Performance optimization Workload control In-line analytics Deep compression Data movement and transformation Administration and control Database management IBM DB2 Warehouse 16
Recent IBM Investments Further Contributing to Adoption of System z for New Workloads [WIP] Application-Driven Projects Web 2.0 apps enabled using WS Portal Workload consolidation benefits through common infrastructures (e.g., WebSphere Application Server) Rapid time to market & low business risk with CICS Service Flow Feature & WebSphere Adapters Ability to integrate existing & new assets via WebSphere MQ messaging backbone and federated ESBs Faster business innovation built through WebSphere Process Server Information-Driven Projects DB2 VUE provides cost effective solution for new workload implementations through an OTC pricing model DB2 Warehouse offering for critical performance for dynamic warehousing and OLTP requirements Information Server for System z enables System z to be the trusted information hub DB2 9 optimized for SAP IMS 10 for critical on-line operational app s Content Manager v8.4 and Content Manager On Demand v8.4 support a strategic ECM platform Application Development and Delivery Infrastructure Incorporating System z Asset Management Quality Management Architecture Management Change and Release Management Skills Management Enterprise IT Management Infrastructure Integrating System z Interdependent Applications Linux and SOA Composite Applications Distributed Environments Core Applications Financial and Service Federated Identity Workload and Availability Storage 17
The Information Challenge Business Challenges Globalization M&As Supply chain INFORMATION IS A Information Challenges Accuracy Timeliness Relevance Risk & compliance STRATEGIC Accessibility Customer loyalty Security ASSET Operational costs... Business velocity... 5X more value creation by 60%+ of CEOs: Need to do People can spend up to 70% organizations using a better job leveraging of their time looking for information effectively information information 18 Sources: IBM Attributes & Capabilities Study, 2005; Client Interviews 2004; IBM CFO Study, 2006
IT Projects Driving New Business Initiatives Dynamic Data Warehousing Co-located OLTP & DW Move to System z decreased reporting times 80% Addressing Key Customer Requirements Workload consolidation Integrating data via SOA Reduced total cost of ownership Next generation warehousing People, Processes, Applications Trusted Information In Context Master Data Management consolidated customer profile to enable customer centric processing for sales and service SOA Data Serving Exposed legacy IMS through Web services Guaranteed integrity on over 4 billion IMS transactions per month SAP Information Infrastructure Significant IT savings Improved customer service Business growth Data Governance & Accountability Automated archiving to manage continuous data growth 19
Information On Demand and SOA Two sides of the same coin SOA provides business flexibility IOD provides the trusted information needed by your services You will waste your investment in SOA unless you have enterprise information that SOA can exploit. Gartner Research 20
Gartner perspective on BI and DWH market System z holds a viable place in the Leaders quadrant The emergence of issues based on a mix of 4 DW workloads Continuous (near-real-time) data loading similar to an OLTP workload Large numbers of standard reports An increasing number of true ad hoc query users An increasing level of analytics and BI-oriented functionality in OLTP The mixed workload performance will become the single most important performance issue in DW The transactional DBMSs have an edge that challenges the DW DBMSs (such as Teradata) to stay ahead 21
IBM System z Data Servers Trusted Platform for SOA IMS 10 XQuery and Web services enhancements ease data service delivery DB2 9 purexml for seamless and efficient integration of XML & relational data Industry s Highest Level of Availability & Scalability Unmatched availability for utility class information access Rapid and non-disruptive scalability for variable mission critical requirements Unmatched Risk and Compliance Management New security capabilities for greater control and business flexibility Improved auditing and accountability Lowest Operating Cost Reduced operating costs through z/os and ziip engine exploitation Superior environmental efficiency the green machine power, cooling, and space 22
Commercial 23
What Exists Today for Data Warehousing on System z 24
Enterprise Data Warehouse using layered data architecture Floor 3 Floor 4 Floor 5 Dashboard Static reports, Fixed Periods Star Schema, Data Mart, Cubes, Duration: Years Summarized Data Performance and rolled up Data Duration: Years Alphablox, DataQuant, Logical DM using Views for flexibility wherever possible, otherwise Physical DM ELT, MQTs, SQL Function Floor 1 Floor 2 Near 3 rd Normal Form, Subject Area, code and Reference tables. Duration: Years ODSs, Staging, detail, denormalized, Raw Source Duration: 60, 120, 180, etc days Mining, scale, growth, Merge Near RealTime, MQ, LOAD, multi-row insert 25
Solution Architecture Member A OLTP Member B OLTP DB2 Data Sharing Group Member CEC One C DWH CF Member CEC Two D DWH DB2 Data Sharing Group CEC One CF CEC Two Within a data sharing environment, the data warehouse resides in the same group as the transactional data. 26
Initial load of the Data Warehouse Member A OLTP Member C DWH CEC One VSAM, IAM & sequential IMS CF Member B OLTP Member D DWH CEC Two Software AG Adabas CA Datacom ETL is done, by using an ETL Accelerator which Extract in it s current implementation is represented by DataStage (IBM Information Server) on a secondary system. Until end of 2007, pseries Load and xseries are the only supported systems. Later on the ETL Accelerator ETL Accelerator will move to zlinux as soon as the Information pseries Server is xseries available there. zlinux z/os CA IDMS Extract JDBC/ODBC Extract JDBC/ODBC The data is extracted Classic Federation from the Server OLTP information Distributed Legacy of Distributed the data Data sources Sourcesare - Integration - sharing group, transformed by DataStage and are integrated directly then loaded integrated through into the the Classic data warehouse tables again. through Federation the IBM Server. This way data Information is extracted server. from DataStage directly DB2 UDBout Oracle of the SQL data Server sources for LUW like IMS, VSAM DataStage Parallel Engine 27
At runtime, the Data Warehouse is updated incrementally Member A OLTP Member B OLTP Change Capture Member C DWH CEC One CF Member D DWH CEC Two Update & Insert MQ MQ ETL Accelerator pseries xseries zlinux Data Event Publisher MQ Data Event Publisher DB2 IMS Software AG Adabas CA- IDMS VSAM Capture z/os Capture Distributed DB2 UDB for LUW Oracle SQL Server 28
In Database ELT is triggered by DataStage Member A Member B OLTP OLTP ELT Member C DWH Member D DWH SQL ETL Accelerator CEC One CF CEC Two pseries xseries zlinux Simple example: Wherever possible, In Database transformations (ELT) are used to spare the transport of the data to the accelerator. But the used SQL is still sent from the ETL Accelerator to the database to have one place of -- Aggregate by salary by department into AGGRSALARY INSERT INTO AGGRSALARY ( DEPTCODE, AVGBAND, AVGSALARY ) documentation for all ETL steps. SELECT DEPTCODE, AVG( BAND ) AS AVGBAND, AVG( SALARY ) AS AVGSALARY FROM STAFF GROUP BY DEPTCODE This can also be used to shift the data up the hierarchy within the Layered Data Architecture. 29
What DW Capabilities Exist Today for System z DB2 z/os V8 & V9 DW Base Functional and performance enhancements Easier online reporting and data management capabilities DB2 family compatibility DW ETL WS DataStage EE, WS Classic Federation Server, WS Data Event Publisher, WS Classic Event Publishers, Distributed DBMS Event Publishers, BatchPipes for OS/390, DB2 Unload/Load Utilities, DataMirror Transformation Server Analyze/Report From IBM: Alphablox, QMF and DataQuant, Cognos From partners: Hyperion, Business Objects, SAS, IBI Performance Management IBM Tivoli Omegamon XE for DB2 Performance Expert on z/os, DB2 Query Monitor, Optimization Service Center (V9) Other IBM Information Server Components WS Information Analyzer, WS QualityStage 30
DB2 for z/os Enhancements for Data Warehousing DB2 Version 8 Usability, Availability & Scalability Online Partitioning Changes, More Partitions, Rotate Schema Evolution: VSCR with System z 64-bit Architecture Data Warehousing Support Star Join Improvements Materialized Query Tables Overall System & Query Performance Locking Improvements Multi-row INSERT & FETCH More index Access for VARCHAR, data type mismatch, DDF Performance Improvements ziip for improved cost of ownership DB2 Version 9 Usability, Availability & Scalability Universal & Partition by Growth Table Space Clone Tables (Online Load Replace) XML Data Warehousing Support Star Join Improvements (Dynamic Index ANDing) SQL Enhancements (RANK, DENSE_RANK, Text, XML, INTERSECT, ) Overall System & Query Performance TRUNCATE Statement for fast delete, MERGE for insert or update Index Compression Significant CPU Reduction for DB2 Utilities & many queries Insert and update scaling 31
DB2 9 for z/os Update: Rapid Adoption Already! Simplification, Reduced TCO Index Compression, Partition By Growth & Cloned tables Workload Consolidation Trusted context and ROLES, Online REBUILD INDEX SOA Enablement purexml, Optimistic locking for WebSphere Dynamic Warehousing Optimization Service Center, New built-n OLAP expressions 32
Specialty engines work together z/os LPAR WAS JAVA DW Application zaap Data Warehouse Application not on System z TCP/IP HiperSockets TCP/IP D R D A DRDA DB2 DRDA Star Schema // query ziip TCP/IP HiperSockets Linux on System z LPAR DW Application z/os LPAR WAS Application zaap IFL DW Application CP The IBM System z9 specialty engines can run independently or complement each other (shown at left, instances where specialty engines can be employed) 33
System Z Green Credentials Environmental Issues are becoming a concern Power Proliferation of servers increase demand on the external infrastructure Cooling Heat generated by increase number of servers has to be considered Space Will the expanding number of servers fit into the available space 34
Commercial Customer visit of Guogen (Gene) Zhang, Database Technology Institute for z/os, IBM Silicon Valley Lab 35
What is DataQuant DataQuant provides a comprehensive query, reporting and data visualization platform for both web and workstation-based environments. DataQuant introduces a wide variety of powerful business intelligence capabilities, from executive dashboards and interactive visual applications to information-rich graphical reports and ad-hoc querying and data analysis. DataQuant provides 2 components DataQuant for Workstation An Eclipse-based environment for the development of query, report and dashboard solutions DataQuant for WebSphere A runtime environment capable of displaying DataQuant content using a thin client model 36
IBM DATA QUANT FOR Z/OS Adds compelling new Warehouse/Business Intelligence component to WH on z Visual Dashboards, Enhanced Graphical Reporting, Security and Personalization, SOA Layer, Enhanced Analytics Offers a thick client with DataQuant for Workstation, or a pure HTML, browser based client with DataQuant for WebSphere 37
But wait There s more 38
39
Cognos 8 Information BI Information Management 2008 Management & IBM Software Cognos 8 BI Portal Ad Hoc Query Deep Comparative Analysis Event Lifecycle Management Dashboarding & Scorecarding All Administration FIREWALL/ENCRYPTION ROUTER/FIREWALL WEB SERVERS: Web Gateways, Web Services API Industry Standard Portals Cognos Application Firewall Cognos Office Connection Business Analysis Business Metrics Business Performance Management Design Implementation Optimization Outsourcing Insurance Industry HTTP Server WebSphere Portal Workplace App. Server Studio QUERY SERVICE JOB SCHEDULING SERVICE PRESENT- ATION SERVICE CONTENT MANAGER SERVICE AUDIT SERVICE METRICS SERVICE EVENT SERVICE MONITORING SERVICE Development Tools email Distribution System Mgmt Job Scheduling System x (Win/Linux) System p (AIX/Linux) FIREWALL/ENCRYPTION SQL MDX COMMON METADATA 40 All RDBMS & OLAP sources WebSphere Information Integrator (Classic, Content, UIMA) Reporting & Analysis Data Warehouse WebSphere Data/MetaStage DB2 CubeViews Rose CONTENT & METRICS STORE DB2 UDB SECURITY PROVIDERS LDAP, Active Directory, NTLM, Netegrity, SAPBW, Cognos Namespace, Custom Providers Directory Server Access Manager AUDIT RDBMS, UNIX, Sys, Logs, NT Event Log System Monitoring DB2 UDB
41
IBM Cognos 8 Business Intelligence (BI) for System z This will be an entirely new offering for the System z platform with a target availability of the second half of 2008. Cognos 8 BI for System z combines the targeted information delivery and analysis capabilities of Cognos 8 BI with the power and reliability of System z to: Enable organizations to report and analyze hundred of millions of transactions directly on the mainframe. Ensure everyone across the organization can quickly identify and respond to critical business trends. Provide a competitive advantage through improved decision making from business data. 42
Steve Mink DB2 9 for z/os Value Unit Edition (VUE) New Workload Offering available OTC Drive Net New Workload Growth in new and existing DB2 for z/os customers by Offering customers DB2 9 as OTC for qualified net new workloads on z/os Net new workloads limited to commercial (packaged) applications, data warehouses, and WebSphere. Applications running on a measurable standalone machine or LPAR. Customers can now add new applications and workload to System z without affecting their MLC cost. Strengthens System z role as a cornerstone for new applications and projects such as SOA, data warehousing, operational BI, ERP, and more 43
IBM Information Server for System z Benefits of this architecture Greater Efficiency Leveraging System z z/os MIPs consumption dramatically reduced vs. USS or MVS approaches Minimizes impact on other z/os software costs All Job Processing is on zlinux (except the z/os data access) DB2 workload on z/os can qualify for offload to ZIIP specialty engines High performance z data connectivity Batch Pipes for DB2 load, DRDA to DB2 over hipersockets SQL to Classic over hipersockets Integration with MQ and therefore with the Data Event Publishers Seamless integration with other IBM Information Server platforms Same operational architecture and metadata Repository Eliminates deployment issues Maintains value of DataStage for z/os investments 44
What else is coming for DW on z? SHORT TERM DB2 9 Even more DB2 compatibility Warehouse, reporting & optimizer enhancements BCU Methodology (Sizing Tool, Best Practices) InfoSphere Master Data Management Server for Linux on System z MEDIUM TERM 1 to 1 1/2 YEARS Shark Large table scans pushed down to Shark Better price/performance Additional Support for DW on z MQT Advisor Cube Views support Common Web-based Admin Console Alphablox on Linux on z LONGER TERM Additional capabilities supporting DW on z Data Mining DWE Design Studio IDE 45
DW on System z Support Initiatives BI Sizing Tool for System z Target availability via Techline early 2H07 Input: Txn rates for 5 different txn profiles & DW raw data size Output: CPU requirements (including ziip estimate) plus guidelines for memory and storage DW on System z Best Practices Guidance Target V1 end of 2Q07 (will be track on IOD 2008) Recommended Architecture & Components DB2 Configuration & Tuning Guidance 46
47
48
And drive carefully out there... 49