Hype Cycle for Data Warehousing, 2003

Transcription

1 K. Strange, T. Friedman Strategic Analysis Report 30 May 2003 Hype Cycle for Data Warehousing, 2003 Data warehousing concepts and approaches have become fairly mature during a decade of refinement. However, the desire to deliver more-strategic business intelligence continues to fuel technology evolution. Management Summary As data warehousing continues to mature, much of the innovation in this area comes not from new technologies, but rather from ongoing extensions to and the evolution of current technology. Multiple technologies continue to converge with core infrastructure offerings for example, DBMS vendors are increasing their delivery of online analytical processing (OLAP), data mining, and extraction, transformation and loading (ETL) capabilities directly within their database management system (DBMS) products. Other significant trends include a focus on performance and scalability-related innovations to address ever-increasing data volumes and complexity, and the slow, but steady, movement of data warehousing toward more real-time delivery of data for analysis. Gartner Reproduction of this publication in any form without prior written permission is forbidden. The information contained herein has been obtained from sources believed to be reliable. Gartner disclaims all warranties as to the accuracy, completeness or adequacy of such information. Gartner shall have no liability for errors, omissions or inadequacies in the information contained herein or for interpretations thereof. The reader assumes sole responsibility for the selection of these materials to achieve its intended results. The opinions expressed herein are subject to change without notice.

2 30 May

3 CONTENTS 1.0 The Hype Cycle On the Rise Data Warehouse Appliance Federated Data Warehouses Scalable ETL Tools Data Mart Consolidation At the Peak DBMS Mixed-Workload Management DBMS-Enabled Data Mining DBMS Complex Query Optimization Real-Time Data Updates (Merging Operational BI and Data Warehouse) Sliding Into the Trough ERP Data Warehouses Climbing the Slope ETL Embedded in DBMS Metadata Tools DBMS-Enabled OLAP Star Schema Entering the Plateau Virtual Data Warehouse Single-Vendor Solution vs. Multivendor Technology Solution Conclusion...11 Appendix A:Hype Cycle Definitions...12 Appendix B:Acronym Key May

4 FIGURES Figure 1. Hype Cycle for Data Warehousing, May

5 1.0 The Hype Cycle Visibility DBMS Complex Query Optimization Key: Time to Plateau DBMS-Enabled Data Mining DBMS Mixed-Workload Management Real-Time Data Updates (Merging Operational BI and Data Warehouse) Less than two years Twotofiveyears Obsolete before plateau ERP Data Warehouses Virtual Data Warehouse Data Mart Consolidation Federated Data Warehouses Scalable ETL Tools Star Schema DBMS-Enabled OLAP Metadata Tools ETL Embedded in DBMS Single-Vendor Solution vs. Multivendor Technology Solution Data Warehouse Appliance As of May 2003 Technology Trigger Peak of Inflated Expectations Trough of Disillusionment Slope of Enlightenment Plateau of Productivity Maturity Acronym Key BI DBMS ERP ETL business intelligence database management system enterprise resource planning extraction, transformation and loading Source: Gartner Research (May 2003) Figure 1. Hype Cycle for Data Warehousing, On the Rise 2.1 Data Warehouse Appliance Definition: Hardware and DBMS software integrated packages that provide support for business intelligence (BI) applications. Justification for Hype Cycle Position/Adoption Speed: This is not really new, but new vendors are coming on the scene with this old idea. However, they have yet to gain any significant market attention. Business Impact Areas: Has the potential to simplify and reduce the cost of data warehouse implementations by reducing the number of components and the administrative overhead. 30 May

6 Selected Vendors: Netezza. 2.2 Federated Data Warehouses Definition: Similar to the virtual data warehouse, but it also comes in the form of integrated data marts. Time to Plateau/Adoption Speed: Obsolete before Plateau. Justification for Hype Cycle Position/Adoption Speed: Some vendors are giving this attention with marketing and product positioning. Business Impact Areas: Moderate usually as an interim solution to a data mart consolidation strategy. Selected Vendors: Sagent Technology. 2.3 Scalable ETL Tools Definition: ETL tools that provide capabilities to significantly increase the volume of data that can be extracted from source data, integrated across a large number of heterogeneous data sources and loaded to the data warehouse in the required time frame. Justification for Hype Cycle Position/Adoption Speed: A few niche vendors have focused specifically on the scalability of ETL tools. Mainstream vendors are beginning to provide enhancements to capabilities to deliver better scalability and performance. Business Impact Areas: Greatly improve the ability to deal with highly complex data integration and increase the availability of data in support of BI. Selected Vendors: Ab Initio Software and Ascential Software. Analysis by Ted Friedman 2.4 Data Mart Consolidation Definition: The widespread adoption of data marts in the mid-1990s raised the awareness of the requirements for BI applications among enterprises of all sizes and diverse industry sectors. Many enterprises implemented multiple data marts in support of varying BI applications, which resulted in high support and administration costs, and increased the complexity of creating and maintaining the various islands of data. Time to Plateau/Adoption Speed: Less than two years. Justification for Hype Cycle Position/Adoption Speed: Many enterprises are moving to consolidate data marts, but the majority continues to operate with a fragmented BI architecture. Business Impact Areas: High impact. Support for the changing business environment and the need to support agile BI. 30 May

7 3.0 At the Peak 3.1 DBMS Mixed-Workload Management Definition: Multiple users performing a variety of queries, including a mixture of simple and complex requests, create a stressful and unpredictable workload. This contrasts with the workload of a transactional environment where the same set of transactions is repeated throughout the day, enabling the DBMS to leverage stored procedures, static SQL or dynamic SQL with cached execution plans. With ad hoc BI environments, the workload is more dynamic and the DBMS needs to provide workload partitioning and balancing. Justification for Hype Cycle Position/Adoption Speed: BI workloads are very diverse and require DBMSs that can juggle the complexity of such an environment. Many enterprises have not yet hit the level of BI diversity and only have begun to recognize the issues here. Business Impact Areas: High impact provides a platform for enterprises to support dynamic and complex query environments without the need to build "specialized" infrastructures to support different types of BI application profiles. Selected Vendors: IBM, Microsoft, Oracle and Teradata. 3.2 DBMS-Enabled Data Mining Definition: Embedded functions in the DBMS to support data mining techniques to enhance performance and improve usability. Justification for Hype Cycle Position/Adoption Speed: Lack of success in pure data mining tool technologies has led DBMS vendors to provide better support. Business Impact Areas: Embedding of data mining in the DBMS will eventually enable seamless deployment of new, exploratory styles of BI. Selected Vendors: IBM, Microsoft, Oracle and Teradata. 3.3 DBMS Complex Query Optimization Definition: Query performance is dependent on the SQL optimizer and the ability to support advanced algorithms for many functions, such as: table join processing algorithms, aggregation techniques, and indexing methods (for example, b-tree, hash and bitmap). Strong SQL optimization (for example, strong query execution plans based on cost and the efficient use of indexing by the optimizer) is an important requirement because of the ad hoc nature of queries to be performed. Justification for Hype Cycle Position/Adoption Speed: As BI moves to the mainstream, enterprises and users have become more sophisticated in the analysis they require. Vendors have talked about 30 May

8 technology improvements to enable optimization of these increasingly complex queries, but marketing by the vendors is ahead of the reality. Business Impact Areas: High impact increased insight into the business and business process issues. Selected Vendors: IBM, Microsoft, Oracle and Teradata. 3.4 Real-Time Data Updates (Merging Operational BI and Data Warehouse) Definition: Acquiring data in a real-time manner for operational BI and other time-sensitive analytics. Justification for Hype Cycle Position/Adoption Speed: Enterprises with true business requirements to deliver data to their data warehouse in real time are encountering challenges in deploying an architecture to accomplish this. Although the real-time enterprise (RTE) is a stated goal for many organizations, the reality is that most organizations have not yet managed to succeed in analyzing historical data. In a recent Gartner study, we learned that most organizations would be satisfied if they could reduce data latency to one day. Business Impact Areas: As the RTE becomes more of a reality, so too will the requirement for analyzing data in real time. 4.0 Sliding Into the Trough 4.1 ERP Data Warehouses Definition: Enterprise resource planning (ERP) vendors, such as SAP and PeopleSoft, are heavily promoting their data warehouse offerings. Each release offers more preconfigured content, additional BI applications and basic data warehouse management capabilities, and ERP vendors are increasing the pressure to use their data warehouse offerings by adding applications that simply require the data warehouse. Justification for Hype Cycle Position/Adoption Speed: ERP vendors control a significant portion of the IT infrastructure and require some products as a part of the software suite. Because of these dynamics, enterprises must evaluate the position of these products in their BI strategies. ERP vendor-based data warehouse solutions are optimized for their own data, although there are mechanisms to include external data. Business Impact Areas: ERP data warehouses are quickly becoming the preferred means for delivering operational reporting, which traditionally was the domain of the ERP package. Therefore, many enterprises will be forced to deal with ERP data warehouses in one way or another. Selected Vendors: PeopleSoft and SAP. Analysis by Ted Friedman 30 May

9 5.0 Climbing the Slope 5.1 ETL Embedded in DBMS Definition: Major DBMS vendors are embedding ETL functionality into the DBMS engine and offering ETL tools bundled with the DBMS. Time to Plateau/Adoption Speed: Less than two years Justification for Hype Cycle Position/Adoption Speed: Major relational DBMS vendors provide some degree of embedded ETL today and are improving their capabilities. However, functionality lags significantly behind leading independent ETL tools vendors. Business Impact Areas: As DBMS vendors strengthen their ETL offerings, low-cost, commodity ETL will become an attractive approach for many enterprises. This will enable more BI projects in cost-constrained enterprises. Selected Vendors: IBM, Microsoft and Oracle. Analysis by Ted Friedman 5.2 Metadata Tools Definition: Metadata tools provide the repository for defining the various components of the data warehouse architecture and the processes by which data is transformed, integrated and moved between them. Time to Plateau/Adoption Speed: Less than two years. Justification for Hype Cycle Position/Adoption Speed: Metadata has been the long-time problem area for data warehousing. Vendors have paid limited attention to this issue, adding only minimal tool-centric support. Focus on metadata across the data warehouse architecture is now increasing. Business Impact Areas: Strong metadata support will smooth data warehouse implementations and increase agility to react to changing business requirements. Selected Vendors: Ascential, Computer Associates International, Informatica and Reveleus Analysis by Ted Friedman. 5.3 DBMS-Enabled OLAP Definition: Embedded functions in the DBMS to better support OLAP applications with improved functionality and potential application performance improvements Time to Plateau/Adoption Speed: Less than two years. Justification for Hype Cycle Position/Adoption Speed: Many DBMSs have a number of features and capabilities to better support OLAP. Tools need to mature in their use of these capabilities. Business Impact Areas: Moderate provides potential for enhanced performance and scalability for increasingly sophisticated OLAP applications. Selected Vendors: IBM, Microsoft, Oracle and Teradata. 30 May

10 5.4 Star Schema Definition: A style of data modeling and schema design based on the concepts of a central fact table and multiple dimension tables linked by surrogate keys. A star schema is a highly denormalized style of design. Time to Plateau/Adoption Speed: Less than two years. Justification for Hype Cycle Position/Adoption Speed: Start schemas have received a significant amount of attention in the market to implement data warehouses. However, such an implementation method has led to "stovepipe" implementations that lack the flexibility that is required for BI. For some applications (for example, sales analysis), star schemas can provide significant performance advantage when implemented as a data mart for a more-tactical implementation. Business Impact Areas: Low-to-moderate value specific (and rigid) BI application implementations. Value proposition is limited and must be clearly understood in support of a BI strategy. Analysis by Ted Friedman 6.0 Entering the Plateau 6.1 Virtual Data Warehouse Definition: The virtual data warehouse is based on the concept of universal data access, an architecture that enables transparent access to data sources regardless of type or platform. Universal data access middleware enables users and applications to issue queries against heterogeneous data sources. Time to Plateau/Adoption Speed: Obsolete before Plateau. Justification for Hype Cycle Position/Adoption Speed: Complexities of dynamically retrieving data from source databases is complex and provides limited value to strategic BI. Business Impact Areas: Minimal impact the virtual data warehouse will see limited usage because of the technical limitations in performance, semantics and data quality issues. Selected Vendors: BEA Systems, IBM, MetaMatrix and Nimble Technology. Analysis by Ted Friedman 6.2 Single-Vendor Solution vs. Multivendor Technology Solution Definition: Complete end-to-end BI and data warehousing solutions provided by single vendors. Such solutions often include data integration, DBMSs and query/reporting capabilities. Time to Plateau/Adoption Speed: Less than two years. Justification for Hype Cycle Position/Adoption Speed: Enterprises continue in their search for singlevendor solutions to implement data warehouses to reduce costs and complexity. However, best-of-breed still provides the significant opportunity for success and competitive advantage. Business Impact Areas: High impact as large vendors embed more BI and data warehouse functionality in core products, best-of-breed vendors will begin to be squeezed out of the market. Enterprises need to select vendors carefully to achieve optimal results and avoid vendor lock-in. Selected Vendors: IBM, Microsoft and Oracle. 30 May

11 Analysis by Ted Friedman 7.0 Conclusion Trends in data warehousing depend on the maturation of current concepts and approaches, as opposed to the introduction of radical new technology. Major data warehousing vendors continue to innovate by addressing the increasingly strategic nature and increasing scale of today's BI requirements, as well as through the increased embedding of data warehouse infrastructure capabilities within other offerings. Although enterprises need to recognize these trends, for the most part, they do not change the best practices for solid data warehouse design and delivery. Enterprises must continue to focus on the planning and design of a data warehouse architecture to achieve the greatest efficiency and agility. 30 May

12 Appendix A: Hype Cycle Definitions Technology Trigger: A breakthrough, public demonstration, product launch or other event generates significant press and industry interest. Peak of Inflated Expectations: During this phase of overenthusiasm and unrealistic projections, a flurry of well-publicized activity by technology leaders results in some successes, but more failures, as the technology is pushed to its limits. The only enterprises making money are conference organizers and magazine publishers. Trough of Disillusionment: Because the technology does not live up to its overinflated expectations, it rapidly becomes unfashionable. Media interest wanes, except for a few cautionary tales. Slope of Enlightenment: Focused experimentation and solid hard work by an increasingly diverse range of organizations lead to a true understanding of the technology's applicability, risks and benefits. Commercial, off-the-shelf methodologies and tools ease the development process. Plateau of Productivity: The real-world benefits of the technology are demonstrated and accepted. Tools and methodologies are increasingly stable as they enter their second and third generations. The final height of the plateau varies according to whether the technology is broadly applicable or benefits only a niche market. Approximately 30 percent of the technology's target audience has or is adopting the technology as it enters the Plateau. Time to Plateau/Adoption Speed: The time required for the technology to reach the Plateau of Productivity. 30 May

13 Appendix B: Acronym Key BI DBMS ERP ETL OLAP RTE business intelligence database management system enterprise resource planning extraction, transformation and loading online analytical processing real-time enterprise 30 May