BEST PRACTICES IN SELECTING AND DEVELOPING AN ANALYTIC APPLIANCE

Transcription

1 BEST PRACTICES IN SELECTING AND DEVELOPING AN ANALYTIC APPLIANCE Author: Dr. Robert McCord

2 BEST PRACTICES IN SELECTING AND DEVELOPING AN ANALYTIC APPLIANCE Author: Dr. Robert McCord Dr. McCord boasts twenty years experience in relational database management. After receiving a Ph.D. from U.C. Berkeley in 1985, he managed architecture, development and product management groups at Ingres, Sybase, and Oracle, leading database companies, and now serves as Chief Architect / CIO (Chief Information Officer) at mlogica. A frequent writer and presenter in the data management field, Dr. McCord is also an excellent teacher and communicator with experience in application servers and mobile/wireless technology. Data warehouses are under severe stress to keep up with ever-growing user and ad hoc reporting workloads. It s no wonder that appliances which integrate servers, storage, databases, and software are popular for off-loading much of these growing query processing requirements. Not all data warehouse appliances are the same, so if your organization is thinking about adopting one, you ll need to consider the architecture, performance and scalability, costs, return on investment, and how to address the risks an appliance may introduce. This paper addresses the best practices in selecting and developing a data warehouse appliance. INTRODUCTION A data warehouse (DW) appliance is an integrated set of servers, storage, operating system (OS), database, and software that is specifically preconfigured and tuned for the rigors of data warehousing. Data warehouse appliances offer an attractive price/performance value proposition: they are frequently a fraction of the cost of traditional data warehouse solutions, and they have reduced maintenance requirements. The underlying architecture of a data warehousing appliance is an important consideration. Some appliances use column-based databases, and others use traditional relational systems.some architectures distribute data onto dedicated disk storage units that are connected to each server. Others have a disk array allowing multiple processors to access and process the data in parallel. For companies with data warehouses particularly those who are experiencing performance or cost issues related to rapid growth in data volumes or in the number of users and queries appliances offer dramatically reduced total cost of ownership (TCO) coupled with high-performance data access. However, it is essential to also consider the various factors associated with planning, migrating, executing, and maintaining an appliance for a particular environment. 1. PICK THE RIGHT DATABASE ARCHITECURE There are two types of database architectures used in appliances: row-based and column-based. Although row-based databases are a popular choice for transaction processing, they have limits in decision support applications. While the row-based approach is well suited to OLTP (online transaction processing), which requires updating an entire record (stored as a row) for each new customer transaction, the column-based approach is ideal for OLAP (online analytical processing), which requires reading only a small set of attributes for each record, stored in particular columns. In a row-oriented DBMS, the data for an entire row is accessed for each operation, and to process a query, the DBMS must read entire rows (including columns not needed or requested). Some databases use special features such as bitmap indices, data cubes, and materialized views to reorganize data more optimally for specific types of queries, but these data structures are complex to design and can impose dramatic overhead for storage space and in many cases degrade overall operational performance. The column-based approach looks at data vertically. In the case of a customer database, for example, it independently stores and retrieves the data for each column: all the first names, all the last names, all the cities, all the zip codes and so on. Since each 1

3 column is physically stored independently on disk, only the columns in a row that are required to resolve a query are retrieved from disk. Another benefit of this physical separation is that each column can be retrieved in parallel, which is extremely efficient. In effect, all columns in all tables are automatically indexed but without the overhead (storage, management, and tuning) that is associated with traditional approaches to indexing. This approach dramatically improves query performance by eliminating unnecessary disk and memory I/O. no longer a need to build summary and aggregate tables to boost performance. The important concept to note is that this is actual compression of the size of the database relative to the size of the input data. This means that for each terabyte of input data, the fully indexed physical storage requirements in the columnar database will be less than a terabyte. With data growing over 60% a year 1 in some organizations, this results in significant cost savings. While many row-based database vendors have started to introduce compression, it is only focused on the actual physical input data; all of the storage overhead of multiple indices, summary and aggregate tables remains. Also, since they are still physically storing data of widely varying datatypes on the same disk page, the compression algorithms cannot be as efficient as those used by column-based databases. Therefore with row-based databases the physical storage required is still many multiples of the size of the actual input data. What this means, in summary, is that for most analytic queries, the amount of data coming back from a column-based database is going to be significantly less than from a conventional database, so it s going to require less I/O. This results in the potential for dramatically higher query performance. The example below quantifies the potential for dramatic I/O savings. Storage Savings Query: How many males are not insured in United Kingdom? One by-product of using a column-based approach is the ability to achieve significant data compression. Column-based vendors claim that data can be compressed up to 70%, even taking into account the fully indexed data set. They also assert that there is Using the traditional row-based method, this query would require 800 bytes x 20M/16K page = 1,000,000 I/Os 2

4 Using the column-based approach, this query would require 470 I/Os. Column-based databases are well proven in the market and have been rapidly growing in deployments. As an example, the leading columnar database, Sybase IQ, was the first column-based database. Having been in market for over a decade, it now has more than 2000 implementations worldwide. Based on growing market acceptance, a number of additional vendors have entered the market including Vertica, ParAccel, InfoBright, Sand Technology and Calpont. 2. IS THE DESIGN SIMPLE? The leading reason that companies use appliances is for simplicity, so it is important to understand how a specific appliance makes supporting data warehouse operations simple. Make sure that the appliance has all the ingredients needed to make the solution simple. Look under the covers to understand what software components are in the system, whether administration tools are included for the whole system, not just the database, and how easily it scales. Make sure you can add servers, memory, CPU, and storage easily. You do not want to have an appliance run out of capacity within a short amount of time. Main Ingredients: Do you have single vendor support for all the tools required in the appliance? Does it pre-integrate the hardware, storage and software including database, ETL and BI tools? Administration: Does the appliance have an administration module? Does it have a point and click interface? Does it monitor the operating system and storage? How does it monitor the database? Does it monitor the ETL and BI tools? How easy is it to add users? Scalability: Can you easily add more capacity in terms of concurrent users? Can you just turn on more processors for more power and pay as you go? Does the system support linear scalability? Storage: Can you easily pay as you go to increase data capacity? Do you have to buy a whole new system when you reach your data capacity? Can you store online all your relevant history? The appliance vendors fit into two basic categories: Optimized/Balanced Warehouses and DW appliances. Optimized/Balanced Warehouses: IBM, Dell, Oracle are working with their partners to deliver pre-configured systems using off-the-shelf database systems. These warehouses are pre-tested to support a specific data load. What they are missing is the ability to administer the appliance easily as well as to scale easily. They are missing integrated BI and ETL tools, and sometimes lack single vendor support. The Data Warehouse Appliance vendors typically offer single vendor support, and deliver the database software, operating system, and hardware. Netezza, DataAllegro, Teradata and Sybase are examples that fall into this category. Sometimes the hardware is proprietary, as is the case with Netezza and Teradata. Sybase teamed with MicroStrategy to include BI tools along with Sybase ETL tools delivered on a standard, off-the-shelf hardware and operating system platform. 3. IS IT FAST FOR AD HOC QUERIES AND CONCURRENCY? Even if you have selected a column-based over a row-based approach, don t assume that all columnbased databases offer comparable performance. Specialized indexes can further accelerate ad hoc query performance.these include bitwise indexes for low cardinality data (which further reduces storage requirements, and improves query performance through the use of tokenization), grouped data, range data, pre-joined columns, textual analysis (providing analytics on unstructured data that may be combined with structured analysis), real-time comparisons for Web applications, date and time analysis. 3

5 Example of Bitwise Indexing If you have selected a traditional relational database with the appliance, then you will need to add in administration time to tune each query. Each key needs to be indexed, as well as creating summary tables and materialized views for faster execution.this method will also increase the overall size of your system. Be aware of the number of concurrent users you will have. Look for consistent performance even with unpredictable queries or increased workloads. The promise of the appliance is to deliver fast response times as your users increase in number. This usually means running queries in parallel to take advantage of the processing power of the appliance. A key best practice during appliance evaluation is to make sure that realistic workloads, both in terms of query mix and user concurrency, are being tested. Many appliance vendors will ask you to provide them your toughest query and they will demonstrate how fast they can process it. However their approach may not work well across the broad mixture of queries your organization needs to process in parallel, or for the concurrent user workloads required. Be careful that you are not being drawn into a single-query-ata-time batch architecture. Just as with mainframes, this type of approach means you may need to significantly over-invest to your peak workload needs. A good test point is to test at least 50 to 100 concurrent users running a widely varied query mix in parallel. Also, do not share the test queries with the vendor until they are onsite to help you run the tests. This will prevent unrealistic one-off tuning efforts that may skew your test results. 4. IS IT AFFORDABLE? PRICE THE PERFORMANCE Most data warehouse appliance vendors have different configurations they offer. Some are configured for different anticipated workloads from thedepartment to the enterprise while others target a range of data volumes from 1 to 4 terabytes. From a purely technical perspective, the leading relational databases used in Optimized/Balanced configurations can continue to scale (Oracle and IBM claim scalability into petabytes of data). But the real question is, how much will it cost and how difficult will it be to manage? Appliances are gaining ground precisely because they are faster, easier, and less expensive to deploy and maintain than traditional databases, which have to be tuned, optimized and, in the case with Oracle, clustered to perform in large-scale deployments. In the case of Teradata, which was the first to exploit the MPP shared-nothing architecture long ago, the upstart appliances are gaining ground primarily on price. To get the real price, both data and user capacity need to be examined. Price per terabyte is a good gauge for data, but be aware of other criteria such as cost per concurrent user. If you have selected an appliance with a traditional relational database, understand the true storage capacity required. Remember, these databases grow in size. On the other hand, if you are using a column-based database, your database might be smaller than the input data due to compression. Also take into consideration the backup and redundancy factors.how much of the storage is usable data? Do you have automatic failover to another disk in case of a failure? Understand the amount of processing power you will need to get sub-second response time for both your ad hoc queries as well as the complex analytical queries. There are TPC-H benchmarks to help validate performance and price performance characteristics, but not all vendors have participated and it is hard to make an apples-to-apples comparison. 4

6 5. CHECK THE ADMINISTRATION CAPABILITIES IDC recently conducted a survey of appliance customers, and the top feature they reported missing from most appliances today is administration. In the rush to create new solutions, vendors are configuring the hardware and database for the appliance, but forget that there is an OS and storage to manage. The administration module that manages the database should handle the OS, Storage, ELT, Data Loading, and BI tools. Without this feature, customers have the added burden of administering each environment separately. Some appliance vendors are offering remote administration. This could be very important to companies that have little to no additional staff to support the appliance. Other options include supporting the appliance in a hosted or Software-asa-Service environment. 6. CAN YOU REALLY BE UP AND RUNNING IN DAYS The claim, up and running in days is widely used among the data warehouse appliance vendors. The advantage that most appliance vendors deliver is the ability to pre-configure the hardware and database software for maximum performance. This will help alleviate weeks of work, and sometimes more. The bigger time allocation is in moving the specific applications and data to the appliance. The database schema needs to be recreated from the source data warehouse and loaded into the appliance. Then, data needs to be loaded, and in some cases transformation is required. Users, permissions, and security, need to be established and tuning needs to be completed.finally, reports and dashboards need to be created. When evaluating an appliance, look to see how easily they can be plugged into your existing environment. Do they have tools to reverse engineer your data warehouse and automatically build your schema? Can you use the same schema designs that are already in use? Do they have tools to handle the data transformations required? How fast can they load data? Do they have the ability to handle near realtime updates? Do they have tools to quickly provision users with appropriate permissions? Do they have built-in BI tools for easy creation of reports? 7. LOOK AT PROPRIETARY vs STANDARD HARDWARE The first appliance vendors, Netezza and Teradata, used proprietary hardware. Most of the newer appliances in market today use standard hardware. Is there a difference? Yes, there is a big difference. One lesson from the early days of relational databases was Britton-Lee. They created a relational database on proprietary hardware, butrealized that they could not keep up with the rapid evolution of hardware architecture; therefore the performance advantage of the system was short-lived. Companies like Sybase, Informix and Oracle were able to deliver better database performance using off-the-shelf hardware. Hardware companies such as IBM, Sun, HP and Intel are making constant improvements in redundancy, energy management, processor speed, speed of disk access and many more aspects of system design. The appliance vendors that use off-the-shelf platforms are able to take advantage of these new features and capabilities as they are released. 8. IS IT GREEN? Data center energy management has become increasingly important as the price of energy increases and system deployments grow beyond the capacity of current facilities to supply their power and cooling needs. This becomes more important in the data warehousing world, because these systems require a lot of processing power and storage capacities. In fact, data centers have doubled their energy use in the past five years. Becoming a green business is not something that happens overnight. You need to find the right strategy that works for your business. When looking at the appliance, take into consideration the footprint required. Look for machines that have the ability to cool the inflow of air, some show a 55% to 60% drop in heat output. Also try to consolidate workloads, and strive for fewer, more efficient machines. 5

7 THE BOTTOM LINE Data Warehouse Appliances are here to stay. They are providing dramatically reduced total cost of ownership (TCO) coupled with high-performance data access. These appliances provide compelling reasons for companies with large data warehouses to seriously consider complementing their systems with DW appliances. Not all appliance vendors are the same, and if your organization is thinking of adopting one, consider these basic tips in evaluating your solution. Just as if you were building a house, start with the architecture, and make sure the appliance is fast, easy and affordable while living up to its claims. 1. Source: research study from IBC and EMC 6