Database Performance Tuning and Optimization. Using Oracle

Transcription

1 Database Performance Tuning and Optimization Using Oracle

2 Springer New York Berlin Heidelberg Hong Kong London Milan Paris Tokyo

3 Sitansu S. Mittra Database Performance Tuning and Optimization Using Oracle With 214 Illustrations Includes CD-ROM 123

4 Sitansu S. Mittra Senior Principal Engineer and Database Management Specialist Computer Sciences Corporation 5 Cambridge Center Cambridge, MA USA mittra4@aol.com Library of Congress Cataloging-in-Publication Data Mittra, Sitansu S. Database performance tuning and optimization : using Oracle / Sitansu S. Mittra. p. cm. Includes bibliographical references and index. ISBN (alk. paper) 1. Database management. 2. Oracle (Computer file). II. Title. QA76.9.D3 M dc ACM computing Classification (1998): C.4, E.1, E.2, H.2, H.3, H.4 ISBN Printed on acid-free paper Springer-Verlag New York, Inc. All rights reserved. This work may not be translated or copied in whole or in part without the written permission of the publisher (Springer-Verlag New York, Inc., 175 Fifth Avenue, New York, NY 10010, USA), except for brief excerpts in connection with reviews or scholarly analysis. Use in connection with any form of information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed is forbidden. The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights. Printed in the United States of America SPIN Springer-Verlag New York Berlin Heidelberg A member of BertelsmannSpringer Science+Business Media GmbH Disclaimer: This ebook does not include the ancillary media that was packaged with the original printed version of the book.

5 To my family Pranati, Partha, and Ansuman for their unwavering support during my solitary sojourn into the intricacies and excitement of the database world

6 Preface Scope The book provides comprehensive coverage of database performance tuning and optimization using Oracle 8i as the RDBMS. The chapters contain both theoretical discussions dealing with principles and methodology as well as actual SQL scripts to implement the methodology. The book combines theory with practice so as to make it useful for DBAs and developers irrespective of whether they use Oracle 8i. Readers who do not use Oracle 8i can implement the principles via scripts of their own written for the particular RDBMS they use. I have tested each script for accuracy and have included the sample outputs generated from them. An operational database has three levels: conceptual, internal, and external. The conceptual level results from data modeling and logical database design. When it is implemented via an RDBMS such as Oracle, it is mapped onto the internal level. Database objects of the conceptual level are associated with their physical counterparts in the internal level. An external level results from a query against the database and, as such, provides a window to the database. There are many external levels for a single conceptual level. The performance of an OLTP database is measured by the response times of the database transactions. The response time depends on the efficiency of all three levels. A query on a well-designed conceptual level can run slowly if the SQL formulation of the query is poorly done, or if the database objects involved in the query are fragmented, or if a table used by the query has excessive chaining. Likewise, a well-formulated query can run slowly if the design of the conceptual level is bad. Such examples abound in database applications. The book addresses each level separately by focusing first on the underlying principles and root causes of problems and then offering solutions, both on a theoretical level and with Oracle SQL scripts with sample outputs. Even if all three levels of a database are kept properly tuned, its performance may suffer due to other factors. For example, the CPU may remain pegged at or very near to 100%, the memory may be inadequate causing excessive paging and swapping bordering on thrashing, disk controllers may be inefficient, etc. These factors are outside the realm of the database and, therefore, are not treated in this book. Some discussion of

7 viii Preface tuning the CPU and the memory as a part of the internal level of a database appears in Chapter 6. The theory of relational databases as propounded by Codd has its foundation rooted in mathematics. Consequently, database topics can often be discussed using the mathematical language and notations. Due to the inherent precision of such language I have used it in my treatment of database topics, whenever appropriate. Overview of the Book The book consists of 12 chapters grouped in three parts, five appendices, and an index. Each part starts with an overview of the part and a brief description of each chapter included in the part. Each chapter starts with a list of topics and an overview and ends with a list of key words used in the chapter, a section on references and further reading, and exercises, where appropriate. Part 1: Chapters 1 to 3 Part 1, Methodology, consists of Chapters 1 to 3 that cover the methodology aspect of database performance tuning and optimization. The goal of Part 1 is to establish a sound conceptual framework for identifying tuning issues and taking a well-planned approach to address them. As such, it is primarily theoretical in nature and avoids, as far as practicable, references to any particular RDBMS. The methods and principles discussed in this part can be applied to any RDBMS with which the reader works. Chapter 1, Database Application Development, contains a detailed discussion of the five phases of building a database application starting with the information requirements analysis, continuing through logical and physical database designs, and ending in database implementation. This is a one-time effort. When the database becomes operational, its ongoing maintenance phase begins. The issues of performance tuning happen primarily during this phase, although initial performance checks are done during the development. This chapter is targeted primarily for the developers and the system analysts. Chapter 2, Performance Tuning Methodology, describes the three levels of a database and emphasizes that a database must be well tuned at all three levels in order to run optimally. It provides the methodology for performance tuning. Both the DBAs and the developers will benefit from this chapter. Chapter 3, Tuning the Conceptual Level of a Database, explores a variety of issues underlying the tuning process of the conceptual level. It covers three major areas: denormalization, partitioning of tables and indices, and data replication among multiple locations of a database. The primary readers of this chapter are the developers.

8 Preface ix Part 2: Chapters 4 to 10 Part 2, Oracle Tools for Tuning and Optimization, consists of Chapters 4 to 10 that cover the Oracle tools for monitoring the performance of a database and tuning and optimizing its internal and external levels, as needed. This part is specific to Oracle 8i with a glimpse into Oracle 9i in Chapter 10. It is intended for the DBAs. The goal of Part 2 is to provide the underlying tuning principles and a repertoire of Oracle tools to implement these principles. Chapters 4 to 7 discuss the tuning of the internal level and Chapters 8 to 9 that of the external level. Chapter 10 describes several features of Oracle 8i not covered in the earlier chapters and gives an overview of several tuning features of Oracle 9i. Chapter 4, Internal Level of an Oracle Database, introduces the structure of the internal level comprising an instance and a database. The instance consists of a set of memory-resident data structures and a set of background processes. The database consists of a set of disk-resident data structures, namely, the tablespaces, tables, indices, etc. Chapter 5, Tuning of Disk-Resident Data Structures, discusses in detail the tuning principles of the components of an Oracle database. Fragmentation and chaining are two major areas that the DBAs need to track and address for tuning. Ample scripts are provided to implement the tuning principles. Chapter 6, Tuning of Memory-Resident Data Structures, discusses in detail the tuning principles of the System Global Area (SGA) of an Oracle instance and the administration and optimization of the background processes. The chapter includes separate sections dealing with the tuning of the CPU and memory-related objects. Ample scripts are provided to implement the tuning principles. Chapter 7, Oracle Utility for Tuning and Optimization, involves a detailed discussion of the two Oracle diagnostic tools, UTLBSTAT and UTLESTAT, that collect detailed statistics during a data collection period specified by the DBA and generate a report containing an analysis of the collected statistics along with recommendations for improvement, as needed. The chapter includes detailed directions for interpreting the output and for taking corrective actions to address the deficiencies identified in the output. Chapter 8, Optimization of the External Level of a Database, introduces the mathematical theory underlying the query optimization process. This is followed by a detailed treatment of Oracle s optimization tools such as EXPLAIN PLAN, SQLTRACE and TKPROF, and AUTOTRACE. Chapter 9, Query Tuning and Optimization Under Oracle 8i, discusses the rule-based and the cost-based optimizers, various joining techniques, and the use of hints in queries to suggest specific query execution plans. Chapter 10, Special Features of Oracle 8i and a Glimpse into Oracle 9i, covers several special features of Oracle 8i pertaining to performance tuning and optimization that were not covered in the earlier chapters. It closes with an overview of some of the features of Oracle 9i.

9 x Preface Part 3: Chapters 11 and 12 Part 3, Contemporary Issues, consists of two chapters that discuss two specific contemporary issues, namely, the tuning principles of data warehouses and of Web-based databases. Although the design principles of an OLTP database are different from those of a data warehouse or a Web-based database, their tuning principles are substantially the same. In this part we capture only those tools and techniques that are unique to these two special types of databases. Chapter 11, Tuning Data Warehouses at All Levels, starts with a discussion of the design principles of a data warehouse and identifies the structural differences between an OLTP database and a data warehouse. It then introduces the data loading principles for a data warehouse. The chapter closes with a discussion of the tuning principles for a data warehouse at the internal and the external levels. Chapter 12, Tuning Web-Based Databases, starts with an introduction to the three-tier and n-tier architectures of client-server applications with emphasis on the Web-based applications. The Oracle product OAS is discussed in detail and an overview is offered of Oracle s new product ias. The chapter closes with a discussion of the tuning principles for a Web-based database at the internal and the external levels. Appendices A to E The five appendices discuss several DBA issues, although they are not directly related to performance tuning and optimization. They are included here to make the book useful for addressing issues outside the realm of performance monitoring and tuning. Appendix A, Sizing Methodology in Oracle 8i, contains the algorithms and two C programs for estimating the storage space needed for tables, indices, and tablespaces during the physical design phase, as discussed in Chapter 1. The two C programs implement the sizing algorithms. Sizing of the tables, indices, and tablespaces constitutes the capacity planning activity which is extremely important for smooth and optimal operation of a database. Appendix B, Instance and Database Creation, contains detailed instructions for creating an Oracle instance and an Oracle database under UNIX. Sample files such as init.ora, config.ora, etc. are included as attachments to clarify the steps. Inexperienced DBAs often find the creation of an instance quite a formidable job because of the following two reasons: Various files and directories must be properly set up before the instance can be created. After the database is created, its tablespaces, tables, indices, views, user roles and privileges, etc. must be created within the database. Appendix C, Instance and Database Removal, offers a similar step-by-step methodology to drop an instance and its associated database.

10 Preface xi Appendix D, Database Refresh with Transportable Tablespaces, provides a methodology for refreshing an existing database with data from another database with identical structure. The transportable tablespaces introduced in Oracle 8i are used for this purpose. The appendix also includes the procedure to determine if a set of tablespaces is transportable. Appendix E, Mathematical Foundation of Relational Databases, contains relevant mathematical materials that were used by Dr. Codd to establish the relational database theory on a sound mathematical foundation. He introduced relational algebra and relational calculus as the two alternative but equivalent mechanisms for query languages and query processing. The later sections of the appendix contain discussions of three data structures, namely, linear linked list, search tree, and hash tables, as the underlying mechanisms for sequential, indexed, and direct search of relations in a database. Appendix E is included primarily for the mathematically inquisitive readers. A graduate level course normally labeled as discrete mathematics in the current day computer science curricula will provide adequate mathematical preparation to understand this appendix. But a lack of understanding of this appendix will not interfere with reading and using the rest of the book. The key words appearing at the end of each chapter and each appendix are combined into a single comprehensive index for the whole book. It is included at the end of the book. Unique Features The book offers several unique features that distinguish it from other books with similar titles and scope that are currently available in the market. (a) Three Levels of a Database: A clear knowledge of the three levels and their mutual relationships is crucial to the understanding of database operations and performance tuning issues. Chapter 2 explains this background information with examples and emphasizes that a database must run optimally at each level for overall optimal performance. Also, tuning may have to be done at multiple levels to resolve a problem. (a) Ideal Mix of Theory and Practice: Throughout the book I have described the underlying principles for handling database performance and then included Oracle SQL scripts to implement these principles. This is the standard error-fixing methodology for software and database problems. One first localizes a problem, then finds the root cause(s), and finally corrects it. My treatment allows the reader to understand the principles and then apply them to other RDBMSs besides Oracle. (a) Web Based Databases: In today's software industry n-tier (n 3) architecture with the database residing on the nth tier is very common. The principles of database design and performance tuning remain largely the same in such applications. However, a few nuances need to be clarified. Chapter 12 addresses that area. (a) Complete Instructions for Creating an Instance and a Database: From my own experience I have found that creating an Oracle instance often poses a more serious

11 xii Preface problem than running a script to create the database. Creation of an instance involves an understanding of the disk storage structure, defining the initialization parameters properly, and setting up the necessary directories. Appendix B contains a complete set of instructions to implement a database and all its objects. (a) Mathematical Foundation of Relational Databases: Dr. Edgar Codd established the relational database theory on a sound mathematical foundation. He introduced relational algebra and relational calculus as the two alternative but equivalent mechanisms for query languages and query processing. Appendix E contains the relevant mathematical materials along with the three data structures that are used for sequential, indexed, and direct searches of a relational database. (a) References and Further Reading: Titles included in this section at the end of each chapter will help the more inquisitive readers in finding additional materials relevant to the topics. Some of the references offer an alternative approach to handle the performance and tuning issues. (a) Exercises: The exercises at the end of each chapter are optional. Occasionally they introduce topics that are an extension to the main body of the text. But they are intended primarily for students and instructors when the book is used as a text for a database course. In today s educational systems often colleges and universities offer certificate courses in specialized areas such as performance tuning of Oracle databases. The exercises can then be used as assignments for the students. Reader Community The book assumes an understanding of relational database concepts and a familiarity with Oracle. Some knowledge of college-level mathematics will be helpful. Thus, it is not intended for the beginners. It is designed primarily for database professionals such as DBAs, developers, and system analysts with two to three years of experience in the development, implementation, and maintenance of relational databases using some RDBMS, preferably Oracle. A junior DBA or a developer can use the scripts of the book under the guidance of a senior DBA or developer. The principles and tools presented here arise out of my personal experience with Oracle and several other RDBMSs (e.g., SQL Server, INGRES, LOGIX, RIM, etc.) that I have used since I have been using them regularly as a part of my current and previous positions as a senior Oracle DBA, Data Architecture Group Manager, Development and Architecture Manager, and QA Manager, in industry. In addition, I have taught graduate-level database courses at Boston University, Northeastern University, and Wentworth Institute of Technology. I have used the collective experience from these activities in writing this book. The scripts and their sample output included in the book were written over several years as my ideas shaped up through academic and industrial experience. Colleges and universities do not offer full semester courses dealing with database performance and tuning per se. Usually such materials are included in other database courses at an advanced level, in the certificate and state-of-the-art programs offered for the industry professionals, or simply as courses in continuing education programs. The book

12 Preface xiii can be used as a text for such courses. The instructors can use the exercises at the end of each chapter as student assignments. Some of the exercises explore new topics not discussed in the accompanying chapters. These can be used as group projects for the students. In order to keep the book self-contained I have avoided the use of any third party tools. The only software being used is Oracle 8i RDBMS along with SQL*Plus and PL/SQL. Almost all of the scripts included in the book are completely portable between UNIX and NT and can be used in any Oracle 8i installation under either operating system. The scripts are collected into a separate CD that is included with the book. Some readers will read the book from beginning to end. Others will use it as a reference and utilize the tuning guidelines and accompanying scripts to resolve specific problems. Possibilities are quite varied. Acknowledgments I got much valuable information from Oracle MetaLink (accessible through the web site I was able to resolve several issues and clarify concepts with the help of the information. In addition, I benefited from phone conversations with members of the Oracle Technical Group in response to TARs that I opened with Oracle. I am highly impressed with the courtesy, promptness, and professionalism of the Oracle technical staff. I acknowledge the friendly support of the staff of Springer-Verlag to make the publication of the book a success. In particular, I thank Wayne Yuhasz, the Executive Editor of Computing and Information Science, Wayne Wheeler, the Assistant Editor of Computing and Information Science, and Robert Wexler, who converted the whole manuscript into the format needed for publication. Sudbury, Massachusetts, USA July 2002 Sitansu S. Mittra

13 Contents Preface vii Part 1 Methodology 1 1 Database Application Development 3 Outline... 3 Overview of the Chapter s Software Technology Era Role of the Database in SDLC and SMLC Enterprise Modeling Physical Database Design Database Implementation Database Maintenance Naming Guidelines for Database Objects Key Words References and Further Reading Exercises Performance Tuning Methodology 29 Outline Overview of the Chapter Three Levels of a Database Optimization at Each Level Process and Metric for Performance Tuning Key Words References and Further Reading Exercises Tuning the Conceptual Level of a Database 41 Outline Overview of the Chapter Three Versions of the Conceptual Level... 42

14 xvi Contents 3.2 Performance Issues at the Conceptual Level Denormalization of the Conceptual Level Optimal Indexing of Tables Integration of Views into Queries Partitioning of Tables and Indices Data Replication Key Words References and Further Reading Exercises Part 2 Oracle Tools for Tuning and Optimization 55 4 Internal Level of an Oracle Database 57 Outline Overview of the Chapter Components of the Internal Level Oracle Instance Oracle Database Internal Level Tuning Methodology Oracle Data Dictionary V$ Views and X$ Tables Initialization Parameters for an Oracle Database Key Words References and Further Reading Exercises Tuning of Disk Resident Data Structures 81 Outline Overview of the Chapter Disk-Resident Data Structures Performance Tuning of Disk-Resident Data Structures Baseline of Disk-Resident Data Structures Segments with Information on Extents Changes to Database Schema Data Block Structure Used Space Fragmentation at the Segment Level Severity of Free Space Shortage Free Space Fragmentation at Tablespace Level Row Chaining and Row Migration in Tables Performance Tuning of Rollback Segments Key Words References and Further Reading Exercises

15 Contents xvii 6 Tuning of Memory-Resident Data Structures 137 Outline Overview of the Chapter Memory Resident Data Structures Performance Tuning Data Block Buffers Redo Log Buffer Shared SQL Pool Background Processes Tuning the Memory Tuning the CPU Pinning Packages in Memory Latching Mechanism for Access Control Key Words References and Further Reading Exercises Oracle Utility for Tuning and Optimization 189 Outline Overview of the Chapter Scope of Oracle Utilities UTLBSTAT and UTLESTAT Utilities Location and Function of the Utility Scripts Procedure for Running the Utilities UTLBSTAT/ESTAT Performance Report Analysis Comprehensive Tuning Plan for Internal Level Performance Tracking Tuning Activities Key Words References and Further Reading Exercises Optimization of the External Level of a Database 229 Outline Overview of the Chapter Contents of the External Level Principles of Query Optimization Query Optimization in Oracle Optimal Indexing Guidelines Methodology for Optimal Indexing Implementation of Indices in Oracle

16 xviii Contents 8.7 Tools for Tracking Query Performance Key Words References and Further Reading Exercises Query Tuning and Optimization Under Oracle 8i 283 Outline Overview of the Chapter Oracle Query Performance Query Tuning in Oracle: General Principles Query Tuning in Oracle: Cost-based Optimizer Query Tuning in Oracle: Rule-based Optimizer Tuning of Join Queries Statistical Forecasting for Tracking Performance Key Words References and Further Reading Exercises Special Features of Oracle 8i and a Glimpse into Oracle 9i 315 Outline Overview of the Chapter Scope of the Chapter Evolution of Oracle Through Versions 8 and 8i Partitioning of Tables and Indices Materialized Views Defragmentation via Local Tablespace Management LOB Data Type Versus LONG Data Type Multiple Buffer Pools in the SGA Query Execution Plan Stability via Stored Outlines Index Enhancements Query Rewrite for Materialized Views Online Index Creation, Rebuild, Defragmentation ANALYZE versus DBMS_STATS Optimization of Top_N Queries Glimpse into Oracle 9i Key Words References and Further Reading Part 3 Contemporary Issues Tuning the Data Warehouse at All Levels 361 Outline Overview of the Chapter Advent of Data Warehouse

17 Contents xix 11.2 Features of Data Warehouse Design Issues of Data Warehouse Structure of Data Warehouse Proliferation from Data Warehouse Metadata Implementation and Internal Level Data Loading in Warehouse Query Processing and Optimization Key Words References and Further Reading Exercises Web-Based Database Applications 385 Outline Overview of the Chapter Advent of Web-Based Applications Components of Web-Based Applications Oracle Application Server (OAS) Database Transaction Management Under OAS Oracle Internet Application Server (ias) Performance Tuning of Web-Based Databases Tuning of Internal Level Tuning of External Level Key Words References and Further Reading Exercises Appendices 399 Appendix A Sizing Methodology in Oracle 8i 401 Outline Overview of the Appendix A1. Transition from Logical to Physical Database Design A2. Space Usage via Extents A3. Algorithms for Sizing Tables, Indices, Tablespaces A4. STORAGE Clause Inclusion: Table and Index Levels A5. Sizing Methodology A6. RBS, SYSTEM, TEMP, and TOOLS Tablespace Sizing Key Words References and Further Reading Appendix B Instance and Database Creation 413 Outline Overview of the Appendix

18 xx Contents B1. Preparation B2. Instance Startup B3. Database Creation B4. Creation of Users, Roles, Privileges, Schema B5. Miscellaneous Informational Items Attachments Appendix C Instance and Database Removal 433 Outline Overview of the Appendix C1. Preparation C2. Locating the Components C3. Removing the Components C4. Verification Appendix D Database Refresh with Transportable Tablespaces 439 Outline Overview of the Appendix D1. Database Refresh Process D2. Detailed Methodology with Scripts D3. Time Estimates D4. Internal Consistency Key Words Appendix E Mathematical Foundation of Relational Databases 449 Outline Overview of the Appendix E1. Relational Database Systems Foundation Pillars E2. Relation E3. Functional Dependency E4. Query Languages E5. Relational Algebra: Prescriptive Query Languages E 6. Primitive and Derived Operations E7. Closure Property for Relational Algebra E8. Relational Calculus: Descriptive Query Languages E9. Tuple Relational Calculus E10. Domain Relational Calculus E11. Equivalence Theorem for Algebra and Calculus E12. Data Structures for Search Algorithms E13. Linear Linked List E14. Search Tree E15. Hash Table

19 Contents xxi E16. Performance Metrics Key Words References and Further Reading Index 481