Index Creation Performance during Data Loading

Size: px

Start display at page:

Download "Index Creation Performance during Data Loading"

Vivian Joseph
6 years ago
Views:

1 Technical Note 6: Index Creation Performance during Data Loading Technical Note 6 Index creation performance during Data Load Last Modified: October 2, 2003 Version 1 Area: Data Loading Siebel Releases: Siebel 7 Windows Releases Windows 2000 Server, Advanced Server, Data Center SQL Server Releases SQL Server 2000 Siebel ebusiness Applications are comprised of several customer and employee facing modules that support all major database vors and platforms. These modules are fully configurable and once implemented the queries that carry out application logic will use only a small portion of the defined indexes originally shipped with the application. Different databases handle these null indexes in different ways. SQL Server stores all of these extraneous indexes even though they are null. You can obtain significant gains in performance and in storage by eliminating these null indexes during data loading as well as gaining other benefits during normal operations such as reduced time to backup the database. Always consult your Siebel TAM before dropping any indexes. Goals: 1. Speed index creation time during data loading by eliminating the creation of poor/nonfunctional indexes 2. Use less disk space during loads by reducing the amount of transaction logs generated. SQL Server Books online states the following regarding switching recovery models to perform data loading and index creation: Switching Recovery Models Note: Please read SQL Server Books Online and understand the recovery implications for your specific database before starting. You can switch a database from one recovery model to another in order to meet changing business needs. For example, a mission-critical online transaction processing (OLTP) system requires full recoverability but periodically underes bulk load and indexing operations. The recovery model for the database can be changed to Bulk-Logged for the duration of the load and indexing operations and then returned to Full Recovery. This increases performance and reduces the required log space while maintaining server protection. However, you should understand that while bulk logged recovery produces smaller logs, it can make for larger log backups. The following is an explanation from SQL Server Backup and Recovery, With bulk-logged changes, the data on those extents is logically part of the transaction log, and thus included in the next log backup after the bulk change is made. Additionally,

2 this is why a bulk logged transaction backup can be larger than a normal transaction log backup under the full mode. 1 Note: Switching recovery models during a bulk load operation is permitted. The logging of the bulk operation changes appropriately. The following table indicates what action to take when switching from one recovery model to another. From To Action Description Full Recovery Bulk-Logged No action Requires no change in backup strategy. Recovery Continue to perform periodic database, log, and (optionally) differential backups. Full Recovery Simple Recovery Optionally back up the transaction log prior to the change Bulk-Logged Recovery Bulk-Logged Recovery Executing a log backup immediately before the change permits recovery to that point. After switching to the simple model, stop executing log backups. Full Recovery No action Requires no change in backup strategy. Recovery to any point in time is enabled after the next log backup. If point-in-time recovery is important, execute a log backup immediately after switching. Simple Recovery Optionally back up the transaction log prior to the change Simple Recovery Full Recovery Back up the database after the change Simple Recovery Bulk-Logged Recovery Back up the database after the change Executing a log backup immediately before the change permits recovery to that point. After switching to the simple model, stop executing log backups. Execute a database or differential backup after switching to the Full Recovery model. Begin executing periodic database, log, and (optionally) differential backups. Execute a database or differential backup after switching to the bulk-logged model. Begin executing periodic database, log, and (optionally) differential backups. The advantage of bulked-logged is that it is a minimally logged operation meaning that the chain of transaction logging is not broken so no additional backups need to be taken to insure recoverability yet a minimum number of transaction logs are generated thus reducing the load on the sever as well as the disk space required to produce the transaction logs. Only create non-null indexes This is an Example of a table from Analytics but the concepts presented here apply to EIM as well as ETL. At the of this note you will find a SQL script that will provide index statistics for all indexes in the database against which it is run. There is also a script for identifying the 10 largest tables and analyzing the indexes of those tables. 1 McBath, Frank. SQL Server Backup and Recovery. First Edition. Upper Saddle River, NJ. Prentice Hall PTR., 2002

3 Take a look at Table W_PERSON_SCD. Run DBCC SHOW_STATISTICS(W_PERSON_SCD, W_PERSON_SCD_M10) This should give a poor density 2 or nothing at all: Statistics for INDEX 'W_PERSON_SCD_M10'. DBCC execution completed. If DBCC printed error messages, contact your system administrator. To verify that all rows are one value and therefore null, run: select count(distinct(job_category_i)) from W_PERSON_SCD (nolock) The result should be one. This can further be verified by looking at the data itself, for example: select JOB_CATEGORY_I from W_PERSON_SCD (nolock) where JOB_CATEGORY_I is NOT NULL The result should be no rows returned. This would verify that the index is 100% NULL. Only create the non-null indexes when recreating indexes after a bulk data load. Since SQL Server stores nulls for null indexes this will make the database smaller and easier to manage. Note: There are always exceptions to the rule. The only time a poor index would be used is when a scan on that column is needed. For example, someone wanted to count all the rows that have NULL for JOB_CATEGORY_I. In that case the poor index would be used because it is faster to scan index pages than to scan data pages. However, this is a rare exception. Run the four tests below: The tests will illustrate the performance and space savings obtained by creating indexes in a Bulk_Logged mode as well as the value of only creating non-null indexes. FULL BULK_LOGGED CREATE ALL INDEXES Test 1 Test 3 CREATE NON-NULL INDEXES Test 2 Test 4 The test will establish a baseline for illustrating the performance gain for the recommed procedures. Run the following tests in a test environment or test database such as TempDB. Perform these steps with Full or Simple Recovery mode. Simple will record all transaction log information but will truncate the log on checkpoint See BOL for more information on recovery models. 1. Make a base line creating all the indexes on W_PERSON_SCD. Run the following at the top and again at the bottom of the script: 2 For more information on Density, consult BOL by Pasting the following URL into GO! URL: tsqlref.chm::/ts_dbcc_6jlf.htm. Also consult Inside SQL Server 2000.

4 Syntax: alter database <dbname> set recovery = <FULL SIMPLE BULK_LOGGED > alter database <dbname> set RECOVERY = FULL select getdate() set statistics io on... create indexes here... select getdate() 2. Set the database back to the original mode now. alter database <dbname> set recovery = SIMPLE (or original setting) 3. Record the number of IOs for all the indexes created and save them to a file. Record the actual run time of the and using the getdate() function as shown. 4. Examine the indexes on the table. Look for allocated_pages with real uniformity. This is the clue that they are 100% NULL or one value (ex. FLAG = 'Y'). select id, name, indid, dpages from sysindexes (nolock) where id=object_id( W_PERSON_SCD ) Look for uniform values. This would t to indicate indexes that are 100% of one value (ie. NULL): Examples of this are: obj_id index_id dpages Notice how all the entries are allocated_pages 5. Run the following to look at the index name: select name from sysindexes where indid = 20 and id = Now look at the columns being used in that index:

5 Sp_helpindex <index_name> 7. Now run the following query to verify that all rows are one value. select count(distinct(job_category_i)) from W_PERSON_SCD (nolock) 8. You can also verify by looking at the DBCC SHOW_STATISTICS. This will show a poor density or nothing at all: DBCC SHOW_STATISTICS(W_PERSON_SCD, W_PERSON_SCD_M10) Statistics for INDEX 'W_PERSON_SCD_M10'. DBCC execution completed. If DBCC printed error messages, contact your system administrator. 9. Once you have determined which indexes are NOT being used, then redo this test without the 100% NULL indexes. Get the times and IOs for the test. Now, redo both test (index creation time for all indexes and only non-null/usable indexes) in BULK_LOGGED Recovery mode. This will illustrate how much faster the CREATE INDEX is without intensive database logging. Capture the run times for the test and compare results. You should find that the bytes required (total dpages used in sysindexes) and index time creation is substantially faster with BULK_LOGGED recovery model and all non-null indexes removed. Supplemental Reading: Books Online Inside SQL Server 2000 SQL Server Backup and Recovery SQL Script for dbcc SHOW_STATISTICS on all tables set nocount on sysname sysname declare tablecursor cursor for select name from sysobjects where id > 100 order by id asc

6 open tablecursor fetch next from tablecursor while = 0 declare indexcursor cursor for select name from sysindexes where id = object_id(@tablename) and indid > 0 and indid < 255 order by indid asc open indexcursor fetch next from indexcursor = 0 print 'dbcc show_statistics (' + ',' + ')' dbcc show_statistics (@tablename,@indexname) fetch next from indexcursor close indexcursor deallocate indexcursor fetch next from tablecursor close tablecursor deallocate tablecursor

7 SQL Script for creating a stored procedure for analyzing the indexes of the 10 largest tables. /* exec poor_index */ alter proc poor_index as set nocount on if exists (select * from dbo.sysobjects where id = object_id(n'[dbo].[#temp_tbl_ind]') and OBJECTPROPERTY(id, N'IsTable') = 1) drop table #temp_tbl_ind if exists (select * from dbo.sysobjects where id = object_id(n'[dbo].[ #temp_idx]') and OBJECTPROPERTY(id, N'IsTable') = 1) drop table #temp_idx if exists (select * from dbo.sysobjects where id = object_id(n'[dbo].[#temp_col]') and OBJECTPROPERTY(id, N'IsTable') = 1) drop table #temp_col -- get the top 10 tables and indexes from the largest tables select top 100 id, indid, dpages, rows, name into #temp_tbl_ind from sysindexes (nolock) where -- indid = 0 or -- indid = 1 (status & 64=0) and indid < 255 and id > and --rows > 1000 order by dpages desc, id desc, indid

8 -- next, get just the tables select id, indid, name into #temp_idx from #temp_tbl_ind where indid > 0 order by id, indid -- then roll through the indexes and get the column name for the first column SELECT id, indid, index_col(object_name(id),indid,1) 'col_name' into #temp_col from #temp_idx if exists(select 1 from sysobjects where name like 'dist_vals') drop table dist_vals create table dist_vals(tbl sysname, indid int, first_col sysname, uniq bigint, rows bigint) if exists(select 1 from sysobjects where name like 'qry') drop table qry create table qry(sql_txt varchar(500)) -- build the query to get the data on the usefullness of the index first column insert into qry select 'insert into dist_vals select '+char(39)+object_name(id)+char(39)+',' +char(39)+cast(indid as varchar(3))+char(39)+ ','+char(39)+col_name+char(39)+ ', count(distinct(isnull('+col_name+',1))),'+ 'count(isnull('+col_name+',1)) from ['+object_name(id)+'] (nolock)'-- +char(13)+' '+char(13) from #temp_col

9 -- select * from qry truncate table dist_vals -- run the queries and log the data nvarchar(500) DECLARE sql_cursor CURSOR FOR SELECT sql_txt FROM qry OPEN sql_cursor FETCH NEXT FROM sql_cursor = 0 BEGIN -- exec FETCH NEXT FROM sql_cursor END CLOSE sql_cursor DEALLOCATE sql_cursor -- analyze the results select 'poor index',tbl 'table_name',name 'index_name', first_col, uniq 'unique_values_in_column',a.rows 'rows_in_table',convert(numeric,dpages*8*1024)'bytes_used' from dist_vals a, sysindexes b where (a.uniq=1 or a.uniq=2) and object_id(tbl)=b.id and a.indid=b.indid and a.rows > 0

General Concepts SQL SERVER REDUCING INDEX FRAGMENTATION

General Concepts SQL SERVER REDUCING INDEX FRAGMENTATION Article: Reducing Index Fragmentation Date: 08/03/2012 Posted by: HeelpBook Staff Source: Link Permalink: Link SQL SERVER REDUCING INDEX FRAGMENTATION General Concepts When you perform any data modification