Efficient Bulk Deletes for Multi Dimensional Clustered Tables in DB2

Size: px

Start display at page:

Download "Efficient Bulk Deletes for Multi Dimensional Clustered Tables in DB2"

Letitia Porter
5 years ago
Views:

1 Efficient Bulk Deletes for Multi Dimensional Clustered Tables in DB2 Bishwaranjan Bhattacharjee, Timothy Malkemus IBM T.J. Watson Research Center Sherman Lau, Sean McKeough, Jo-anne Kirton Robin Von Boeschoten, John P Kennedy IBM Toronto Laboratories bhatta@us.ibm.com

2 Bulk Deletes (aka Mass Delete, Rollout) Frequent in Data Warehouses Often multi dimensional Maintenance windows for it are slowly diminishing Customers expect system availability when rolling out An online, multi dimensional rollout mechanism is very important for a db engine 2

3 Major Issues In Rollout Response time of Rollout and Rollback : Maintenance windows shrinking Locks : Lock escalation a problem Bad for concurrency Impacts response time Logging : Complicates applications Have to use Fetch First n Rows Only (FFnRO) Bad for concurrency Impacts response time Secondary Index Update : Severely impacts response time Consumes resources like log space, CPU Results in lots of synchronous IO 3

4 Road Map Multi Dimensional Clustering (MDC) in DB2 MDC Rollout Performance Evaluation of MDC Rollout Related Work Conclusion 4

5 MDC Motivation: Multidim ensionality clustering index on region region 997, Mexico, 997, M exico, 2 997, M exico, 2 998, M exico, 2 table 997, Canada, 997, Canada, 2 997, Canada, 2 998, Canada, 2 index on year item Id year(orderdate) S ingle dim ensional index clustering is inadequate. "E fficient Query Processing for M ulti-dimensionally Clustered T ables in DB2.", V LD B "M ulti-dimensional Clustering: A New Data Layout Scheme in DB2", SIGMOD "Automating the design of multi-dimensional clustering tables in relational databases", V LD B "Predicate Derivation and M onotonicity Detection in DB2 UDB", ICDE "Performance Study of Rollout for M ulti Dimensional Clustered T ables in DB2", EX PDB

6 MDC Table Syntax CREATE TABLE MDCTABLE ( orderdate DATE, Nation CHAR(25), itemid INT,... ) ORGANIZE BY( orderdate, Nation, itemid ) CREATE TABLE MDCTABLE2 ( orderdate DATE, Nation CHAR(25), itemid INT, orderyear generated always as ((INTEGER(orderDate)/0000),... ) ORGANIZE BY( orderyear, Nation, itemid ) * no need to plan for or define explicit range boundaries 6

7 How MDC Works : Blocks, Cells, Slices nation Mexico Canada Key for Canada: Keypart 4,0 2,0 48,0 52,0 76,0 80,0 itemid 2,0 00,0 26,0 6,0 60,0 292,0 444,0 8,0 450,0 304, year(orderdate) Blocks Pages (2-256) of records BID (block id) = <first pool relative page of block, 0> Block Indexes Structurally, B-Tree indexes.. Key has list of BIDs Small compared to RID indexes Cells Canada 2,0 4,0 6,0 2,0 8,0 48,0 52,0 76,0 80,0 00,0 60,0 26,0 292,0 304,0 444,0 450,0 0 or more blocks Entry in composite block index when cell has blocks 7

8 MDC Supports Additional RID Indexes region 997, Mexico, 997, Mexico, 2 997, M exico, 2 998, Mexico, 2 997, Canada, 997, Canada, 2 997, Canada, 2 998, Canada, 2 item Id year(orderdate) RID ship status RID discount RID ship mode Please see previous papers for more details on MDC and MDC query performance 8

9 Road Map Multi Dimensional Clustering (MDC) in DB2 MDC Rollout Performance Evaluation of MDC Rollout Related Work Conclusion 9

10 How MDC Delete Works Only Block Locks For Full Cell Deletes Block Map MDC non MDC Number of locks Locks for sample rollout Rid Index (optional) Table Dimension Block Index Blocks Log Records 0

11 MDC Immediate Rollout (GA DB2 V8.2.2) Faster DELETE along cell or slice boundaries Compiler determines if DELETE statement qualifies for ROLLOUT No need for a specialized statement or command Example: MDC table with 3 dimensions (nation, year, product ID) DELETE FROM table WHERE year = 992 and product_id = Before After Mexico Mexico Canada Canada

12 How MDC Immediate Rollout Works Record Producer (2,0) Composite Block Index (8,0) (20,0) (22,0) (24,0) (36,0) (38,0) (40,0) (42,0) (48,0) StoreID ShipDateId BIDS (Page,Slot) (20,0)(36,0)(38,0)(50,0) (22,0)(24,0)(40,0) (48,0)(54,0)(78,0)(90,0) (2,0)(8,0)(42,0)(56,0) (50,0) (54,0) (56,0) Result : Improved response time (78,0) (90,0) 2

13 How MDC Immediate Rollout Works In A Block Block Map MDC Rollout MDC Delete non MDC Delete Rid Index (optional) Block Log Records Dimension Block Index Impact: Log space savings Improved response time Same transaction does not reuse delete space before commit 3 Response Time (in seconds) SR Dimension SR 3 Dimension 4 52

14 MDC Immediate Rollout Working Summary Performance improvements by avoiding per-row logging and by path length reduction Clears the slot directory on each page in the block Writes one small log record per page - rather than a log record per row (containing row data) Secondary indexes still updated synchronously (immediately) Must scan the rows (as usual) to update each index to remove keys Index logging is unchanged 4

15 Impact of RID Index Cluster Ratio On Immediate Rollout 0000 MDC Rollout MDC Delete Response Time (in seconds) Index Page Reads Physical Logical No Rid Index Receiptdate Partkey No Rid Index Receiptdate Partkey Index Cluster Ratio Receiptdate : 38% Partkey : 4% 5

16 Comparison Of Logging Space Used In Immediate Rollout Number of RID Indexes Rollout (in Bytes) Delete (in Bytes) % Gain

17 MDC Deferred Rollout Aims in DB2 Viper 2 Response time of Rollout with rid indexes to improve significantly Index page IO to reduce significantly Index log space requirement to come down significantly Will simplify application logic. No need for FFnRO Table scanners and Block Index scanners will not be impacted Rid index scanners could be slowed down slightly 7

18 MDC Deferred Rollout HLD LOG AIC Mexico Canada TranCB TCB Rid Index Rid Rollout index will cleanup not update to be performed any existing by rid a asynchronous indexes background Rollout cleaner to delete the table records and update the block indexes. Rid index scanners will check an in memory data structure to skip One rolled cleaner out per blocks rid index. Will become active when block 8 needs cleaning AIC will write log record for a index leaf and commit work one time per leaf extent AIC will prefetch index leaf pages AIC will write its resume position and log it. It can restart from that spot

19 Rollout Block Bitmap (ROBB) Design Considerations Fast Probes Memory Considerations Commit/Rollback Memory Restrictions ROBB Operations Probe (Query, AIC) Set and Clear (Rollout, AIC) Merge and Subtract (Commit, Rollback) Recreate (Recovery) 9

20 Example Rollout Block Bit Map (ROBB) Design 64 Bit ROBB Lvl Should fit in register = 892 Bit ROBB Lvl 2 Should fit in the data cache 892 Pointers in ROBB Lvl Sub bitmaps

21 Performance Evaluation Of Rollout Setup similar to that of some customers who run ERP over MDC tables Experimental Setup DB2 UDB Viper 2 64 bit AIX IBM C4 6 GB of main memory 4 x 453 MHz Table : 2 Dimensional MDC Fact Table million rows in pages Indexes : 9 RID Indexes Unique RID index of 3276 pages and 8 Non Unique RID index of ~ 4700 pages each 3 Indexes with < 5% clustering, 2 Indexes with ~ 35% clustering and 4 Indexes with > 95% clustering 3 Block Indexes 2

22 Response Time Relative Response Time (In %) Delete Rollout (Immediate) Rollout (Deferred) + AIC Rollout (Deferred) 0 0.3%.5% 3% 30% 97% Percentage of table deleted 22

23 IO Waits Incurred Relative IO Waits (In %) Delete Rollout (Immediate) Rollout (Deferred) 0 0.3%.5% 3% 30% 97% Percentage of table deleted 23

24 Index Logical Reads Index Logical Reads Millions Delete Rollout (Immediate) Rollout (Deferred + AIC) 0.3%.5% 3% 30% 97% Percentage of table deleted 24

25 Log space consumption Relative Log Space Consumption (In %) Delete Rollout (Immediate) Rollout (Deferred) 0.3%.5% 3% 30% 97% Percentage of table deleted 25

26 Workload Performance (start clock, delete, query, stop clock) Immediate Rollout Deferred Rollout time table scan block iscan fetch rid iscan only rid iscan fetch rid iscan fetch 26

27 Related Work Horizontal, record based with rid index update one at a time. Not optimized for bulk delete Detach in Range Partitioning Generally needs queries to drain Special syntax (attach, detach) Deletes on B+ Tree Tables Rid indexes could be deleted horizontally, in parallel in some implementation Online Bulk Deletion, Lilja et al, ICDE 2007 Optimize B+ Tree Table deletes Vertical Deletes Efficient Bulk Deletes in Relational Databases, Gartner et al, ICDE 200 Assumes table will be x locked and indices would be offline for the delete Addresses response time but not locking or logging Deferred Maintenance Differential Files: Their Application to the Maintenance of Large Databases, Severance et al, ACM TDBS 97 Differential file used as a book errata list to identify and collect pending record changes When Differential file gets large, reorganization will incorporate changes into the database 27

28 Conclusion MDC Rollout provides a mechanism for mass delete of data which Is able to significantly reduce the response time of mass deletes compared to previous deletes in DB2. Even when a lot of badly clustered rid indexes are defined on the table While consuming significantly lower amount of system resources (locking, logging, IO) compared to a previous delete in DB2 In this talk we described how these challenges were addressed 28

Robin Von Boeschoten, John P Kennedy IBM Toronto Laboratories Markham, Ontario, Canada

Robin Von Boeschoten, John P Kennedy IBM Toronto Laboratories Markham, Ontario, Canada Efficient Bulk Deletes for Multi Dimensional Clustered Tables in DB Bishwaranjan Bhattacharjee, Timothy Malkemus IBM T.J. Watson Research Center Hawthorne, NY, USA {bhatta, malkemus} @us.ibm.com Sherman