Enhance Database Performance and Scalability

Size: px

Start display at page:

Download "Enhance Database Performance and Scalability"

Hilda Pearson
5 years ago
Views:

1 Enhance Database Performance and Scalability Progress OpenEdge: Database Table Partitioning Workshop Paul Koufalis Progresswiz informatique

2 Who Is This Paul Koufalis? Where s Rich Banville? Progress DBA and UNIX sysadmin for 20 years Wide spectrum of clients Financial services providers, distributors, manufacturers, retailers $10M to $Billions in revenue DBs from 1Mg to 100 s of Gb 10 to almost 2000 concurrent users 2

3 Agenda Getting Connected to Your Virtual Machines Introduction to Table Partitioning LABS! LABS! LABS! I Changed My Mind: Changing/Deleting Geek BONUS: ABL API 3

4 Connecting to Your VM

5 LAB 0 Connecting to Your VM First thing s first: make sure you can access your private virtual machine Requirements: Internet access (the VMs are in the cloud) Web Browser to access OpenEdge Management SSH tool like putty 5

6 Introduction

7 We will not be going into every nitty-gritty detail of partitioning No time. You will listen to me for 3h instead of doing labs Don t miss Getting Started with Table Partitioning Tuesday, October 7, 2014, 11:00 AM 12:00 PM Havard Danielsen, Software Engineer; Rich Banville, Software Fellow Table Partitioning Application and Design Wednesday, October 8, 2014, 1:30 PM 2:30 PM Rich Banville, Software Fellow

8 The Steps to Table Partitioning 1. Identify tables 2. Prepare a. Type II Storage Areas b. Recid/rowid usage c. Aligned field assignments d. Partition aligned (local) indexes 3. Plan strategy a. Dump / Load 4. Enable partitioning a. Database b. Index rebuild c. Tables / indexes / lobs 5. Define policies 6. Split out existing data 7. Truncate / de-allocate initial partition b. In place migration 8

Identify Tables Stable data logically grouped by Chronological events (range partitions) Discrete list of values (list partitions) Sub-partition partitioning Partitioning the same table according to

9 Identify Tables Stable data logically grouped by Chronological events (range partitions) Discrete list of values (list partitions) Sub-partition partitioning Partitioning the same table according to the values of more than one column Table / index maintenance too high Operational time Data availability See white paper Selecting Tables for Partitioning : community.progress.com/technicalusers/f/18/t/9 294.aspx 9

10 Select Partition Columns Best if application queries on partitioned-aligned columns Less use to partition by State if no one ever queries by State Partition columns should not change [often] Otherwise may require the record to change partitions Partition column(s) are leading component(s) of an index Partition-aligned local index Consider adding such an index if it does not exist REMEMBER: Every column value must fit in a partition There is no default partition 10

11 Prepare [Optionally] create Type II S.A for each partition Decide where to put each physical extent Example: Historical data on cheap SATA drives Clean your data No UNKNOWN values in partition columns Clean your code Assign partition columns in same ASSIGN when creating records Avoids default value assignments followed by record to move to another partition Watch out for ROWID() Will change if record moves to a different partition RECID() has NOT been updated to support TP 11

12 Example: List + Range 9 partition example Index #1 Index #2 9 B-trees supporting 3 index definitions Order Table Index #2 Sub-partition by region & orderdate w/in region Local indexes on orderdate, name Local indexes on orderdate, S-rep AMER 12/31/2012 EMEA 12/31/2012 APAC 12/31/2012 AMER 12/31/2013 EMEA 12/31/2013 APAC 12/31/2013 AMER 12/31/2014 EMEA 12/31/2014 APAC 12/31/2014 Global indexes span partitions Indexed field need not be partition aligned Index Components Partition Aligned Index #1 (local) {Region, Order-Date, Name} YES Index #2 (local) {Region, Order-Date, S-rep} YES Index #3 (global) {Cust-num} NO 12

13 Migration Strategy Dump & Load Dump data Delete data (truncate area hopefully) Define partitions Load data May be faster than in-place migration BUT data will be unavailable for longer Best during a maintenance window Obligatory if you want to change an already partitioned table 13

14 Migration Strategy In Place Composite partition created in area where data resides Data is logically segregated (no physical changes) Ready to be split into physical partitions AKA Partition 0 AMER Order Table 06/30/2014 EMEA 06/30/2014 APAC 06/30/

15 Migration Strategy In Place Split out data Example: 3 of 9 partitions have been split out of composite partition Order Table AMER 06/30/ /30/2014 AMER 12/31/2014 Local Index #10 Partition #1 10 Composite Index #10 Partition #0 10 EMEA 06/30/ /30/2014 EMEA 12/31/2014 Local Index #10 Partition #2 10 APAC 06/30/ /30/2014 APAC 12/31/2014 Local Index #10 Partition #

16 Enable Partitioning Command line proutil <db> -C enabletablepartitioning proutil <db> -C enabletpidxbld Or OpenEdge Management/Explorer ONLY can enable TP Online requires exclusive schema access 16

17 Define Policies Use OpenEdge Management/Explorer Advanced users can use new set of ABL classes 17

18 Split Out Existing Data Identify created partition as a split target Move data into target partitions proutil <db> -C partitionmanage split table <name> partition <name> composite "initial useindex <name> recs <number> Recovery of operation restarts where it left off Online operation with full access to non-split data New split transitional state for partitions Multiple concurrent operations allowed OpenEdge Replication fully supported 18

19 Truncate / Deallocate Initial Partition proutil <db> -C partitionmanage truncate table <tname> composite initial recs <#recs per txn> [ deallocate ] Order Table AMER AMER AMER 12/31/ /30/ /31/2014 EMEA 12/31/2013 EMEA 06/30/2014 Deallocated Partition 0 EMEA 12/31/2014 APAC APAC APAC 12/31/ /30/ /31/

20 LAB 1: Simple List Partitioning

21 Lab 1 Simple List Partition Let s do the first lab together We are going to partition the customer table by country Open up your Lab Guide 21

22 LAB 2: Simple List Partition Continued

23 Lab 2 Simple List Partition Continued Try it yourself This time we will physically partition data to distinct storage areas 23

24 LAB 3: List + Range Partitions

25 Lab 3 List + Range Partitions Split Order by Region and by OrderDate (calendar year) Region = LIST partition Year = RANGE sub-partition REMEMBER: Both the LIST and the RANGE must cover ALL the keys in the partition columns! 25

26 Merge Truncate - Idxbuild

27 Tools Four new proutil options PARTITIONMANAGE SPLIT PARTITIONMANAGE MERGE PARTITIONMANAGE TRUNCATE TPIDXBUILD We already saw split 27

28 PARTITIONMANAGE TRUNCATE Truncate and optionally deallocate a partition proutil <db> -C partitionmanage truncate table <name> partition <name> composite initial [recs <number>] [deallocate] Used to free storage space of a partition DELETES THE DATA CAREFUL: [recs] governs number of deletes per transaction Affects lock table usage and BI size 28

29 PARTITIONMANAGE TRUNCATE 29

30 DB Analysis Truncate CHAIN ANALYSIS FOR AREA "Order EMEA 2013": 67 RM CHAIN ANALYSIS Number of Blocks Object Type Object Partition/Tenant/Group Table PUB.Order:13 P:Order_EMEA_ :5 AREA "Order EMEA 2013": 67 BLOCK ANALYSIS block(s) found in the area. Current high water mark: free block(s) found in the area 508 record block(s) found in the area 0 index block(s) found in the area 0 empty block(s) found in the area 2 object block(s) found in the area 2 cluster list block(s) found in the area 1 object list block(s) found in the area 2 cluster map block(s) found in the area 30

31 DB Analysis Truncate and Deallocate CHAIN ANALYSIS FOR AREA "Order EMEA 2013": 67 RM CHAIN ANALYSIS Number of Blocks Object Type Object Partition/Tenant/Group AREA "Order EMEA 2013": 67 BLOCK ANALYSIS block(s) found in the area. Current high water mark: free block(s) found in the area 0 record block(s) found in the area Notice no more RM blocks 31

32 Clean Up Time (or Is It?) Let s look at the _StorageObject table (area 67 from htplab.st) for each _StorageObject where _area-number = 67: displ _storageobject with 1 col no-box Object-type: 1 Object-number: 13 (This is the Order Table) Object-associate-type: 2 Object-associate: 39 (This is the Primary Index) Area-number: 67 Object-attrib: 264 Object-system: 0 Create-Limit: 150 Toss-Limit: 300 CollationId: 0 Object-state: 2 32

33 Delete the Partition In Partition Policy Manager Details Delete No more _StorageObject record Now you can remove the physical area 33

34 Free Up the Space 34

35 CAREFUL YOU MAY BE TEMPTED to run proutil htplab C truncate area DO NOT. You have been warned (KABOOM!) 35

36 PARTITIONMANAGE MERGE Merge two or more range or list-range partitions proutil <db> -C partitionmanage merge table <name> partition <name start> partition <name end> [ partition <name target> ] useindex <name> recs <number> Specify the start and end partitions Optionally specify a new target partition All partitions between the start and end will be merged 36

37 PARTITIONMANAGE MERGE $ _proutil htplab -C partitionmanage merge table order partition "Order_AMER_ " partition "Order_AMER_ " partition "Order_AMER_ " recs 1000 BEGIN: Merge Operation For Table order... Do you want to continue (y/n)? Partition Order_AMER_ [2]: records removed: Partition Order_AMER_ [2]: has been deleted. Partition Order_AMER_ [1]: records removed: Partition Order_AMER_ [1]: has been deleted. Partition Order_AMER_ [3]: Total records merged: END: Merge Operation For Table order was successful. 37

38 PARTITIONMANAGE MERGE Refresh Partition Policy Detail Screen Partitions for 2013 and 2014 are gone Check _StorageObject table for AMER areas (see htplab.st for area #s) Only area 51 "Order AMER 2015 and 52 "Order AMER 2015 Idx have objects You can delete unused storage areas using prostrct remove 38

39 TPIDXBUILD Index rebuild by partition proutil <db> -C tpidxbuild table <name> [ area <name> index <name> ] partition <name > composite initial [ -T <tmpdir> -SS <srt-file> ] [ -TB # ] [ -TM # ] [ -B # ] [ -SG # ] [ -pfactor # ] Can run multiple tpidxbuild in parallel Index not available during build Index does not become available until older clients logout TPIDXBUILD notifies you and prompts you to wait or cancel 39

40 LAB 4: Merge, Truncate & TPIDXBUILD

41 Lab 4 Merge, Truncate & TPIDXBUILD Merge Order Partitions for AMER Truncate Order Partition 0 Multiple, concurrent partition index rebuilds 41

42 BONUS: ABL API OO Classes

43 ABL API OOC That s a lot of TLAs (Three Letter Acronyms) Documentation: OpenEdge Development: Programming Interfaces Package OpenEdge.DataAdmin 43

44 The DataAdminService Before doing anything, you must instantiate a new DataAdminService using OpenEdge.DataAdmin.DataAdminService from propath. define variable service as DataAdminService no-undo. service = new DataAdminService("htplab"). Remember to delete the object at the end delete object service no-error. 44

45 Create New Partition Policy using OpenEdge.DataAdmin.IPartitionPolicy from propath. define variable partitionpolicy as IPartitionPolicy no-undo. partitionpolicy = service:newpartitionpolicy("orderregnyear"). ASSIGN partitionpolicy:table = service:gettable("order") partitionpolicy:hasrange = yes partitionpolicy:defaultdataarea = service:getarea("order") partitionpolicy:defaultindexarea = service:getarea("order Idx") partitionpolicy:defaultlobarea = service:getarea("order") partitionpolicy:defaultallocation = "None". /* Add fields to the fields collection */ partitionpolicy:fields:add(partitionpolicy:table:fields:find("region")). partitionpolicy:fields:add(partitionpolicy:table:fields:find("orderdate")). /* Add RegnDate to the local index collection */ partitionpolicy:indexes:add(partitionpolicy:table:indexes:find("regndate")). 45

46 Create New Partition Policy Detail using OpenEdge.DataAdmin.IPartitionPolicyDetail from propath. partitionpolicydetail = service:newpartitionpolicydetail("order_emea_2013"). /* Add the new partitionpolicydetail to the Details collection. */ partitionpolicy:details:add(partitionpolicydetail). ASSIGN partitionpolicydetail:issplittarget = no partitionpolicydetail:description = "" partitionpolicydetail:defaultdataarea = service:getarea("order") partitionpolicydetail:defaultindexarea = service:getarea("order Idx") partitionpolicydetail:defaultlobarea = service:getarea("order"). /* Set the values for the policy detail */ partitionpolicydetail:setvalue("emea",1). /* Region as character */ partitionpolicydetail:setvalue(12/31/2013,2). /* OrderDate as date */ service:updatepartitionpolicy(partitionpolicy). 46

47 Split the Data using OpenEdge.DataAdmin.Util.PartitionSplitUtility from propath. define variable split as PartitionSplitUtility no-undo. split = NEW PartitionSplitUtility(partitionPolicyDetail). service:executeutility(split). 47

48 Merge and Truncate Not yet sorry But there are A LOT of cool things you CAN do: Manage storage areas and physical extents Manage authentication systems, domains and users Manage schema changes Manage Tenants in a MT DB 48

49 Import and Export partitionpolicy:export("order.json"). {"partitionpolicies": [ { "name": "OrderRegnYear", "schemaname": "PUB", "tablename": "Order", "table_url": "\/schemas\/pub\/tables\/order", "description": null, "url": "\/partitionpolicies\/orderregnyear", "hasrange": true, "type": "List-range", "partitionpolicydetails_url": "\/partitionpolicies\/orderregnyear\/partitionpolicydetails", "defaultdataareaname": "Order", "defaultdataarea_url": "\/areas\/order", "defaultindexareaname": "Order Idx", "defaultindexarea_url": "\/areas\/order%20idx", "defaultlobareaname": "Order", "defaultlobarea_url": "\/areas\/order", "defaultallocation": "None" } ]} 49

50 Q&A

51 Visit the Resource Portal Get session details & presentation downloads Complete a survey Access the latest Progress product literature

Horizontal Table Partitioning

Horizontal Table Partitioning Dealing with a manageable slice of the pie. Richard Banville Fellow, OpenEdge Development August 7, 2013 Agenda OverviewFunctionalityUsageSummary 2 Agenda 1 Overview 2 Feature