Can you really change DB2 for z/os Schemas online?

Transcription

1 Can you really change DB2 for z/os Schemas online? Steve Thomas BMC Software Session Code: B17 Thursday 11 th November 2010 at 11:15 Platform: z/os Ever since I started working with DB2 back in 1989 with Version 1.3, DBAs have needed the ability to change database structures with minimal or zero interruption to the Business Service. Back then almost all you could do was to add new columns to the end of a Table but fortunately the situation has improved significantly over the years and many more schema changes can now be made online. However, just recently I am speaking to more customers who have needed to change Tablespace characteristics such as DSSIZE or SEGSIZE, or to convert objects to the new UTS format introduced in DB2 9 for z/os. Currently all these changes still require you to Unload and Load the data, leading to an extended business outage. This presentation explains what you can change online and, perhaps more importantly, what you can't. We also discuss some of the other implications when making a Schema change, such as the impact on Packages, Triggers and Dependent Views. We will discuss a vendor neutral method for reducing the outage required when making changes that cannot be accomplished with an Alter Statement. The session will discuss features available in versions up to and including DB2 9 for z/os in detail. I will also cover DB2 10 for z/os, but before this is Generally Available I will only be able to discuss information that is in the public domain due to non-disclosure agreements in place as part of the Vendor Early Support programme. Steve is a Principal Consultant at BMC Software, and supports customers across the UK, Northern Europe, Greece, Turkey and Israel. He has been a DB2 specialist since 1989 and an IBM Information Champion since Steve has presented on a wide range of topics at events across the UK and in Europe. He helps organize the UK DB2 RUG and is a member of the IDUG EMEA Conference Planning Committee. 1

2 What we ll cover today Things to consider when changing schema definitions Online Schema Support in DB2 for z/os What can t you change online? What can you do if you need to implement one of these? What s coming in DB2 10 for z/os (subject to NDA) Slide handouts issued before the Conference cannot include full details on the final section due to Non-disclosure agreements with IBM but will be updated after the GA date 2 Even when the change you need to make it supported changing a DB2 schema online doesn t just involve issuing an Alter Statement. There are many other things you need to consider such as co-ordinating your changes with the Application and the impact on your Plans and Packages including the Auto Rebind process. We ll start by covering some of these items before moving onto discuss what changes can be made today using Generally Available versions of DB2 for z/os. In many ways knowing what you cannot do is actually more important so that you can influence the database design to improve your future flexibility so as well as covering what you can do we will also cover what you can t. We will next briefly describe a method which allows you to make changes while minimizing the outage to your business services because not all changes can be predicted in advance or supported by Online Schema DDL should you need to make them. This section is intended to be vendor neutral so you should be able to use whatever tools are available to assist. I won t have time to go into this area in any great detail, but the Bibliography at the end includes some papers which focus on this area in much more detail. The final topic of the session will cover the improvements that are being introduced in DB2 10 for z/os which is currently in it s Early Support Programme or Beta test. Before the GA date I will only be able to discuss information that has been disclosed in public by IBM due to nondisclosure agreements. However as soon as the new release becomes GA I will release an updated version of the presentation which provides more details in this area. If you can t find a copy on the web you can obtain an updated copy by contacting me using the address on the closing slide. 2

3 What is OLS or Online Schema Change? The ability to change characteristics of DB2 objects, without a drop/recreate, thus improving availability Before V8 most object changes required the following: Analyze the impact of the changes; resolve all conflicts; extract object definitions and retain authorizations Unload the data Drop & Recreate the object with the new definition Re-create dependent objects such as indexes, views, triggers etc. Re-establish authorizations for the newly created objects Reload the data; copy, collect statistics, check RI as needed Rebind packages and plans Big improvements in V8 & DB2 10, smaller ones in DB2 9 3 If you hear me talking about a UDCL process what I m talking about is the old way of making changes a Schema. It stands for Unload Drop Create and Load. The whole purpose of OLS is to avoid having to use UDCL due to the disruption, and the cost to the business, that it causes. Although OLS capabilities have been improving for a long time, the first big step forwards was made in DB2 V8 with the ability to change data types and the introduction of Table Based partitioning. This broke the link between partitioning and clustering for the first time, allowed us to add and rotate partitions amongst other things. Until this feature, changing Partitioned Tables, which are usually your largest objects was a hugely disruptive process. DB2 9 added some useful features such as the ability to rename a column, but on the whole the changes weren t as significant as those in V8. Although it s early days it looks like DB2 10 is another very significant release, introducing the concept of a Deferred ALTER and allowing some previously impossible features such as DSSIZE and SEGSIZE to be changed online. 3

4 What sort of things might we want to change? The obvious - Table, View and Index Changes New, renamed or amended columns Removing columns is more difficult - is this really needed? The slightly less obvious often physical changes Usually invisible to the application and require no code changes Partitioning alterations limit keys, Table Based partitioning Sizing changes such as DSSIZE, SEGSIZE, PIECESIZE Conversion to UTS (likely to become more common) MEMBER CLUSTER The insidious apparently small changes which can have a big impact Changes to UDFs, UDTs and Triggers 4 The most obvious changes we might want to make are usually to the Table and Index structures themselves, such as adding or renaming columns and changing data types. On the whole most of these are achievable online today. The primary exception is that removing a column still requires UDCL, even in DB2 10 as far as I can determine. However, provided your applications are coded correctly and don t use things like SELECT * (in tables or views) is this really necessary? Disk space is cheap and compression (especially if you blank the column) means the effect should be negligible and may not be worth the outage. Most physical changes fit into the less obvious category because in most cases no application changes will be required. Partitioning changes fit intro this category. However, the most disruptive changes over recent years have been changes to DSSIZE, SEGSIZE and PIECESIZE for Indexes. This is where DB2 10 looks like it will really help. Many of the final category of changes are actually relatively simple to make as far as a DB2 DBA is concerned but they can often have a much larger impact in other areas, notably on the Applications using the objects. Typical examples are alterations to UDFs and UDTs which can be implemented with a simple ALTER, but anything using them will need to be examined and most likely changed. Triggers are slightly more tricky as you can t alter them, - these will be discussed later on slide xxx 4

5 Every ALTER can have an Impact Even immediate changes require exclusive use of the Catalog & Directory and can have a performance impact Example Adding a column to a Table or changing data type Advisory Status should be treated as rather more than that! Some require a Utility (usually REORG) to action Example Alter Tablespace COMPRESS YES Others make the object inaccessible until after a utility Example Alter Index changing PIECESIZE (DB2 9 for z/os) Index is placed into PSRBD Status A few require at least one object to be stopped Altering Bufferpool when Data Sharing Adding new workfiles to the Temporary Database 5 Even when a change is supported by OLS, you need to remember that every ALTER is going to have some sort of impact on your system. DB2 requires exclusive control of the Catalog and Directory entries for the object being changed for a very brief moment. Even though this is usually for a very short duration the requirement might as well be huge if it s impossible to obtain due to heavy BIND or Dynamic SQL activity against the Catalog. It s not just the objects you re changing that are impacted. When using DB2 9 Page locks are used which will naturally have an impact other objects defined on the same page. Once an OLS change has completed any affected Packages are likely to be invalidated and as we shall see the resulting Access Path obtained via the Auto Rebind process is quite likely to be suboptimal in terms of performance. In addition to the direct impact many ALTER commands place an object into Advisory Reorg Pending (AREO*) status. Which can have a significant performance hit. AREO* should really be called Reorg immediately if you know what s good for you! Most sites should plan an Online Reorg as soon as possible after making any of these changes otherwise they may be unable to live with the performance degradation, particularly if the table is heavily used. It sounds obvious but wherever possible, all OLS changes should be scheduled during times of minimal activity. Ideally concurrent batch execution should be avoided. Short running transactions can run; at most, a few might experience a short delay which is still much better than the significant outages caused by the older UDCL strategies. 5

6 Even if an ALTER is effective immediately Need to Synchronize Application Code Changes Data Type changes can often be accommodated (temporarily) New data types have to be compatible and large enough Numeric changes more difficult due to Indexes Renamed columns and tables usually need code changes Table Changes may impact other objects Dependent Views Plans, Packages and Dynamic Statement Cache Referential Integrity, Check Constraints or other Data Validation Triggers Stored Procedures, UDFs & UDTs 6 In earlier releases of DB2 if a Data Type wasn t the same as what was defined in a program this was non-indexable that was changed but there s still a data conversion involved. 6

7 Dependent Views needs more research Handled well by DB2 but beware SELECT * When column is altered in a base table DB2 re-evaluates any dependent views during the alteration BUT it does not replace the internal structure DB2 will then regenerate the view each time it is referenced Potential performance degradation depending on size of view and how often it is referenced Changes to any column referenced in a view invalidates all plans, packages, and dynamic statements that are dependent on the view This is one of the areas where a DB2 Administration tool (from whatever Vendor or home grown) comes in handy 7

8 Package Invalidation & Auto Rebind Occurs when changing items bound in static SQL or in Dynamic SQL stored in Statement Cache In most instances the next invocation causes an Auto Rebind Care required with Auto Rebind Access path can switch the index may be unavailable You only get one shot at a Rebind If it fails you get an INOPERATIVE Plan or Package Watch for timeouts in catalog after a change to a busy object Were Stats OK? Were all the objects available? Add REBIND explicitly to any activity that invalidates packages 8 Package invalidations happen when DB2 has to change processing details that are bound up in static SQL. The same process also occurs in Statements bound into the Dynamic Statement Cache, although these are simply invalidated and rebound automatically at the next execution. Sometimes, this happens when we don t expect it. For instance: ALTER TABLE ADD FOREIGN KEY ON DELETE CASCADE or SET NULL (not RESTRICT) Not only invalidates the parent s packages, but also the packages on any tables that cascade down to that parent, so the impact may be a lot larger than you were expecting. You may think that Auto Rebind will take care of things and it certainly helps keep the show on the road. However, it cannot be relied on in every case, or certainly if you re looking to retain optimal performance. When a large number of Packages have been invalidated on a busy system there could be heavy REBIND activity immediately after the change. You only get one shot at an Auto Rebind, and if this attempt fails due to locking contention on the Catalog for instance the Package will be marked inoperative and will require a manual REBIND. Other less obvious problems can occur when Catalog Statistics have been invalidated due to a change or when an object has been marked rebuild pending because the Auto Rebind may occur before the object is available or when it has missing statistics, so the access path obtained is quite likely to be different to what you have designed or were expecting. Best practice involves scheduling a REBIND after an OLS change once you know that all the objects are available again with correct Catalog Statistics. 8

9 Triggers Triggers are database objects which have packages They cannot be Altered (even in DB2 9) If Table is dropped any Triggers must be recreated May need to place tables in read-only to ensure integrity Ensure you retain firing order for multiple triggers Text in triggers can include other Database objects Including calls to Internal and External procedures It can be very hard to track and identify these Does the change you re making impact a Trigger? 9 9

10 Referential Integrity and Constraints Adding a foreign key turns on check pending No SQL data manipulation statements can be issued Unless RULES=STD when checked immediately Adding RI constraints can invalidate plans and packages When using CASCADE or SET NULL (not RESTRICT) This happens for all tables cascading down to the parent! Check pending set if there is a potential violation of a constraint (load, recovery, etc.) Could use a trigger instead 10 The use of Referential Integrity or not is as controversial as ever and the arguments vary as much as there are number of DBA s. Having RI can cause outages for a 24x7 application. Having RI can place tables in check pending and invalidate plans and packages. It is a resource that can cost CPU time and additional locking. Not having RI can place your data in jeopardy and cause downtime with bad data and data that is out of synch. Know your data and our applications. Use RI judicially and wisely. Remember that ALTER TABLE ADD FOREIGN KEY ON DELETE CASCADE or SET NULL (not RESTRICT) invalidates not only the parent s packages but all the tables that cascade down to that parent 10

11 Options for embedded Data Validation Check Constraints Limited Syntax, Best for performance Changing sets CHKP unless using RULES=STD Referential Constraints Only to check existence of key Can change checked values easily using SQL Requires Parent Table and has impact on Recoverability Changing sets CHKP unless RULES=STD Before Triggers Most flexible but worst for performance High Transaction rates can cause performance problems Cannot fire triggers explicitly to Check validation 11 11

12 Stored Procedures and UDFs Can ALTER all options except Parameter Lists These have immediate effect but invalidate packages Remember UDFs can have multiple parameter lists Can sometimes add a new UDF rather than altering This is called POLYMORPHISM 12 The PROCEDURE and FUNCTION syntax has many parameters. All of them except for the parameter list can be altered, and have immediate effect. 12

13 Forward Planning can help (a lot) Disk Space is relatively cheap Don t make restrictive design decisions Although ability to change Data Type helps Anyone remember Y2K? All that work was caused by a popular decision to save 2 bytes... Enforce coding standards at your shop Have you got any SELECT * in your applications? What about in View Definitions? Naming columns on all Insert Statements Particularly try and avoid need to change DSSIZE Not likely to be such a problem from DB2 10 onwards Remember REORG re-establishes FREEPAGE, PCTFREE 13 13

14 A quick word on tools Even with improved OLS capabilities Admin tools help Help identify other objects impacted by a change Particularly Views, Triggers, Referential Integrity View text can be very complicated to replace Retain security if you need to drop the objects Generate Package (re)binds Provide Automated Change Backout strategies Most sites still need a tool of some sort, even if it s home grown using REXX or whatever 14 14

15 What can we change Online in DB2 9 for z/os? We ll now go through each of the major components of a typical schema in turn Highlighting what can and cannot be changed Items in Red were new features of DB2 9 for z/os Intention is not to list every feature or restriction in detail That s what the manuals are for and I ve only got 60 minutes! Rather to give you a feel of what can and cannot be changed 15 15

16 ALTER STOGROUP ADD VOLUMES REMOVE VOLUMES DATACLAS, MGMTCLAS, STORCLAS Immediate effect, when next object allocated No validity checks at time of ALTER Stop & Start Workfile Database for change to take effect The ALTER itself will work What s still missing? Name, VCAT 16 These changes take effect the next time an object is allocated using the STOGROUP. Remember that once a STOP command has been issued the object goes into STOPP (Stop Pending) and no new thread will be able to gain access to it. You cannot cancel a STOP request once issued, so it s worth being very careful in a high availability environment before you issue this. This goes for all STOP commands mentioned throughout the presentation. 16

17 ALTER DATABASE BUFFERPOOL INDEXBP STOGROUP CCSID (compatible CCSIDs only effectively add Euro) Immediate effect for new objects created in the Database No impact on existing objects What s Still Missing? Name, AS WORKFILE clause 17 17

18 ALTER TABLESPACE Everything can be changed online except the following: Name LARGE, LOB DEFINE YES/NO DSSIZE, PAGE SIZE and SEGSIZE MEMBER CLUSTER NUMPARTS Changes take effect at different times see notes See DB2 10 slides at end of presentation for significant developments in this area covering items marked in blue 18 Some of these changes are immediate (such as changing LOCKSIZE to a less granular level for example Row to Tablespace) and some when the next extent is created (PRIQTY, SECQTY), while others require a REORG (such as changing the COMPRESS attribute). See the SQL Reference Manual for full details. Altering the Bufferpool of an Object when using Data Sharing requires that the Tablespace must be stopped. Note the impact of some changes varies depending on what you re doing. For example changing LOCKSIZE to something less Granular (row -> Table -> Tablespace) is an immediate change, whereas going to a more granular level (e.g. Table -> Row) requires a Rebind. 18

19 ALTER TABLE You can now do most things including: Adding or Renaming a Column Altering the Data Type of a Column Change CONSTRAINTS (Add or Remove all types) ADD or DROP a CLONE ADD and ROTATE Partitions Change Partition Limit keys Amend MATERIALIZED QUERY TABLES APPEND YES/NO DATA CAPTURE CHANGES AUDIT value VOLATILE or NON VOLATILE 19 As with Tablespaces, the list of what you can do is so extensive it s probably easier to list what you cannot accomplish. I ll do this in a few slides, after spending some time discussing some of the major changes in Versions 8 and 9. 19

20 Changing Column Data Types Change within type Character, Numeric etc. New value must be large enough to hold all the old data Change is effective immediately New version of object created No conversion of data takes place Data in Materialized or Converted to new format on the fly Reported 10-50% CPU degradation Data Manipulation will cause entire row to be Logged Tablespace placed into AREO* state Dependent indexes also placed into AREO* Except for numeric columns which are set to RDBP Some SQL OK Deletes, Insert & Update with non-unique index There are many restrictions which prevent you changing the data type. Amongst these are: Table has an EDITPROC or VALIDPROC Table is used in an MQT There is an Extended Index that depends on the column The Column is referenced in an RI Constraint or a FIELDPROC The column is a LOB column or is Defined as a Security Label or Row Change Timestamp The Column is an IDENTITY column 20

21 Impact of changing a Data Type Plans, Packages and Statement Cache invalidated See Slide 7 for the impact of this Column based Statistics (e.g. HIGH2KEY) invalidated Distribution Frequency Statistics invalidated STATSTIME in SYSIBM.SYSCOLUMNS set to JAN 1,0001 All Constraints are regenerated All Views are regenerated This may well invalidate more Packages Recover handles the data not the Schema Change All data will be recovered using the current data type 21

22 ALTER TABLE ADD PARTITION Immediate availability Invalidates Packages Because the old path may not consider new partition Partition size inherited from last Partition Change afterwards with an ALTER TABLESPACE If last Part is in REORP the new one will be as well Can only add partitions up to maximum allowed Based on DSSIZE and PAGESIZE Recover to PIT before ADD sets Partition empty 22 22

23 ALTER INDEX Can ALTER most Physical Index Characteristics CLUSTER, PADDED, COMPRESS, PIECESIZE, PRIQTY, SECQTY, ERASE, FREEPAGE, PCTFREE, GBPCACHE Still cannot change UNIQUE setting Cannot Alter existing Columns in an index But you can Add a Column see next slide 23 23

24 Adding Columns to Index ALTER INDEX DSN8810.XDEPT2 ADD COLUMN (ADMREDPT); New column will be appended to end Cannot change order of columns Cannot append column to beginning of Index Not valid for Partitioning Indexes Not valid for indexes enforcing unique key constraints 24

25 Impact of adding a columns to an Index Column Added to TABLE and INDEX in one UOW Index remains available Index put in AREO* exception state Column only added to Index or in different UOW Index put in RBDP exception state Index uses DEFINE NO No exception state No new version of index is created Index changes only reflected in DB2 Catalog 25

26 Changing Partition Limit Keys Rebalance (via REORG) Cannot be used with IBM REORG SHRLEVEL CHANGE Look at DB2 10 or consider an ISV Utility Data may not be evenly spread if many duplicates New limit keys may not be ideally place for growth May need to Reorg twice if cluster doesn t match partitioning ALTER INDEX... ALTER PART n ENDING AT... Affected Partitions go into REORP status and are unavailable Must use SHRLEVEL NONE to resolve (but see DB2 10) Both methods Invalidate Plans and Packages REORG is likely to provide the less intrusive approach 26 26

27 Rotate Partitions Not really an OLS but a design decision Immediate availability Old data deleted so archive it first logged as SQL based DBD lock held for duration and could be extensive LOAD PART n REPALCE with Dummy first? Messy with Physical vs Logical Partitions Particularly if you also add logicals can get out of sequence PIT Recovery not allowed so take a copy! Early doc suggested SYSCOPY/SYSLGRNG rows deleted Actually (DB2 9 NFM) an ALTER row is added to SYSCOPY 27 27

28 Table Controlled Partitioning Several OLS actions convert to Table Controlled Drop Partitioning Index Alter Partitioning Index to NOT CLUSTER Create PARTITIONED Index (either DPSI or PI) Add or Rotate Partitions Create Index with VALUES but without CLUSTER Add Partitioning Key May be OK but you need to be aware that it s happened Changes High Limitkey to high values on conversion Easiest way to deliberately convert to TCP Create a DPSI using DEFER YES and then DROP it 28 28

29 Other Schema related Items you can change ALTER SEQUENCE Everything supported except changing Data Type PACKAGE PATH introduced in DB2 9 Use RESTART Keyword to restart SEQUENCE at a value UDFs and Stored Procedures Everything except parameter lists can be changed Immediate impact 29

30 Schema Items that cannot be changed ALIAS AUXILIARY TABLE Including Drop handled via Base Table DISTINCT TYPE GLOBAL TEMPORARY TABLE SYNONYM TRIGGER VIEW There is an ALTER VIEW but it only allows REGENERATE 30 These items cannot be amended in any way. For Auxiliary Tables it s even more restrictive in that they could not even be dropped the only way to remove them is to drop the underlying Base Table. 30

31 What s coming in DB2 10 for z/os Subject to change until GA date Based on data available in Public IBM Sessions due to NDA 31 The following slides include material previously disclosed by IBM in public sessions. I have more data available but cannot include it here due to Non-Disclosure Agreements. The slides will be updated once the release becomes Generally Available. 31

32 Reduced Catalog and Directory Contention Catalog and Directory being converted to UTS spaces RRF, SEGSIZE 32, DSSIZE 64Gb, MAXPARTITION 1 Row Level locking DB2 Managed, SMS Controlled Online REORG and CHECK Most if not all Internal Links Removed SQL access to SYSLGRNX data Allows more concurrent DDL & Reduced Bind Contention 32 32

33 Deferred ALTER Page Size via Bufferpool (Tablespace and Index) Available for IX in DB2 9 (RDBP) DSSIZE (TS) SEGSIZE (TS) MEMBER CLUSTER (TS) Convert Single Table Segmented TS => UTS PBG Convert Single Table Simple TS => UTS PBG Convert PBG to Hash (Immediate but Index goes RDBP) Changes go Pending and will implemented by REORG 33 Changes to a Bufferpool that don t impact the Page size are still immediate 33

34 How does it work? ALTER Statement Validated at execution Statement put on Pending List SYSIBM.SYSPENDINGDDL Tablespace in AREOR (Advisory) SQL Executed against object returns SQLCODE +610 Changes implemented by next REORG TABLESPACE Includes Index Changes Supported by SHRLEVEL REFERENCE and CHANGE only Inevitably Invalidates Plans and Packages Warns that Partition Statistics may not be reliable ALTER... DROP PENDING CHANGES available 34 34

35 Restrictions Can t mix Pending and Immediate ALTER in single SQL SQLCODE Immediate ALTERS may not be possible while changes are pending Also SQLCODE Only supported in Single Table Tablespaces Requires UTS unless converting Type Exception Pagesize for LOBs Cannot Recover to time before changes materialized You can UNLOAD from old Copies 35 35

36 Immediate ALTERs LOB Inline length and default VERSIONING ACCESS CONTROL MASK & PERMISSION TRIGGER and FUNCTION SECURED TIMESTAMP Precision and Timezone BUFFERPOOL PGSTEAL NONE (in memory objects) MAXPARTITIONS INDEX include columns 36 36

37 Other useful DB2 10 for z/os changes UTS objects can support MEMBER CLUSTER Compress on fly during INSERT After ALTER COMPRESS YES next INSERT triggers action Compression Dictionary built Asynchronously Service Task reads all Table rows and compresses them Tablespace in Memory with ALTER BUFFERPOOL Exploits 1Mb Page size on z10 processor (and z196) Non-unique columns in a Unique Index Included here because can be added via ALTER and REBUILD May allow you to drop Indexes improving performance 37 37

38 More DB2 10 changes Online Add Active Log Simple Tablespaces deprecated but convert with ALTER UTS is the Strategic choice (big hint moving forwards) Several Useful REORG Improvements REBALANCE and ALTER LIMITKEY with LOB columns SHRLEVEL CHANGE supports REBALANCE SHRLEVEL CHANGE and REFERENCE when REORP Changing Index Limit keys used to require SHRLEVEL NONE Online DDF changes (no details yet) 38 38

39 While we re talking about DB2 10 for z/os Don t forget you need to remove at least the following: Private Protocol Plan Tables before DB2 V8 PLANs containing DBRMs ACQUIRE ALLOCATE Old Packages (pre-v5 but haven t we all rebound everything?) XML Extender Deprecated in DB2 10 Simple Tablespaces Partitioned Tablespaces other than UTS NEWFUN(YES) and NEWFUN(NO) And several others see documentation when GA 39 39

40 Bibliography Maintaining near 24x7 Availability During Table Changes IDUG 2002 & 2003 James Surber & Terry Burman (DST Systems) Making DB2 Changes in Minutes IDUG Jonathon Herbert (UBS) Numerous OLS presentations from IDUG and elsewhere Various presenters ask me for a list DB2 10 for z/os Webex and other pre-release sessions Roger Miller and David Beulke 40 40

41 Steve Thomas BMC Software Session B17 Can you really change DB2 for z/os Schemas online? 41