Data Warehousing & Big Data at OpenWorld for your smartphone

Transcription

1

2 Data Warehousing & Big Data at OpenWorld for your smartphone Smartphone and tablet apps, helping you get the most from this year s OpenWorld Access to all the most important information Presenter profiles Must-see sessions Must-attend hands-on labs Useful links

3 If you think Partitioning is only for Performance, think again Matthias Fuchs DW Architect, ISE Information Systems Engineering Hermann Bär Database Product Management, Oracle

4 Partitioning for Data Management Concepts and functionality Some tips and tricks Partitioning at work Q&A

5 Oracle Partitioning in Oracle Database 12c Business Modeling, Performance, and Data Management Oracle 8.0 Oracle 8i Core functionality Performance Manageability Range partitioning Global Range indexes Hash partitioning Range-Hash partitioning Static partition pruning Partition-wise joins Dynamic partition pruning Basic maintenance: ADD, DROP, EXCHANGE Expanded maintenance: MERGE Oracle 9i List partitioning Global index maintenance Oracle 9i R2 Range-List partitioning Fast partition SPLIT Oracle 10g Global Hash indexes Local Index maintenance Oracle 10g R2 1M partitions per table Multi-dimensional pruning Fast DROP TABLE Oracle 11g Oracle 11g R2 Virtual column based partitioning More composite choices REF partitioning Hash-Hash partitioning Expanded REF partitioning AND pruning Oracle 12c R1 Interval-REF partitioning - Partition Maintenance on multiple partitions - Partial local and global indexes - Zone Maps - Interval partitioning - Partition Advisor - Incremental statistics mgmt Multi-branch execution - Asynchronous global index maintenance for DROP/TRUNCATE - Online partition MOVE - Cascading TRUNCATE/EXCHANGE

6 Partitioning for Data Management Logical attributes Partitioning setup Indexing and index maintenance Read only (in conjunction with tablespace separation) Physical attributes Data placement Segment properties in general

7 Nonpartitioned Tables Physical and Logical Attributes Logical table properties Columns and data types Constraints Indexes, Physical table properties Table equivalent to segment Tablespace Compression, [ Logging nologging ], In-memory Properties managed and changed on segment level

8 Partitioned Tables Physical and Logical Attributes Table Partition Partition Logical table properties Columns and data types Constraints Partial Indexes, Physical property directives Physical [sub]partition properties [Sub]partition equivalent to segment Tablespace Compression, [ Logging nologging ], In-memory Properties managed and changed on segment level

9 Partitioned Tables Physical and Logical Attributes Table is metadata-only and directive for future partitions No physical segments on table level Physical attributes become directive for new partitions, if specified Single-level partitioned table Partitions are equivalent to segments Physical attributes are managed and changed on partition level Composite-level partitioned tables Partitions are metadata only and directive for future subpartitions Subpartitions are equivalent to segments

10 Aspects of Data Management Fast population of data Fast removal of data Fast reorganization of data

11 Aspects of Data Management Addressable with Partition Maintenance Operations Fast population of data EXCHANGE Per-partition direct path load Fast removal of data DROP, TRUNCATE, EXCHANGE Fast reorganization of data MOVE, SPLIT, MERGE

12 Partition Maintenance Operations on multiple Partitions Partition Maintenance on multiple partitions in a single operation Full parallelism Transparent maintenance of local and global indexes Often combined with change of physical properties Location, compression Jan 2009 Feb 2009 Mar 2009 Apr2009 Nov 2009 Dec 2009 Quarter Apr2009 Nov 2009 Dec 2009 ALTER TABLE orders MERGE PARTITIONS Jan2009, Feb2009, Mar2009 INTO PARTITION Quarter1_2009 COMPRESS FOR ARCHIVE HIGH;

13 Online Partition Move Transparent MOVE PARTITION ONLINE operation Concurrent DML and Query Index maintenance for local and global indexes Jan2012 Aug2012 Sep2012 Oct2012 Nov2012 Dec2012

14 Asynchronous Global Index Maintenance Usable global indexes after DROP and TRUNCATE PARTITION without index maintenance Affected partitions are known internally and filtered out at data access time DROP and TRUNCATE become fast, metadata-only operations Significant speedup and reduced initial resource consumption Delayed Global index maintenance Deferred maintenance through ALTER INDEX REBUILD COALESCE Automatic cleanup using a scheduled job

15 Indexing with Oracle Partitioning Partial indexes span only some partitions Applicable to local and global indexes Full Indexing Global Non-Partitioned Index Global Partitioned Index Local Partitioned Index Complementary to full indexing Full support of online index maintenance Indexing on Table Partition Table Partition Table Partition Partial Local Partitioned Index Indexing off Partial Indexes Partial Global Partitioned Index No Indexing Partial Global Index

16 Partitioning for Data Management Concepts and functionality Some tips and tricks Partitioning at work Q&A

17 Fast Data Loading with Exchange Optimize incremental statistics management Create table to being exchanged with synopsis prior to table statistics collection Table level preference INCREMENTAL => TRUE INCREMENTAL_LEVEL => TABLE

18 Online Move for non-interrupted Application Support Plan for best time window Minimize concurrent DML operations if possible Require additional disk space and resources for journaling Journal will be applied recursively after initial bulk move The larger the journal, the longer the runtime Concurrent DML has impact on compression efficiency Best compression ratio with initial bulk move

19 Index Maintenance for Partition-based Data Management Choose your optimal strategy Online index maintenance available for both global and local indexes Global index maintenance since Oracle 9i, local index maintenance since Oracle 10g Decision is always performance versus availability Rebuild of index always faster when more than 5%-10% of data are touched Consider partial indexing for both old and new data Not all data has to be indexed to begin with

20 Partition Maintenance Operations Best Practices - Miscellaneous Issue partition maintenance operations in batch Every partition DDL creates a new version of the partition metadata Leverage new multi-partition operation whenever possible Many ADD SUBPARTITION versus single ADD PARTITION Use partition extended syntax for bulk loads X-lock only on partition that is being locked Enables concurrent parallel direct path load from external programs Requires clean data

21 Partitioning for Data Management Concepts and functionality Some tips and tricks Partitioning at work Q&A

22 ISE Information Systems Engineering Enable extreme Performance. Established in 1991 IT and Business Consulting for large and medium-sized companies Main focus: Oracle Core Technology Database (RAC), Application Server (WebLogic) Oracle Exadata / Exalogic / Exalytics extreme Performance Oracle Datawarehousing & Business Intelligence Oracle Exadata Migrations Performance Analysis & Optimization Employees: 60 Headquarter in Grafenberg, Branches in Munich and Nuremberg (Germany) Copyright (C) ISE GmbH - All Rights Reserved 24

23 ISE Oracle Technology Center Enable extreme Performance. First and only Exastack Technology Center in Germany - Nuremberg Copyright (C) ISE GmbH - All Rights Reserved 25

24 Partition Maintenance at work - Examples Enable extreme Performance. Partition maintenance operations for data management used for both data warehousing and OLTP environments Data warehousing HCC compression for read-mostly partitions Move partitions for data storage tiering OLTP Purging of old data Copyright (C) ISE GmbH - All Rights Reserved 26

25 Partition Maintenance in Data Warehousing Enable extreme Performance. Typical data management operations Establish compression for read-mostly partitions ALTER TABLE F_EI_POSITIONS MOVE PARTITION DFIPS_201406_R COMPRESS FOR QUERY HIGH UPDATE INDEXES PARALLEL 4 Tablespace reorganization and storage tiering ALTER TABLE P_I_POSITIONS MOVE PARTITION DPIPS_201407_R NOCOMPRESS UPDATE INDEXES PARALLEL 4 TABLESPACE DMDATA Key design decisions Parallelism required to meet customers SLAs (data maintenance window) Varying degree based on average partition size Update indexes mandatory to keep indexes usable Pure business driver Online capabilities not mandatory (yet) Copyright (C) ISE GmbH - All Rights Reserved 27

26 Self-Service Partition Maintenance Enable extreme Performance. Business requirement to allow partition compression for every user (schema), but Not every end user is capable of doing so Lack of administrative control Idea Create a package to manage partition maintenance - but not to execute immediately Run maintenance on a scheduled base Scheduled in online mode, runs throughout production hours Log and monitor the process Solution Schema owner decides which tables are compressed Detailed configuration maintained in configuration table Table name, compression type, frequency, destination tablespace, etc. Schema owner gets information per about the compression state Copyright (C) ISE GmbH - All Rights Reserved 28

27 Self-Service Partition Maintenance Enable extreme Performance. PLSQL package configuration Implementation started with 10g 11g interval partitioning added 12c plans: interval-reference partitioning, possibly online partition move Copyright (C) ISE GmbH - All Rights Reserved 29

28 Self-Service Partition Maintenance Enable extreme Performance. Example configuration entries Example log entries Copyright (C) ISE GmbH - All Rights Reserved 30

29 Self-Service Partition Maintenance Enable extreme Performance. Wrapper around fundamental partition maintenance operation v_sql_str := 'ALTER TABLE ' a_tab_part_tab(v_tab_part_idx).table_owner '.' a_tab_part_tab(v_tab_part_idx).table_name ' ' v_admin_str ' PARTITION ' a_tab_part_tab(v_tab_part_idx).partition_name ' TABLESPACE ' r_part_admin_master.dest_tablespace ' ' v_storage ' PCTFREE 0' ' ' v_update_indexes ' ' v_compress ' ' v_tab_parallel; exec_sql_str(v_sql_str); /* several tasks added for bitmap indexes auto rebuild indexes */ Also checks for index, tablespace etc. Rebuild index if necessary, check for unusable index Copyright (C) ISE GmbH - All Rights Reserved 31

30 Self-Service Partition Maintenance Enable extreme Performance. Impact of newly registered tables for compression (two schemas as example) Copyright (C) ISE GmbH - All Rights Reserved 32

31 Defragmentation and Tablespace Maintenance Enable extreme Performance. Automated script generation for tablespace reorganization and storage tiering Script generation only To be run in offline hours Business operations Move data to second diskgroup/storage box Defragmentation Empty tablespaces for shrink Copyright (C) ISE GmbH - All Rights Reserved 33

32 Defragmentation and Tablespace Maintenance Enable extreme Performance. Sample processing flow script generation with logging dbms_output.put_line('rem Move NonPartitioned Tables'); movenonparttables; dbms_output.put_line('rem Move Partitioned/SubPartitioned Tables'); moveparttables; dbms_output.put_line('rem Move Lobs'); movelob; dbms_output.put_line('rem Rebuild NonPartitioned Indexes'); rebuildnonpartind; dbms_output.put_line('rem Rebuild Partitioned/SubPartitioned Indexes'); rebuildpartind; Copyright (C) ISE GmbH - All Rights Reserved 34

33 Defragmentation and Tablespace Maintenance Enable extreme Performance. Procedure to create SQL scripts for all objects inside a table: Physically move all partitions of the table to new tablespace Modify default tablespace for future partitions Copyright (C) ISE GmbH - All Rights Reserved 35

34 Data Management in Oracle SOA Suite, Release 11g Enable extreme Performance. Oracle has data management challenges, too ;-) Oracle SOA Suite, Release 11g SOA stores process data for every process step, so-called instances Many rows in many tables for each instance After a process is finished, data can (must) be deleted Deletion must be done with DML Current data cleanup is inefficient Delete script, even in parallel, can take many hours Massive generation of Redo and Undo Increasing data fragmentation through massive deletes Copyright (C) ISE GmbH - All Rights Reserved 36

35 Data Management with Partitioning in Oracle SOA Suite, Release 12c Enable extreme Performance. Oracle s solution: Eat your own dog food Oracle SOA Suite, Release 12c Out-of-the-box implementation of Partitioning for core tables Interval-Reference Partitioning Enforces referential integrity and equipartioning for parent and child tables Automatic generation of new hierarchical partition tree Partitioning key contains the state of each flow (open, closed). When the master is checked and everything is closed, dependent tables with the same partition range can be dropped. Out-of-the-box data cleanup using drop -> points out which partition can be dropped Moving Active, Long Running Instances to a Different Partition Creating separate partition for long running (new_partition_date) Attention row movement with many instances can take while Copyright (C) ISE GmbH - All Rights Reserved 37

36 Summary Partitioning is more than performance Data maintenance with Partitioning is powerful and focus for future development Hundreds of successful customer use case scenarios Just like the ISE use case thanks Matthias for presenting Oracle wants to hear from you Interesting use case scenarios and implementations Enhancement requests

37 Questions

38

39