Oracle LogMiner (10g)

Transcription

1 Oracle LogMiner (10g) Introduction and Case Studies Martin Frauendorfer, SAP Active Global Support 2008

2 Agenda 1. Introduction 2. Case Study 1: ORA Problem 3. Case Study 2: Core Dump Problem SAP 2007 / Page 2

3 Oracle LogMiner (1) What is the LogMiner? Based on LogMiner it is possible to extract data from the redo logs, e.g.: DML operations that were executed (redo information) DML operations to undo the executed DML operations (undo information) DDL operations Timestamp of the executed operations SCN of the executed operations Using LogMiner on Oracle 10g is rather simple in the standard case: Make sure that the necessary redo logs are available in the archive directory Start LogMiner and provide the timeframe you want to analyze LogMiner will then make the redo log data available via V$LOGMNR_CONTENTS Select the desired data from V$LOGMNR_CONTENTS SAP 2007 / Page 3

4 Oracle LogMiner (2) Redologs t EXECUTE DBMS_LOGMNR.START_LOGMNR - ( STARTTIME => TO_DATE(' :00:00', 'DD.MM.YYYY HH24:MI:SS'), - ENDTIME => TO_DATE(' :00:00', 'DD.MM.YYYY HH24:MI:SS'), ); Data extraction V$LOGMNR_CONTENTS Individual selections WHERE SEG_NAME = TATAF WHERE SQL_REDO LIKE DROP% WHERE OPERATION = DDL SAP 2007 / Page 4

5 Oracle LogMiner (3) Data Extraction: Command: EXECUTE DBMS_LOGMNR.START_LOGMNR - ( STARTTIME => <start_time> - ); ENDTIME => <end_time> - OPTIONS => DBMS_LOGMNR.DICT_FROM_ONLINE_CATALOG+ - Explanation of options: DBMS_LOGMNR.CONTINUOUS_MINE+ - DBMS_LOGMNR.COMMITTED_DATA_ONLY - DICT_FROM_ONLINE_CATALOG: Dictionary information is retrieved from data dictionary of current database CONTINUOUS_MINE: Necessary redo logs are determined by Oracle automatically COMMITTED_DATA_ONLY: Output contains only already committed data Compared to Oracle 9i it is usually no longer necessary to manually create a dictionary or make the redo logs known to LogMiner. Instead the specification of a start time and an end time is usually sufficient. SAP 2007 / Page 5

6 Oracle LogMiner (4) V$LOGMINER_CONTENTS: Important values available for each change: SCN: System change number TIMESTAMP: Timestamp COMMIT_TIMESTAMP: Commit timestamp SEG_OWNER: Owner of involved segment SEG_NAME: Name of involved segment SESSION_INFO: Information related to Oracle session OPERATION: Type of change (INSERT, UPDATE, DELETE, DDL, ) SQL_REDO: SQL statement to redo the change SQL_UNDO: SQL statement to undo the change SAP 2007 / Page 6

7 Oracle LogMiner (5) SELECT TIMESTAMP, OPERATION, SQL_REDO FROM V$LOGMNR_CONTENTS WHERE SEG_NAME = '<segment_name>'; SELECT SQL_UNDO FROM V$LOGMNR_CONTENTS WHERE OPERATION = 'DELETE' AND SEG_NAME = '<segment_name>'; Changes of a certain segment Generation of statements to undo DELETE operations related to a segment Recorded COMMITs per minute SELECT TO_CHAR(TIMESTAMP, 'DD.MM.YYYY HH24:MI') "MINUTE", COUNT(*) NUMBER_OF_COMMITS FROM V$LOGMNR_CONTENTS GROUP BY TO_CHAR(TIMESTAMP, 'DD.MM.YYYY HH24:MI') ORDER BY 1; V$LOGMNR_CONTENTS DROP of a table DDL statements SELECT TIMESTAMP, SQL_REDO FROM V$LOGMNR_CONTENTS WHERE OPERATION = 'DDL' AND UPPER(SQL_REDO) LIKE 'DROP%<segment_name>%'; SELECT TIMESTAMP, SQL_REDO, SEG_NAME SEGMENT_NAME FROM V$LOGMNR_CONTENTS WHERE OPERATION = 'DDL'; SAP 2007 / Page 7

8 Oracle LogMiner (6) Obvious situations when LogMiner can be pretty useful: A table was dropped accidentally You can determine the exact SCN of the DROP. Then on a separate system a recovery can be done up to the SCN before the DROP. So all data can be retrieved that existed in the table at the time of the DROP. It is not clear when and how table data was changed You can determine exact timestamps and other details of the DML operations that accessed the table. Based on this you can be able to understand the root cause. You want to understand why a high amount of redo logs is generated The LogMiner information is able to tell you which kind of changes happen most frequently. Less obvious situations when LogMiner can be pretty useful: Errors ORA-08103: object no longer exists See subsequent case studies Core dumps during DML operations See subsequent case studies SAP 2007 / Page 8

9 Case Study 1: ORA (1) What does ORA-08103: object no longer exists indicate? In most of the cases this error is an indication for a block corruption. Rather than containing data the block is completely filled with zero values. In some cases like in this case study it really means that an accessed object no longer exists. This scenario can happen if a long running SELECT is started and during its execution the accessed table is dropped. In SAP environments this will usually not happen because normally no DROP is executed. Nevertheless there are some situations where DROPs take place and theoretically an ORA can result, e.g.: During BRSPACE online reorganizations when the source table is dropped During certain BI activities when tables or partitions are dropped SAP 2007 / Page 9

10 Case Study 1: ORA (2) What was the initial situation in the concrete case? SAP 2007 / Page 10 A customer suffered from sporadic, not reproducible ORA errors. Consistency checks were always fine so there was no block corruption. Tracefiles were generated in <sapdata_home>/saptrace/usertrace containing information like the following: *** ACTION NAME:(5896) :37: *** MODULE NAME:(SAPLRSDRS) :37: *** SERVICE NAME:(<sid>.world) :37: *** SESSION ID:(82.14) :37: *** :37: ksedmp: internal or fatal error ORA-08103: object no longer exists Current SQL statement for this session: SELECT /*+ STAR_TRANSFORMATION FACT( T_00 ) */... FROM "/BIC/FZPCA01" T_00... The related BI fact table was always /BIC/FZPCA01. As no block corruption exists, a DROP should be responsible for the error. In order to find out more the above error on 23:37:12 was checked in detail.

11 Case Study 1: ORA (3) Analysis steps: A DROP may not immediately result in an error if a complex join is processed (like in this case). Therefore it is important to check for DROP operations between the start of the SQL statement (SQL_ID: g09p4p0k4qp38) and the error time. Based on the Oracle active session history it was possible to determine that the execution started at 23:32:17: Now the following LogMiner extraction can be started: EXECUTE DBMS_LOGMNR.START_LOGMNR - ( STARTTIME => TO_DATE(' :32:00', 'DD.MM.YYYY HH24:MI:SS'), - ); ENDTIME => TO_DATE(' :38:00', 'DD.MM.YYYY HH24:MI:SS'), - OPTIONS => DBMS_LOGMNR.DICT_FROM_ONLINE_CATALOG+ - DBMS_LOGMNR.CONTINUOUS_MINE - ASH Information SAP 2007 / Page 11

12 Case Study 1: ORA (4) Analysis steps (2): As next step a SELECT for DDL operations can be performed on V$LOGMNR_CONTENTS: SELECT TO_CHAR(TIMESTAMP, 'dd.mm.yyyy hh24:mi:ss') TIMESTAMP, SQL_REDO FROM V$LOGMNR_CONTENTS WHERE OPERATION = 'DDL'; Among others the output contained the following: TIMESTAMP SQL_REDO :35:08 ALTER TABLE "/BIC/FZPCA01" DROP PARTITION "/BIC/FZPCA "; Similar checks for other occurrences of ORA also returned DROP PARTITION operations on table /BIC/FZPCA01. At this point it was very likely that the DROP PARTITION command is the root cause of the ORA SAP 2007 / Page 12

13 Case Study 1: ORA (5) Analysis steps (3): But what is special in this system? If there is a general problem in BI / SEM we should see the ORA more frequently DROP PARTITION was executed in this case as part of a BI compression activity. According to the customer it is guaranteed that the SELECT operations always retrieve data from partitions that are not dropped at the same time. Further analysis showed that this information was really correct. So the big question was: Why are SELECT operations cancelled although they need not to read any data from the dropped partition? SAP 2007 / Page 13

14 Case Study 1: ORA (6) Analysis steps (4): The answer can be found by looking at the execution plan of the SELECT: Execution Plan Although only data from one partition is needed, the CBO decides to perform a PARTITION RANGE ALL! As a consequence also the dropped partition is accessed! But why does the CBO decide to use this bad execution plan instead of a single partition access? Because the statistics indicate that the table is empty: Table /BIC/FZPCA01 SAP 2007 / Page 14 Last statistics date Analyze Method Compute Number of rows 0 Number of blocks allocated 0 Number of empty blocks Average space Chain count 0 Average row length 0 Partitioned YES

15 Case Study 1: ORA (7) Analysis steps (5): If the table is empty, it doesn t matter for the CBO if it has to scan only 1 or many empty partitions. The costs are optimal in any case. So it decided to do the PARTITION RANGE ALL access. Instead of being empty the table had 1.8 million records at the time of the problem. This resulted in a rather long SELECT runtime and the intermittent ORA errors. What is the final solution? New CBO statistics had to be created. Then the CBO recognized that a single partition scan is much better. The dropped partitions were no longer accessed. The ORA no longer appeared And last but not least: The query performance was also much better. Lesson learned: An ORA can be caused by outdated CBO statistics! SAP 2007 / Page 15

16 Case Study 2: Core Dumps What was the problem situation? A BI compression run failed reproducible with ORA ORA indicates that no connection between SAP work process and Oracle shadow process exists any more. There can be dozens of root causes for ORA-03113, but in this particular case on Oracle side a trace file with the following content was written: Exception signal: 11 0xf454 (0000F454) e This means that the Oracle shadow process was terminated due to signal 11 from the operating system which is the main problem here. Unfortunately not any additional information was available: No stack trace in the Oracle trace file No core dump written No SQL statement logged Checks for corruptions and bitmap index issues didn t reveal any problem. From the Oracle active session history it was only clear that an UPDATE on table /BIC/EC3CACCGCR must have been failed, but it was not clear, which concrete values were used in the failing SQL statement. SAP 2007 / Page 16

17 Case Study 2: Core Dumps (2) Analysis Steps (1): At this point it is important to determine the exact SQL statement (with real values, not bind variables) that runs into the core dump, but this is not easy at all because: The failure happens within a stored procedure generated by BI where millions of records are updated. Therefore not the individual UPDATE statements, but only the stored procedure is displayed on SAP side in transactions like ST05 (SQL trace) or ST22 (short dump) or in work process trace files. The Oracle active session history only displays the SQL statement with bind variables, but not with real values. Activating a SQL trace on Oracle side results in a significant slowdown and huge trace files (that exceeded the available filesystem space at the customer). Therefore it was useful to check the individual UPDATE statements based on LogMiner. SAP 2007 / Page 17

18 Case Study 2: Core Dumps (3) Analysis Steps (2): The problem was reproduced again. In the Oracle ORA trace file the timestamp :47:34 was found. LogMiner was started with a restriction to the short time frame of the termination: EXECUTE DBMS_LOGMNR.START_LOGMNR - ( STARTTIME => TO_DATE( :47:34', 'DD.MM.YYYY HH24:MI:SS'), - ENDTIME => TO_DATE( :47:38', 'DD.MM.YYYY HH24:MI:SS'), - OPTIONS => DBMS_LOGMNR.DICT_FROM_ONLINE_CATALOG+ - DBMS_LOGMNR.CONTINUOUS_MINE - ); Now the SCN of the most recent UPDATE can be determined: SELECT TO_CHAR(TIMESTAMP, dd.mm.yyyy hh24:mi:ss ) TIMESTAMP, SCN FROM V$LOGMNR_CONTENTS WHERE OPERATION = UPDATE AND SEG_NAME = /BIC/EC3CACCGCR ORDER BY SCN; SAP 2007 / Page 18

19 Case Study 2: Core Dumps (4) Analysis Steps (3): The result was: TIMESTAMP SCN :47: :47: :47: :47: :47: :47: :47: :47: :47: :47: :47: :47: The ten records with SCN must have been the last successful UPDATEs before the failure. SAP 2007 / Page 19

20 Case Study 2: Core Dumps (5) Analysis Steps (4): The simplified next steps are now: Determine E fact table ROWID of the last successful records based on the SQL_REDO information in V$LOGMNR_CONTENTS Determine E fact table key columns of the last successful records based on the ROWID Determine F fact table records with same key columns Determine subsequent E and F fact table record Perform the UPDATE on the subsequent E fact table record by adding the F fact table delta values Core Dump! At this point the failing UPDATE could be simplified until it turned out that the following column command was the problem: UPDATE "/BIC/EC3CACCGCR" SET "/BIC/N_VP_LSAL" = "/BIC/N_VP_LSAL" WHERE KEY_C3CACCGCRP = 0 AND KEY_C3CACCGCRT = 147 AND KEY_C3CACCGCRU = 9 AND KEY_C3CACCGCR1 = 23 AND KEY_C3CACCGCR2 = 19 AND KEY_C3CACCGCR3 = 2 AND KEY_C3CACCGCR4 = 2 AND KEY_C3CACCGCR5 = 20; SAP 2007 / Page 20

21 Case Study 2: Core Dumps (6) Analysis Steps (5): The former value of /BIC/N_VP_LSAL was E-33. A dump of this number delivered: DUMP("/BIC/N_VP_LSAL") Typ=2 Len=3: 79,1,102 It turned out that this number is an invalid FLOAT number that was inserted in the database due to Oracle bug described in SAP note As the small value was a consequence of rounding errors it was okay to update the wrong value with a correct 0 value and the problem was resolved. SAP 2007 / Page 21

22 Thank you! SAP 2007 / Page 22

23 Copyright 2008 SAP AG All rights reserved No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained herein may be changed without prior notice. Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors. SAP, R/3, mysap, mysap.com, xapps, xapp, SAP NetWeaver, Duet, Business ByDesign, ByDesign, PartnerEdge and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and in several other countries all over the world. All other product and service names mentioned and associated logos displayed are the trademarks of their respective companies. Data contained in this document serves informational purposes only. National product specifications may vary. The information in this document is proprietary to SAP. This document is a preliminary version and not subject to your license agreement or any other agreement with SAP. This document contains only intended strategies, developments, and functionalities of the SAP product and is not intended to be binding upon SAP to any particular course of business, product strategy, and/or development. SAP assumes no responsibility for errors or omissions in this document. SAP does not warrant the accuracy or completeness of the information, text, graphics, links, or other items contained within this material. This document is provided without a warranty of any kind, either express or implied, including but not limited to the implied warranties of merchantability, fitness for a particular purpose, or non-infringement. SAP shall have no liability for damages of any kind including without limitation direct, special, indirect, or consequential damages that may result from the use of these materials. This limitation shall not apply in cases of intent or gross negligence. The statutory liability for personal injury and defective products is not affected. SAP has no control over the information that you may access through the use of hot links contained in these materials and does not endorse your use of third-party Web pages nor provide any warranty whatsoever relating to third-party Web pages Weitergabe und Vervielfältigung dieser Publikation oder von Teilen daraus sind, zu welchem Zweck und in welcher Form auch immer, ohne die ausdrückliche schriftliche Genehmigung durch SAP AG nicht gestattet. In dieser Publikation enthaltene Informationen können ohne vorherige Ankündigung geändert werden. Einige von der SAP AG und deren Vertriebspartnern vertriebene Softwareprodukte können Softwarekomponenten umfassen, die Eigentum anderer Softwarehersteller sind. SAP, R/3, mysap, mysap.com, xapps, xapp, SAP NetWeaver, Duet, Business ByDesign, ByDesign, PartnerEdge und andere in diesem Dokument erwähnte SAP-Produkte und Services sowie die dazugehörigen Logos sind Marken oder eingetragene Marken der SAP AG in Deutschland und in mehreren anderen Ländern weltweit. Alle anderen in diesem Dokument erwähnten Namen von Produkten und Services sowie die damit verbundenen Firmenlogos sind Marken der jeweiligen Unternehmen. Die Angaben im Text sind unverbindlich und dienen lediglich zu Informationszwecken. Produkte können länderspezifische Unterschiede aufweisen. Die in diesem Dokument enthaltenen Informationen sind Eigentum von SAP. Dieses Dokument ist eine Vorabversion und unterliegt nicht Ihrer Lizenzvereinbarung oder einer anderen Vereinbarung mit SAP. Dieses Dokument enthält nur vorgesehene Strategien, Entwicklungen und Funktionen des SAP -Produkts und ist für SAP nicht bindend, einen bestimmten Geschäftsweg, eine Produktstrategie bzw. -entwicklung einzuschlagen. SAP übernimmt keine Verantwortung für Fehler oder Auslassungen in diesen Materialien. SAP garantiert nicht die Richtigkeit oder Vollständigkeit der Informationen, Texte, Grafiken, Links oder anderer in diesen Materialien enthaltenen Elemente. Diese Publikation wird ohne jegliche Gewähr, weder ausdrücklich noch stillschweigend, bereitgestellt. Dies gilt u. a., aber nicht ausschließlich, hinsichtlich der Gewährleistung der Marktgängigkeit und der Eignung für einen bestimmten Zweck sowie für die Gewährleistung der Nichtverletzung geltenden Rechts. SAP übernimmt keine Haftung für Schäden jeglicher Art, einschließlich und ohne Einschränkung für direkte, spezielle, indirekte oder Folgeschäden im Zusammenhang mit der Verwendung dieser Unterlagen. Diese Einschränkung gilt nicht bei Vorsatz oder grober Fahrlässigkeit. Die gesetzliche Haftung bei Personenschäden oder die Produkthaftung bleibt unberührt. Die Informationen, auf die Sie möglicherweise über die in diesem Material enthaltenen Hotlinks zugreifen, unterliegen nicht dem Einfluss von SAP, und SAP unterstützt nicht die Nutzung von Internetseiten Dritter durch Sie und gibt keinerlei Gewährleistungen oder Zusagen über Internetseiten Dritter ab. Alle Rechte vorbehalten. SAP 2007 / Page 23