Oracle Trace File Analyzer (TFA) Diagnostics for the Cloud

Transcription

1 Oracle Trace File Analyzer (TFA) Diagnostics for the Cloud

2 Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle s products remains at the sole discretion of Oracle. 2

3 Why TFA? Easy to use real-time health monitoring, fault detection & diagnosis via a single interface Secure consolidation of distributed diagnostic collections Continuous availability so you always get what s needed for resolution Machine learning driven, autonomous degradation detection, reduces your overheads 3

4 Autonomous Usage Diagnostics are collected 2 3 Distributed diagnostics are consolidated and packaged TFA Oracle Grid Infrastructure & Databases 1 TFA detects a fault 4 Notification of fault is sent Oracle Support 5 Diagnostic collection is uploaded to Oracle Support for root cause analysis & resolution 4

5 On-Demand Usage 2 Real-time status summary 3 Diagnose with DB tools 4 Perform diagnostic collection Upload diagnostic collection to Oracle Support 5 TFA Oracle Grid Infrastructure & Databases 1 Request desired action on-demand Oracle Support 5

6 Command Interfaces Command line Shell Menu Specify all command options at the command line tfactl <command> 1. Set and change context 2. Run commands from within the shell tfactl tfaclt > database MyDB MyDB tfactl > oratop 1. Select menu navigation options then choose the command you want to run tfactl menu 6

7 Data Redaction mask_strings.xml <mask_strings> <mask_string> <original>widgetnode1</original> <replacement>node1</replacement> </mask_string> <mask_string> <original> </original> <replacement>node1-ip</replacement> </mask_string> <mask_string> <original>widgetnode2</original> <replacement>node2</replacement> </mask_string> <mask_string> <original> </original> <replacement>node2-ip</replacement> </mask_string> </mask_strings> If you want to mask sensitive data Hostname IP address Etc Create mask_strings.xml In the directory tfa_home/resources Define all data replacements you want 7

8 Configure User Access Users & groups authorized to access tfactl by default Enable or disable user access tfactl access enable tfactl access disable TFA List user access tfactl access lsusers Add users All commands apply cluster wide unless -local used to restrict to local node tfactl access add user <user> [-local] Remove users or everyone tfactl access remove user <user> [-local] tfactl access removeall [-local] Reset to default access Grid Home Owner DB Home Owners Everyone Else tfactl access reset 8

9 Set Notification Addresses TFA can send notification when faults are detected To set notification for any problem detected: tfactl set To set notification for specific ORACLE_HOMEs include the OS owner: tfactl set 9

10 On-Demand Usage 2 Real-time status summary 3 Diagnose with DB tools 4 Perform diagnostic collection Upload diagnostic collection to Oracle Support 5 TFA Oracle Grid Infrastructure & Databases 1 Request desired action on-demand Oracle Support 10

11 Real-time Status Summary High-level summary of all Database components tfactl summary Choose an option to drill down 11

12 Real-time Status Summary Drill Down Drill downs show real-time analytics & details of any problems found 12

13 On-Demand Usage 2 Real-time status summary 3 Diagnose with DB tools 4 Perform diagnostic collection Upload diagnostic collection to Oracle Support 5 TFA Oracle Grid Infrastructure & Databases 1 Request desired action on-demand Oracle Support 13

14 Investigate Logs & Look for Errors Analyze all important recent log entries: tfactl analyze last 1d Search recent log entries: tfactl analyze -search ora-006" -last 8h Searching for ora

15 Perform Analysis Using the Included Tools Tool orachk or exachk oswatcher procwatcher Description Provides health checks for the Oracle stack. Oracle Trace File Analyzer will install either Oracle EXAchk for Engineered Systems, see document for more details or Oracle ORAchk for all non-engineered Systems, see document for more details Collects and archives OS metrics. These are useful for instance or node evictions & performance Issues. See document for more details Automates & captures database performance diagnostics and session level hang information. See document for more details oratop Provides near real-time database monitoring. See document for more details. alertsummary ls pstack Provides summary of events for one or more database or ASM alert files from all nodes Lists all files TFA knows about for a given file name pattern across all nodes Generate process stack for specified processes across all nodes Tool grep summary vi tail param dbglevel history changes calog events managelogs ps Description Search alert or trace files with a given database and file name pattern, for a search string. Provides high level summary of the configuration Opens alert or trace files for viewing a given database and file name pattern in the vi editor Runs a tail on an alert or trace files for a given database and file name pattern Shows all database and OS parameters that match a specified pattern Sets and unsets multiple CRS trace levels with one command Shows the shell history for the tfactl shell Reports changes in the system setup over a given time period. This includes database parameters, OS parameters and patches applied Reports major events from the Cluster Event log Reports warnings and errors seen in the logs Shows disk space usage and purges ADR log and trace files Finds processes Not all tools are included in Grid or Database install. Download from to get full collection of tools triage Summarize oswatcher/exawatcher data Verify which tools you have installed: tfactl toolstatus 15

16 ORAchk & EXAchk (Support Tools Bundle) Oracle Stack Health Checks Oracle EXAchk for Engineered Systems Oracle ORAchk for everything else Proactively scans for the most impactful problems across the layers of your stack Simplifies and streamlines how to investigate and analyze which known issues present a risk to you Lightweight tool that runs within your environment without requiring config data to be sent to Oracle High level reports show your system health risks with the ability to drill down into specific problems and understand their resolutions Can be configured to send notifications when it detects problems Collection Manager, a companion Application Express web app, provides a single dashboard view of collections across your entire enterprise 16

17 OS Watcher (Support Tools Bundle) Collect & Archive OS Metrics Executes standard UNIX utilities (e.g. vmstat, iostat, ps, etc) on regular intervals Built in Analyzer functionality to summarize, graph and report upon collected metrics Output is Required for node reboot and performance issues Simple to install, extremely lightweight Runs on ALL platforms (Except Windows) MOS Note: OS Watcher Users Guide 17

18 Procwatcher (Support Tools Bundle) Monitor & Examine Database Processes Single instance & RAC Generates session wait, lock and latch reports as well as call stacks from any problem process(s) Ability to collect stack traces of specific processes using Oracle Tools and OS Debuggers Typically reduces SR resolution for performance related issues Runs on ALL major UNIX Platforms MOS Note: Procwatcher Install Guide 18

19 oratop (Support Tools Bundle) Near Real-Time Database Monitoring Single instance & RAC Monitoring current database activities Database performance Identifying contentions and bottleneck 19

20 Analyze Each tool can be run using tfactl in shell mode Start tfactl shell with tfactl Run a tool with the tool name tfactl > orachk 1. Where necessary set context with database <dbname> tfactl > database MyDB 2. Then run tool MyDB tfactl > oratop 3. Clear context with database MyDB tfactl > database 20

21 Automatic Database Log Purge TFA can automatically purge database logs OFF by default Except on a Domain Service Cluster (DSC), which it is ON by default Turn auto purging on or off: tfactl set managelogsautopurge=<on OFF> Will remove logs older than 30 days configurable with: Purging runs every 60 minutes configurable with: tfactl set managelogsautopurgepolicyage=<n><d h> tfactl set managelogsautopurgeinterval=<minutes> 21

22 Manual Database Log Purge TFA can manage ADR log and trace files Show disk space usage of individual diagnostic destinations Purge these file types based on diagnostic location and or age: "ALERT, "INCIDENT, "TRACE, "CDUMP, "HM, "UTSCDMP, "LOG tfactl managelogs <options> Option show usage -show variation older <n><m h d> -purge older <n><m h d> gi database [all dbname] -dryrun Description Shows disk space usage per diagnostic directory for both GI and database logs Use to determine per directory disk space growth. Shows the disk usage variation for the specified period per directory. Remove all ADR files under the GI_BASE directory, which are older than the time specified Restrict command to only diagnostic files under the GI_BASE Restrict command to only diagnostic files under the database directory. Defaults to all, alternatively specify a database name Use with purge to estimate how many files will be affected and how much disk space will be freed by a potential purge command. Runs as the ADR home owner. So will only be able to purge files this owner has permission to delete May take a while for a large number of files 22

23 Manual Database Log Purge tfactl managelogs show usage tfactl managelogs show variation older <n><m h d> Use -gi to only show grid infrastructure Use database to only show database 23

24 Manual Database Log Purge tfactl managelogs purge older n<m h d> -dryrun tfactl managelogs purge older n<m h d> Use dryrun for a what if 24

25 Disk Usage Snapshots TFA will track disk usage and record snapshots to: tfa/repository/suptools/<node>/managelogs/usage_snapshot/ Snapshot happens every 60 minutes, configurable with: tfactl set diskusagemoninterval=<minutes> Disk usage monitoring is ON by default, configurable with: tfactl set diskusagemon=<on OFF> 25

26 Change Trace Levels Sometimes Oracle Support may require certain specific trace levels to be enabled when reproducing a problem TFA makes it easy to enable and disable the correct trace levels with a single command All the required trace level settings are organised into problem specific profiles The relevant profile trace logging can be enabled with: tfactl dbglevel set <profile> Profiles only available for Clusterware today, Database profiles coming soon.. 26

27 On-Demand Usage 2 Real-time status summary 3 Diagnose with DB tools 4 Perform diagnostic collection Upload diagnostic collection to Oracle Support 5 TFA Oracle Grid Infrastructure & Databases 1 Request desired action on-demand Oracle Support 27

28 One Command SRDCs For certain types of problems Oracle Support will ask you to run a Service Request Data Collection (SRDC) Previously this would have involved: Reading many different support documents Collecting output from many different tasks Gathering lots of different diagnostics Packaging & uploading Now just run: tfactl diagcollect -srdc <srdc_type> -sr <SR#> 28

29 Collection Uploaded to My Oracle Support At the end of the collection the file is automatically uploaded to the SR 29

30 Upload any File With Wallet: tfactl upload -wallet -sr <sr_number> <file1 file2> Without wallet: tfactl upload -user <user_id> -sr <sr_number> <file1 file2> 30

31 Faster & Easier SR Data Collection tfactl diagcollect srdc <srdc_type> -sr <SR#> Type of Problem ORA Errors Other internal database errors Database performance Database patching Database resource XDB installation or invalid object Database install / upgrade ORA ORA ORA ORA ORA ORA ORA internalerror dbperf SRDC dbpatchinstall dbpatchconflict dbunixresources dbxdb dbinstall dbupgrade dbpreupgrade ORA ORA ORA ORA ORA ORA Database storage Type of Problem Excessive SYSAUX Space used by the Automatic Workload Repository (AWR) Database startup / shutdown Data Guard Enterprise Manager tablespace usage metric Enterprise Manager general metrics page or threshold problems - Run all three SRDCs Enterprise Manager target discovery / add Enterprise Manager OMS restart asm SRDC dbawrspace dbshutdown dbstartup dbdataguard emtbsmetrics emdebugon emdebugoff emmetricalert emcliadd emclusdisc emdbsys emgendisc emprocdisc emrestartoms 31

32 One Command SRDCs Examples of What s Collected ORA-04031: tfactl diagcollect srdc ORA IPS Package 2. Patch Listing 3. AWR report 4. Memory information 5. RDA Database Performance tfactl diagcollect srdc dbperf 1. ADDM report 2. AWR for good and problem period 3. AWR Compare Period report 4. ASH report for good and problem period 5. OS Watcher 6. IPS Package (if errors during problem period) 7. ORAchk (performance related checks) 32

33 Manual Data Gathering vs One Command SRDC Manual Data Gathering 1. Generate ADDM reviewing Document Identify good and problem periods and gather AWR reviewing Document Generate AWR compare report (awrddrpt.sql) using good and problem periods 4. Generate ASH report for good and problem periods reviewing Document Collect OSWatcher data reviewing Document Check alert.log if there are any errors during the problem period 7. Find any trace files generated during the problem period 8. Collate and upload all the above files/outputs to SR TFA SRDC 1. Run tfactl diagcollect srdc dbperf 2. Upload resulting zip file to SR 33

34 One Command SRDC Interactive Mode tfactl diagcollect srdc <srdc_type> 4. All required files are identified 5. Trimmed where applicable 1. Enter default for event date/time and database name 6. Package in a zip ready to provide to support 2. Scans system to identify recent 10 events in the system (ORA600 example shown) 3. Once the relevant event is chosen, proceeds with diagnostic collection 34

35 One Command SRDC Silent Mode tfactl diagcollect srdc <srdc_type> -database <db> -for <time> 1. Parameters(date/time, DB name) are provided in the command 2. Does not prompt for any more information 3. All required files are identified 4. Trimmed where applicable 5. Package in a zip ready to provide to support 35

36 Default Collection Run a default diagnostic collection if there is not yet an SRDC about your problem: tfactl diagcollect Will trim & collect all important log files updated in the past 12 hours: Collections stored in the repository directory Change diagcollect timeframe with last <n>h d 36

37 Custom Collection Options Time tfactl diagcollect <option> Option -last <n><h d> Description Collect since the previous n hours or days -from <yyyy-mm-dd> Collect from the date and optionally time specified. Valid date / time formats: "Mon/dd/yyyy hh:mm:ss" or "yyyy-mm-dd hh:mm:ss" or "yyyy-mm-ddthh:mm:ss" or "yyyy-mm-dd " Option from "yyyy-mm-dd" -to "yyyy-mm-dd" -for yyyy-mm-dd Description Collect to the date and optionally time specified. Valid date / time formats: "Mon/dd/yyyy hh:mm:ss" or "yyyy-mm-dd hh:mm:ss" or "yyyy-mm-ddthh:mm:ss" or "yyyy-mm-dd" Collect for the specified date. Valid date formats: "Mon/dd/yyyy" or "yyyy-mm-dd" 37

38 Custom Collection Options Nodes, Databases & Components tfactl diagcollect <option> Option -node -database -asm -crsclient -dbclient -dbwlm -tns -rhp -procinfo -afd -crs -wls Description Limit collection to comma separated list of nodes Limit collection to comma separated list of databases Collect ASM logs Collect Client Logs that are under GIBASE/diag/clients Collect Client Logs that are under DB ORABASE/diag/clients Collect DBWLM logs Collect TNS logs Collect RHP logs Collect Gathers stack and fd from /proc for all processes Collect AFD logs Collect CRS logs Collect WLS logs Option -emagent -oms -ocm -emplugins -em -acfs -install -cfgtools -os -ashhtml -ashtext -awrhtml Description Collect EMAGENT logs Collect OMS logs Collect OCM logs Collect EMPLUGINS logs Collect EM logs Collect ACFS logs and Data Collect Oracle Installation related files Collect CFGTOOLS logs Collect OS files such as /var/log/messages Collect Generate ASH HTML Report Collect Generate ASH TEXT Report Collect AWRHTML logs (requires database option as well) 38

39 Custom Collection Options Misc tfactl diagcollect <option> Option Description -collectdir -tag Collects other files in specified comma separated directory list. Only files updated in the relevant time range (12 hours by default) get collected. Store the collection in a use a custom directory name within the repository, using the tag value specified -z Append the value specified to the collection zip name -nocopy -notrim -silent -nocores Prevent the copying of zip files back to the initiating node Prevent the trimming of files Start collection as a background process so output is not echoed to the console Do not collect core files 39

40 Simple IPS (Incident Packaging Service) interface Database incidents are stored in the ADR (Automatic Diagnostic Repository) IPS packages details about these incidents for later diagnosis TFA can invoke IPS to query and collect these packages tfactl ips <options> or tfactl diagcollect ips <options> Command tfactl ips tfactl ips show incidents tfactl ips show problems tfactl ips show package tfactl diagcollect -ips help tfactl diagcollect ips tfactl diagcollect -ips -adrbasepath <adr_base> -adrhomepath <adr_home> tfactl diagcollect -ips -incident <incident id> tfactl diagcollect -ips -problem <problem id> Description Invoke IPS Show all IPS incidents Show all IPS problems Show all IPS packages Show all available diagcollect IPS options Perform an IPS collection following prompts Perform an IPS collection in silent mode. Collect ADR details about a specific incident id collect ADR details about a specific problem id 40

41 Show Incidents Display details of incidents recorded in ADR tfactl ips show incidents Incident ID Problem Key Create Time 41

42 Show Problems Display details of problems recorded in ADR tfactl ips show problems Problem ID Problem Key Last Incident Last Incident Time 42

43 Show Package Display details of packages recorded in ADR tfactl ips show package List of packages & details of each package 43

44 Collect Packages Select ADR home Rest of collection Generate package Resulting collection zips tfactl diagcollect -ips 44

45 Maintenance Option 1 Applying standard PSUs will automatically update TFA PSUs do not contain Support Tools Bundle updates Option 2 To update with latest TFA & Support Tools Bundle

46 Safe Harbor Statement The preceding is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle s products remains at the sole discretion of Oracle. 46

47