Oracle Trace File Analyzer (TFA) Diagnostics for the Cloud

Oracle Trace File Analyzer (TFA) Diagnostics for the Cloud 12.2.1.3.0

Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle s products remains at the sole discretion of Oracle. 2

Why TFA? Easy to use real-time health monitoring, fault detection & diagnosis via a single interface Secure consolidation of distributed diagnostic collections Continuous availability so you always get what s needed for resolution Machine learning driven, autonomous degradation detection, reduces your overheads 3

Autonomous Usage Diagnostics are collected 2 3 Distributed diagnostics are consolidated and packaged TFA Oracle Grid Infrastructure & Databases 1 TFA detects a fault 4 Notification of fault is sent Oracle Support 5 Diagnostic collection is uploaded to Oracle Support for root cause analysis & resolution 4

On-Demand Usage 2 Real-time status summary 3 Diagnose with DB tools 4 Perform diagnostic collection Upload diagnostic collection to Oracle Support 5 TFA Oracle Grid Infrastructure & Databases 1 Request desired action on-demand Oracle Support 5

Command Interfaces Command line Shell Menu Specify all command options at the command line tfactl <command> 1. Set and change context 2. Run commands from within the shell tfactl tfaclt > database MyDB MyDB tfactl > oratop 1. Select menu navigation options then choose the command you want to run tfactl menu 6

Data Redaction mask_strings.xml <mask_strings> <mask_string> <original>widgetnode1</original> <replacement>node1</replacement> </mask_string> <mask_string> <original>192.168.5.1</original> <replacement>node1-ip</replacement> </mask_string> <mask_string> <original>widgetnode2</original> <replacement>node2</replacement> </mask_string> <mask_string> <original>192.168.5.2</original> <replacement>node2-ip</replacement> </mask_string> </mask_strings> If you want to mask sensitive data Hostname IP address Etc Create mask_strings.xml In the directory tfa_home/resources Define all data replacements you want 7

Configure User Access Users & groups authorized to access tfactl by default Enable or disable user access tfactl access enable tfactl access disable TFA List user access tfactl access lsusers Add users All commands apply cluster wide unless -local used to restrict to local node tfactl access add user <user> [-local] Remove users or everyone tfactl access remove user <user> [-local] tfactl access removeall [-local] Reset to default access Grid Home Owner DB Home Owners Everyone Else tfactl access reset 8

Set Email Notification Addresses TFA can send email notification when faults are detected To set notification email for any problem detected: tfactl set notificationaddress=john.doe@oracle.com To set notification email for specific ORACLE_HOMEs include the OS owner: tfactl set notificationaddress=oracle:another.person@oracle.com 9

Real-time Status Summary High-level summary of all Database components tfactl summary Choose an option to drill down 11

Real-time Status Summary Drill Down Drill downs show real-time analytics & details of any problems found 12

Investigate Logs & Look for Errors Analyze all important recent log entries: tfactl analyze last 1d Search recent log entries: tfactl analyze -search ora-006" -last 8h Searching for ora-00600 14

Perform Analysis Using the Included Tools Tool orachk or exachk oswatcher procwatcher Description Provides health checks for the Oracle stack. Oracle Trace File Analyzer will install either Oracle EXAchk for Engineered Systems, see document 1070954.1 for more details or Oracle ORAchk for all non-engineered Systems, see document 1268927.2 for more details Collects and archives OS metrics. These are useful for instance or node evictions & performance Issues. See document 301137.1 for more details Automates & captures database performance diagnostics and session level hang information. See document 459694.1 for more details oratop Provides near real-time database monitoring. See document 1500864.1 for more details. alertsummary ls pstack Provides summary of events for one or more database or ASM alert files from all nodes Lists all files TFA knows about for a given file name pattern across all nodes Generate process stack for specified processes across all nodes Tool grep summary vi tail param dbglevel history changes calog events managelogs ps Description Search alert or trace files with a given database and file name pattern, for a search string. Provides high level summary of the configuration Opens alert or trace files for viewing a given database and file name pattern in the vi editor Runs a tail on an alert or trace files for a given database and file name pattern Shows all database and OS parameters that match a specified pattern Sets and unsets multiple CRS trace levels with one command Shows the shell history for the tfactl shell Reports changes in the system setup over a given time period. This includes database parameters, OS parameters and patches applied Reports major events from the Cluster Event log Reports warnings and errors seen in the logs Shows disk space usage and purges ADR log and trace files Finds processes Not all tools are included in Grid or Database install. Download from 1513912.1 to get full collection of tools triage Summarize oswatcher/exawatcher data Verify which tools you have installed: tfactl toolstatus 15

ORAchk & EXAchk (Support Tools Bundle) Oracle Stack Health Checks Oracle EXAchk for Engineered Systems Oracle ORAchk for everything else Proactively scans for the most impactful problems across the layers of your stack Simplifies and streamlines how to investigate and analyze which known issues present a risk to you Lightweight tool that runs within your environment without requiring config data to be sent to Oracle High level reports show your system health risks with the ability to drill down into specific problems and understand their resolutions Can be configured to send email notifications when it detects problems Collection Manager, a companion Application Express web app, provides a single dashboard view of collections across your entire enterprise 16

OS Watcher (Support Tools Bundle) Collect & Archive OS Metrics Executes standard UNIX utilities (e.g. vmstat, iostat, ps, etc) on regular intervals Built in Analyzer functionality to summarize, graph and report upon collected metrics Output is Required for node reboot and performance issues Simple to install, extremely lightweight Runs on ALL platforms (Except Windows) MOS Note: 301137.1 OS Watcher Users Guide 17

Procwatcher (Support Tools Bundle) Monitor & Examine Database Processes Single instance & RAC Generates session wait, lock and latch reports as well as call stacks from any problem process(s) Ability to collect stack traces of specific processes using Oracle Tools and OS Debuggers Typically reduces SR resolution for performance related issues Runs on ALL major UNIX Platforms MOS Note: 459694.1 Procwatcher Install Guide 18

oratop (Support Tools Bundle) Near Real-Time Database Monitoring Single instance & RAC Monitoring current database activities Database performance Identifying contentions and bottleneck 19

Analyze Each tool can be run using tfactl in shell mode Start tfactl shell with tfactl Run a tool with the tool name tfactl > orachk 1. Where necessary set context with database <dbname> tfactl > database MyDB 2. Then run tool MyDB tfactl > oratop 3. Clear context with database MyDB tfactl > database 20

Automatic Database Log Purge TFA can automatically purge database logs OFF by default Except on a Domain Service Cluster (DSC), which it is ON by default Turn auto purging on or off: tfactl set managelogsautopurge=<on OFF> Will remove logs older than 30 days configurable with: Purging runs every 60 minutes configurable with: tfactl set managelogsautopurgepolicyage=<n><d h> tfactl set managelogsautopurgeinterval=<minutes> 21

Manual Database Log Purge TFA can manage ADR log and trace files Show disk space usage of individual diagnostic destinations Purge these file types based on diagnostic location and or age: "ALERT, "INCIDENT, "TRACE, "CDUMP, "HM, "UTSCDMP, "LOG tfactl managelogs <options> Option show usage -show variation older <n><m h d> -purge older <n><m h d> gi database [all dbname] -dryrun Description Shows disk space usage per diagnostic directory for both GI and database logs Use to determine per directory disk space growth. Shows the disk usage variation for the specified period per directory. Remove all ADR files under the GI_BASE directory, which are older than the time specified Restrict command to only diagnostic files under the GI_BASE Restrict command to only diagnostic files under the database directory. Defaults to all, alternatively specify a database name Use with purge to estimate how many files will be affected and how much disk space will be freed by a potential purge command. Runs as the ADR home owner. So will only be able to purge files this owner has permission to delete May take a while for a large number of files 22

Manual Database Log Purge tfactl managelogs show usage tfactl managelogs show variation older <n><m h d> Use -gi to only show grid infrastructure Use database to only show database 23

Manual Database Log Purge tfactl managelogs purge older n<m h d> -dryrun tfactl managelogs purge older n<m h d> Use dryrun for a what if 24

Disk Usage Snapshots TFA will track disk usage and record snapshots to: tfa/repository/suptools/<node>/managelogs/usage_snapshot/ Snapshot happens every 60 minutes, configurable with: tfactl set diskusagemoninterval=<minutes> Disk usage monitoring is ON by default, configurable with: tfactl set diskusagemon=<on OFF> 25

Change Trace Levels Sometimes Oracle Support may require certain specific trace levels to be enabled when reproducing a problem TFA makes it easy to enable and disable the correct trace levels with a single command All the required trace level settings are organised into problem specific profiles The relevant profile trace logging can be enabled with: tfactl dbglevel set <profile> Profiles only available for Clusterware today, Database profiles coming soon.. 26

One Command SRDCs For certain types of problems Oracle Support will ask you to run a Service Request Data Collection (SRDC) Previously this would have involved: Reading many different support documents Collecting output from many different tasks Gathering lots of different diagnostics Packaging & uploading Now just run: tfactl diagcollect -srdc <srdc_type> -sr <SR#> 28

Collection Uploaded to My Oracle Support At the end of the collection the file is automatically uploaded to the SR 29

Upload any File With Wallet: tfactl upload -wallet -sr <sr_number> <file1 file2> Without wallet: tfactl upload -user <user_id> -sr <sr_number> <file1 file2> 30

Faster & Easier SR Data Collection tfactl diagcollect srdc <srdc_type> -sr <SR#> Type of Problem ORA Errors Other internal database errors Database performance Database patching Database resource XDB installation or invalid object Database install / upgrade ORA-00020 ORA-00060 ORA-00600 ORA-00700 ORA-01555 ORA-01628 ORA-04030 internalerror dbperf SRDC dbpatchinstall dbpatchconflict dbunixresources dbxdb dbinstall dbupgrade dbpreupgrade ORA-04031 ORA-07445 ORA-27300 ORA-27301 ORA-27302 ORA-30036 Database storage Type of Problem Excessive SYSAUX Space used by the Automatic Workload Repository (AWR) Database startup / shutdown Data Guard Enterprise Manager tablespace usage metric Enterprise Manager general metrics page or threshold problems - Run all three SRDCs Enterprise Manager target discovery / add Enterprise Manager OMS restart asm SRDC dbawrspace dbshutdown dbstartup dbdataguard emtbsmetrics emdebugon emdebugoff emmetricalert emcliadd emclusdisc emdbsys emgendisc emprocdisc emrestartoms 31

One Command SRDCs Examples of What s Collected ORA-04031: tfactl diagcollect srdc ORA-04031 1. IPS Package 2. Patch Listing 3. AWR report 4. Memory information 5. RDA Database Performance tfactl diagcollect srdc dbperf 1. ADDM report 2. AWR for good and problem period 3. AWR Compare Period report 4. ASH report for good and problem period 5. OS Watcher 6. IPS Package (if errors during problem period) 7. ORAchk (performance related checks) 32

Manual Data Gathering vs One Command SRDC Manual Data Gathering 1. Generate ADDM reviewing Document 1680075.1 2. Identify good and problem periods and gather AWR reviewing Document 1903158.1 3. Generate AWR compare report (awrddrpt.sql) using good and problem periods 4. Generate ASH report for good and problem periods reviewing Document 1903145.1 5. Collect OSWatcher data reviewing Document 301137.1 6. Check alert.log if there are any errors during the problem period 7. Find any trace files generated during the problem period 8. Collate and upload all the above files/outputs to SR TFA SRDC 1. Run tfactl diagcollect srdc dbperf 2. Upload resulting zip file to SR 33

One Command SRDC Interactive Mode tfactl diagcollect srdc <srdc_type> 4. All required files are identified 5. Trimmed where applicable 1. Enter default for event date/time and database name 6. Package in a zip ready to provide to support 2. Scans system to identify recent 10 events in the system (ORA600 example shown) 3. Once the relevant event is chosen, proceeds with diagnostic collection 34

One Command SRDC Silent Mode tfactl diagcollect srdc <srdc_type> -database <db> -for <time> 1. Parameters(date/time, DB name) are provided in the command 2. Does not prompt for any more information 3. All required files are identified 4. Trimmed where applicable 5. Package in a zip ready to provide to support 35

Default Collection Run a default diagnostic collection if there is not yet an SRDC about your problem: tfactl diagcollect Will trim & collect all important log files updated in the past 12 hours: Collections stored in the repository directory Change diagcollect timeframe with last <n>h d 36

Custom Collection Options Time tfactl diagcollect <option> Option -last <n><h d> Description Collect since the previous n hours or days -from <yyyy-mm-dd> Collect from the date and optionally time specified. Valid date / time formats: "Mon/dd/yyyy hh:mm:ss" or "yyyy-mm-dd hh:mm:ss" or "yyyy-mm-ddthh:mm:ss" or "yyyy-mm-dd " Option from "yyyy-mm-dd" -to "yyyy-mm-dd" -for yyyy-mm-dd Description Collect to the date and optionally time specified. Valid date / time formats: "Mon/dd/yyyy hh:mm:ss" or "yyyy-mm-dd hh:mm:ss" or "yyyy-mm-ddthh:mm:ss" or "yyyy-mm-dd" Collect for the specified date. Valid date formats: "Mon/dd/yyyy" or "yyyy-mm-dd" 37

Custom Collection Options Nodes, Databases & Components tfactl diagcollect <option> Option -node -database -asm -crsclient -dbclient -dbwlm -tns -rhp -procinfo -afd -crs -wls Description Limit collection to comma separated list of nodes Limit collection to comma separated list of databases Collect ASM logs Collect Client Logs that are under GIBASE/diag/clients Collect Client Logs that are under DB ORABASE/diag/clients Collect DBWLM logs Collect TNS logs Collect RHP logs Collect Gathers stack and fd from /proc for all processes Collect AFD logs Collect CRS logs Collect WLS logs Option -emagent -oms -ocm -emplugins -em -acfs -install -cfgtools -os -ashhtml -ashtext -awrhtml Description Collect EMAGENT logs Collect OMS logs Collect OCM logs Collect EMPLUGINS logs Collect EM logs Collect ACFS logs and Data Collect Oracle Installation related files Collect CFGTOOLS logs Collect OS files such as /var/log/messages Collect Generate ASH HTML Report Collect Generate ASH TEXT Report Collect AWRHTML logs (requires database option as well) 38

Custom Collection Options Misc tfactl diagcollect <option> Option Description -collectdir -tag Collects other files in specified comma separated directory list. Only files updated in the relevant time range (12 hours by default) get collected. Store the collection in a use a custom directory name within the repository, using the tag value specified -z Append the value specified to the collection zip name -nocopy -notrim -silent -nocores Prevent the copying of zip files back to the initiating node Prevent the trimming of files Start collection as a background process so output is not echoed to the console Do not collect core files 39

Simple IPS (Incident Packaging Service) interface Database incidents are stored in the ADR (Automatic Diagnostic Repository) IPS packages details about these incidents for later diagnosis TFA can invoke IPS to query and collect these packages tfactl ips <options> or tfactl diagcollect ips <options> Command tfactl ips tfactl ips show incidents tfactl ips show problems tfactl ips show package tfactl diagcollect -ips help tfactl diagcollect ips tfactl diagcollect -ips -adrbasepath <adr_base> -adrhomepath <adr_home> tfactl diagcollect -ips -incident <incident id> tfactl diagcollect -ips -problem <problem id> Description Invoke IPS Show all IPS incidents Show all IPS problems Show all IPS packages Show all available diagcollect IPS options Perform an IPS collection following prompts Perform an IPS collection in silent mode. Collect ADR details about a specific incident id collect ADR details about a specific problem id 40

Show Incidents Display details of incidents recorded in ADR tfactl ips show incidents Incident ID Problem Key Create Time 41

Show Problems Display details of problems recorded in ADR tfactl ips show problems Problem ID Problem Key Last Incident Last Incident Time 42

Show Package Display details of packages recorded in ADR tfactl ips show package List of packages & details of each package 43

Collect Packages Select ADR home Rest of collection Generate package Resulting collection zips tfactl diagcollect -ips 44

Maintenance Option 1 Applying standard PSUs will automatically update TFA PSUs do not contain Support Tools Bundle updates Option 2 To update with latest TFA & Support Tools Bundle 1513912.1 45

Safe Harbor Statement The preceding is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle s products remains at the sole discretion of Oracle. 46