Oracle WebLogic Server 11g: Diagnostics and Troubleshooting

Transcription

1 Oracle WebLogic Server 11g: Diagnostics and Troubleshooting Volume II Student Guide D61523GC20 Edition 2.0 March 2011 D72554

2 Author Bill Bell Technical Contributors and Reviewers Will Lyons TJ Palazzolo Serge Moiseev Editors Richard Wallis Malavika Jinka Publishers Jobi Varghese Shaik Mahaboob Basha Copyright 2011, Oracle and/or it affiliates. All rights reserved. Disclaimer This document contains proprietary information and is protected by copyright and other intellectual property laws. You may copy and print this document solely for your own use in an Oracle training course. The document may not be modified or altered in any way. Except where your use constitutes "fair use" under copyright law, you may not use, share, download, upload, copy, print, display, perform, reproduce, publish, license, post, transmit, or distribute this document in whole or in part without the express authorization of Oracle. The information contained in this document is subject to change without notice. If you find any problems in the document, please report them in writing to: Oracle University, 500 Oracle Parkway, Redwood Shores, California USA. This document is not warranted to be error-free. Restricted Rights Notice If this documentation is delivered to the United States Government or anyone using the documentation on behalf of the United States Government, the following notice is applicable: U.S. GOVERNMENT RIGHTS The U.S. Government s rights to use, modify, reproduce, release, perform, display, or disclose these training materials are restricted by the terms of the applicable Oracle license agreement and/or the applicable U.S. Government contract. Trademark Notice Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.

3 Contents 1 Course Overview Course Objectives 1-2 Target Audience 1-3 Introductions 1-4 Course Schedule 1-5 Course Appendix 1-7 Course Practices 1-8 Classroom Guidelines 1-9 For More Information 1-10 Related Training 1-11 Oracle by Example (OBE) WLST Monitoring Objectives 2-2 WLS Domains: Review 2-3 Java Management Extension (JMX): Review 2-4 WLS MBean Hierarchies 2-5 WLS MBean Reference Documentation 2-6 Console Monitoring: Review 2-8 WebLogic Scripting Tool (WLST): Review 2-9 WLST MBean Syntax: Review 2-10 Domain Runtime 2-11 Basic Jython Syntax: Review 2-12 Basic WLST Commands 2-13 Variable Declaration 2-14 Password Management 2-15 Error Handling 2-16 File I/O 2-17 Standard Jython Libraries 2-18 WLST Example: Monitor a JMS Server 2-19 Quiz 2-20 Summary 2-23 Practice 2-1 Connecting to the Classroom Grid 2-24 Practice 2-2 Developing a Custom Monitoring Script Guardian Objectives 3-2 Guardian Capabilities 3-3 iii

4 Using Guardian 3-4 Guardian Architecture 3-5 Agent Installation 3-6 Collected Data 3-7 Client Installation 3-8 Guardian User Interface 3-9 Activating a Domain 3-10 Creating a Domain Inventory 3-11 Signatures and Bundles 3-12 Updating the Signature Repository 3-13 Signature Annotations 3-14 Evaluating a Domain 3-15 Evaluation Summary 3-16 Generating a Support Request 3-17 Command-Line Interface 3-18 Quiz 3-19 Summary 3-22 Practice 3-1 Using Guardian to Evaluate a Domain Diagnostic Framework Essentials Objectives 4-2 Road Map 4-3 WebLogic Diagnostic Framework (WLDF) 4-4 WLDF Architecture 4-5 WLS Logging: Review 4-6 Log Severity Thresholds 4-7 Application Logging 4-8 Server Logging Bridge 4-9 WLDF Configuration: Overview 4-10 Diagnostic Images 4-11 Capturing a Server Diagnostic Image 4-12 WLST: Downloading Diagnostic Image Files 4-13 Diagnostic Archives 4-14 Configuring Server Diagnostic Archives 4-15 Archive Retirement Policies 4-16 Archive Database Schema 4-17 Viewing Archive Contents 4-18 Creating a Diagnostic Module 4-19 WLDF WLST Examples 4-20 Section Summary 4-22 Road Map 4-23 iv

5 Harvester Architecture 4-24 Metric Collector Definitions 4-25 Configuring a Metric Collector 4-26 Watches and Notifications 4-28 Configuring a Watch 4-29 Watch Alarms 4-31 Configuring a JMS Notification 4-32 Configuring an Notification 4-33 Harvester WLST: Example 4-34 Watch WLST: Example 4-35 WLDF Sample Framework 4-36 Section Summary 4-37 Practice 4-1 Harvesting Diagnostic Metrics 4-38 Road Map 4-39 New Monitoring Dashboard 4-40 Viewing the Dashboard 4-41 Monitoring Dashboard Interface 4-42 Views 4-43 Built-In Views 4-44 Metric Browser 4-45 Creating a Custom View 4-46 Anatomy of a Chart 4-47 Chart and Graph Properties 4-48 Chart Styles 4-49 Current and Historical Data 4-50 Section Summary 4-51 Practice 4-2 Monitoring Diagnostic Metrics 4-52 Road Map 4-53 Subsystem Debugging 4-54 Console Debug Scopes 4-55 Debug Scopes: Examples 4-56 Debug Logging 4-57 WLST Debugging: Examples 4-58 Section Summary 4-59 Quiz 4-60 Summary Diagnostic Instrumentation Objectives 5-2 Road Map 5-3 Instrumentation Scenarios 5-4 v

6 Instrumentation Architecture 5-5 Monitor Actions 5-6 Application-Scoped Modules 5-8 WLS Monitor Library 5-9 Deployment Plan Review 5-11 WLDF Deployment Plans 5-12 WLDF Deployment Plan: Example 5-13 WLDF Hot Swap 5-14 Configuring a System-Scoped Monitor 5-15 Configuring an Application-Scoped Monitor 5-17 Aspect-Oriented Programming (AOP) Concepts 5-18 Custom Monitors 5-19 Instrumentation WLST: Example 5-20 Instrumentation and Request Performance 5-21 Section Summary 5-22 Practice 5-1 Configuring and Monitoring Diagnostic Events 5-23 Road Map 5-24 Request Context ID 5-25 Viewing Context IDs 5-26 Request Dying 5-27 Available Dyes 5-28 Configuring a Dye Injection Monitor 5-29 Event Filtering 5-30 Configuring Dye Masks 5-31 Event Throttling 5-32 Configuring Throttle Properties 5-33 Section Summary 5-34 Quiz 5-35 Summary 5-38 Practice 5-2 Tracing a Client Request JVM Diagnostics Objectives 6-2 Road Map 6-3 Basic Java Concepts 6-4 Java Virtual Machine (JVM): Review 6-5 Oracle JVM Support 6-6 JVM Recommendations 6-7 JVM Memory 6-8 Garbage Collection 6-9 Sun HotSpot Garbage Collection 6-10 vi

7 Garbage Collection (GC) Types 6-11 Setting WLS JVM Arguments 6-12 Basic Sun JVM Arguments 6-13 JRockit Garbage Collection 6-14 Basic JRockit JVM Arguments 6-15 Out of Memory 6-16 Out-of-Memory Response 6-17 Memory Leak 6-18 JVM Crash 6-19 JVM Error Log 6-20 Section Summary 6-21 Road Map 6-22 JVM Tool Varieties 6-23 Java Stack Trace 6-24 Java Thread Dump: Overview 6-25 Thread Dump Signal 6-26 JVM Crash Actions 6-27 Verbose GC 6-28 Sun JVM Profiler Agent 6-29 Sun JVM Diagnostic Tools: Overview 6-30 Sun Diagnostic Tools: Examples 6-31 JVisualVM 6-33 Using JVisualVM 6-34 Section Summary 6-36 Practice 6-1 Troubleshooting a Running JVM 6-37 Road Map 6-38 Console JVM Monitoring 6-39 JVM WLST: Example 6-40 WLS Low Memory Detection 6-41 Configuring Low Memory Detection 6-42 Section Summary 6-43 Road Map 6-44 JRockit Diagnostic Tools: Overview 6-45 JRockit Diagnostic Tools: Examples 6-46 Management Communication 6-47 JRockit Mission Control (JRMC) 6-48 JRockit Discovery Protocol (JDP) 6-49 JVM Browser 6-50 Management Console: Features 6-51 Management Console: General > Overview 6-52 Management Console: Runtime > Threads 6-53 vii

8 Management Console: MBeans > Triggers 6-54 JRockit Flight Recorder (JFR) 6-55 Integration of JRockit Flight Recorder and WLDF 6-56 Starting the Flight Recorder from JRMC 6-57 Flight Recorder Output 6-58 General > Overview 6-59 Memory: Object Statistics 6-60 Code > Overview 6-61 Memory Leak Detector (Memleak): Features 6-62 Memleak: Trend Tab 6-63 Memleak: Type Graph 6-64 Section Summary 6-65 Quiz 6-66 Summary 6-70 Practice 6-2 Troubleshooting Applications on JRockit Troubleshooting Java Applications Objectives 7-2 Java Exception-Handling Concepts 7-3 Exception Chains 7-4 Class Not Found Errors 7-5 Class Cast Errors 7-6 Classpath: Review 7-7 WebLogic Start Script: Review 7-8 Viewing the WLS Classpath 7-9 Manifest Files and the Classpath 7-10 Domain Libraries 7-11 Java Class Loaders 7-12 Searching Class Loaders 7-13 Searching Class Loaders: Example 7-14 Default WLS Class Loader Hierarchy 7-15 Java EE Packaging: Review 7-16 Prefer Web Application Classes 7-17 Prefer Enterprise Application Classes 7-18 Client Library Errors 7-19 Null Pointer Errors 7-20 Stack Overflow Errors 7-21 Too Many Open Files Errors 7-22 Quiz 7-23 Summary 7-25 Practice 7-1 Investigating Classpath Problems 7-26 viii

9 8 Troubleshooting Servers Objectives 8-2 Road Map 8-3 WLS Message Catalog: Review 8-4 Server Startup Errors 8-5 Boot Identity Errors 8-6 WLS Native Libraries 8-7 Setting the Native Library Path 8-8 Causes of Unresponsive Servers 8-9 WLS Threading Architecture 8-10 Execute Thread State 8-11 Work Managers 8-12 Work Manager Architecture 8-13 Creating a Work Manager 8-14 Creating and Using a Request Class 8-15 Assigning Work Managers to Applications 8-16 Monitoring a Server Thread Pool 8-17 Monitoring Individual Server Threads 8-18 Server Monitoring: WLST Examples 8-19 Server WLDF Image Contents 8-20 Java Deadlock Concepts 8-21 Thread Analysis 8-22 Lock Chains 8-23 Stuck Thread Detection 8-24 Overload Protection 8-25 Configuring Overload Protection 8-26 Section Summary 8-27 Practice 8-1 Investigating Server Problems 8-28 Road Map 8-29 WLS Deployment: Review 8-30 Deployment Errors 8-32 Application Staging 8-33 Deployment Memory Errors 8-34 Shared Library: Review 8-35 Library Errors 8-36 Deployment Debug Flags 8-37 Application Error Handling 8-38 Application Monitoring: Review 8-39 Application Monitoring: WLST Examples 8-40 Section Summary 8-41 ix

10 Quiz 8-42 Summary Troubleshooting JDBC Objectives 9-2 JDBC: Review 9-3 Data Sources: Review 9-4 JDBC Management: WLST Examples 9-5 JDBC Runtime Attributes 9-6 JDBC Monitoring: WLST Examples 9-7 JDBC WLDF Image Contents 9-8 JDBC WLDF Monitor: Review 9-9 Data Source Diagnostic Profiling 9-10 Configuring Diagnostic Profiling 9-11 JDBC Debug Flags 9-12 Other JDBC Debugging Tools 9-13 Common Configuration Errors 9-14 Configuration Error Examples 9-15 Insufficient Connection Errors 9-16 Connection Leaks 9-17 Database Cursor Considerations 9-18 Common Connection Errors 9-19 Statement Timeout 9-20 Data Sources and Database Availability 9-21 Retry Frequency and Login Timeout 9-22 Connection Testing: Review 9-23 Testing Trusted Connections 9-24 Firewall Considerations 9-25 Multi Data Source: Overview 9-26 Multi Data Source: Architecture 9-27 Java Persistence API (JPA): Overview 9-28 JPA Configuration: Overview 9-29 Troubleshooting JPA: Overview 9-30 Quiz 9-31 Summary 9-34 Practice 9-1 Investigating JDBC Problems Troubleshooting JMS Objectives 10-2 JMS: Review 10-3 WebLogic JMS Configuration: Review 10-5 x

11 JMS Transactions: Review 10-7 JMS Management: Overview 10-8 Console JMS Management 10-9 JMS Management: WLST Examples JMS Runtime MBean Hierarchy JMS Monitoring: WLST Examples JMS Diagnostic Image Contents JMS Message Logging Configuring JMS Logging JMS Debug Flags Message Type Considerations Common Configuration Errors JMS Client Libraries Out-of-Memory Errors and Quotas Configuring a JMS Server Quota Creating a Destination Quota Message Paging Too Many Pending Messages Quota Blocking Policies Thresholds and Flow Control Configuring Thresholds Tuning Flow Control Lost Messages Time to Live (TTL) Expiration Policies Delivery Mode Message Redelivery Time to Deliver (TTD) Durable Subscriber Review Monitoring and Managing Subscriptions Duplicate Messages Poison Messages Consumer Acknowledgement Modes Messages Out of Sequence Unit of Order (UOO): Overview Unit of Work (UOW): Overview Message-Driven Beans (MDBs): Review MDB Capabilities MDB Runtime Attributes MDB Diagnostics and Debugging Quiz xi

12 Summary Practice 10-1 Investigating JMS Problems Troubleshooting Security Objectives 11-2 Road Map 11-3 Secure Sockets Layer (SSL): Review 11-4 SSL Communication: Review 11-5 WebLogic SSL Scenarios 11-6 Proxy Server SSL Scenarios 11-8 Keystore: Review to here 11-9 Trust Keystores Keytool: Review WebLogic SSL Support SSL Configuration: Review Restarting SSL SSL Debug Flags SSL Handshake Trace Other SSL Traces Invalid Format or Cipher Errors Certificate Validation Errors Host Name Verification Errors Certificate Chains WLS Chain Validation Utility Missing Constraint or Policy Errors Section Summary Practice 11-1 Investigating SSL Problems Road Map Security Realm: Review Security Provider Stores Some Security Providers Embedded LDAP: Review Embedded LDAP Backups Embedded LDAP Synchronization Issues Viewing Embedded LDAP Contents LDAP Concepts LDAP Structure LDAP Search Operations Resetting Admin Password in Embedded LDAP Database Store Cache Synchronization Issues Auditing Provider xii

13 Security Audit Events Configuring the Auditing Provider Realm Debug Flags Typical Authentication Trace Typical Role Mapping Trace Typical Authorization Trace LDAP Trace Log Authentication Provider Control Flags External LDAP Authentication Providers LDAP Provider Configuration: Overview Common LDAP Issues Section Summary Quiz Summary Practice 11-2 Investigating Security Realm Problems Troubleshooting Node Manager Objectives 12-2 Node Manager (NM): Review 12-3 Node Manager Types: Review 12-4 Node Manager Configuration: Review 12-5 Basic Java Node Manager Properties 12-6 Java Node Manager Logging 12-7 Java Node Manager Availability 12-8 Basic Script Node Manager Interface 12-9 Node Manager Server Start Parameters Configuring Server Start Parameters Monitoring Node Managers Node Manager: WLST Examples Common Configuration Errors Generating Template Properties for Java NM Node Manager Authentication Configuring Node Manager Credentials Node Manager Trusted Domains Machine Enrollment Server Boot Identity Configuring Node Manager SSL Quiz Summary Practice 12-1 Investigating Node Manager Problems xiii

14 13 Troubleshooting Clusters Objectives 13-2 Road Map 13-3 Cluster Review 13-4 Proxy Plug-in Review 13-5 Obtaining and Using Plug-Ins 13-6 Oracle HTTP Server (OHS) Review 13-7 Oracle Process Manager and Notification Server (OPMN) Review 13-9 OPMNCTL Examples OHS Logs Plug-in Configuration Review Basic Plug-in Parameters Proxy Connection Architecture Dynamic Server List Connection Parameters Common Connectivity Issues Proxy SSL Issues Proxy Debug Page Proxy Debug Log Typical Proxy Trace Section Summary Practice 13-1 Investigating Proxy Problems Road Map Cluster Communication Review Unicast Architecture Session Management Review Session Persistence Review In-Memory Replication Review Cluster Monitoring WLST Examples Session Monitoring WLST Examples Session Monitoring Attribute Session Instrumentation Cluster Debug Flags Typical Cluster Heartbeat Trace Typical Replication Trace: Primary Typical Replication Trace: Secondary Common Replication Issues HttpSession API Overview Serialization Overview Serialization Debug Messages Section Summary xiv

15 Quiz Lesson Summary Practice 13-2 Investigating Cluster Replication Problems Appendix A: WebLogic SNMP Simple Network Management Protocol (SNMP) A-2 SNMP Architecture A-3 Object Identifier (OID) A-4 Management Information Base (MIB) A-5 WLS MIB and OIDs A-6 Common SNMP Message Types A-7 WLS SNMP Architecture A-8 Creating an SNMP Agent A-10 Configuring an SNMP Agent A-11 SNMP Channels A-12 WLS SNMP Notifications A-13 Creating Trap Monitors A-14 Creating Trap Destinations A-15 SNMP Security A-16 Configuring Agent Security A-17 Configuring SNMP V3 Credentials A-18 Configuring Trap Destination Security A-19 WLS SNMP Utility A-20 xv

16 xvi