Oracle WebLogic Server 11g: Diagnostics and Troubleshooting Volume II Student Guide D61523GC20 Edition 2.0 March 2011 D72554
Author Bill Bell Technical Contributors and Reviewers Will Lyons TJ Palazzolo Serge Moiseev Editors Richard Wallis Malavika Jinka Publishers Jobi Varghese Shaik Mahaboob Basha Copyright 2011, Oracle and/or it affiliates. All rights reserved. Disclaimer This document contains proprietary information and is protected by copyright and other intellectual property laws. You may copy and print this document solely for your own use in an Oracle training course. The document may not be modified or altered in any way. Except where your use constitutes "fair use" under copyright law, you may not use, share, download, upload, copy, print, display, perform, reproduce, publish, license, post, transmit, or distribute this document in whole or in part without the express authorization of Oracle. The information contained in this document is subject to change without notice. If you find any problems in the document, please report them in writing to: Oracle University, 500 Oracle Parkway, Redwood Shores, California 94065 USA. This document is not warranted to be error-free. Restricted Rights Notice If this documentation is delivered to the United States Government or anyone using the documentation on behalf of the United States Government, the following notice is applicable: U.S. GOVERNMENT RIGHTS The U.S. Government s rights to use, modify, reproduce, release, perform, display, or disclose these training materials are restricted by the terms of the applicable Oracle license agreement and/or the applicable U.S. Government contract. Trademark Notice Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.
Contents 1 Course Overview Course Objectives 1-2 Target Audience 1-3 Introductions 1-4 Course Schedule 1-5 Course Appendix 1-7 Course Practices 1-8 Classroom Guidelines 1-9 For More Information 1-10 Related Training 1-11 Oracle by Example (OBE) 1-12 2 WLST Monitoring Objectives 2-2 WLS Domains: Review 2-3 Java Management Extension (JMX): Review 2-4 WLS MBean Hierarchies 2-5 WLS MBean Reference Documentation 2-6 Console Monitoring: Review 2-8 WebLogic Scripting Tool (WLST): Review 2-9 WLST MBean Syntax: Review 2-10 Domain Runtime 2-11 Basic Jython Syntax: Review 2-12 Basic WLST Commands 2-13 Variable Declaration 2-14 Password Management 2-15 Error Handling 2-16 File I/O 2-17 Standard Jython Libraries 2-18 WLST Example: Monitor a JMS Server 2-19 Quiz 2-20 Summary 2-23 Practice 2-1 Connecting to the Classroom Grid 2-24 Practice 2-2 Developing a Custom Monitoring Script 2-25 3 Guardian Objectives 3-2 Guardian Capabilities 3-3 iii
Using Guardian 3-4 Guardian Architecture 3-5 Agent Installation 3-6 Collected Data 3-7 Client Installation 3-8 Guardian User Interface 3-9 Activating a Domain 3-10 Creating a Domain Inventory 3-11 Signatures and Bundles 3-12 Updating the Signature Repository 3-13 Signature Annotations 3-14 Evaluating a Domain 3-15 Evaluation Summary 3-16 Generating a Support Request 3-17 Command-Line Interface 3-18 Quiz 3-19 Summary 3-22 Practice 3-1 Using Guardian to Evaluate a Domain 3-23 4 Diagnostic Framework Essentials Objectives 4-2 Road Map 4-3 WebLogic Diagnostic Framework (WLDF) 4-4 WLDF Architecture 4-5 WLS Logging: Review 4-6 Log Severity Thresholds 4-7 Application Logging 4-8 Server Logging Bridge 4-9 WLDF Configuration: Overview 4-10 Diagnostic Images 4-11 Capturing a Server Diagnostic Image 4-12 WLST: Downloading Diagnostic Image Files 4-13 Diagnostic Archives 4-14 Configuring Server Diagnostic Archives 4-15 Archive Retirement Policies 4-16 Archive Database Schema 4-17 Viewing Archive Contents 4-18 Creating a Diagnostic Module 4-19 WLDF WLST Examples 4-20 Section Summary 4-22 Road Map 4-23 iv
Harvester Architecture 4-24 Metric Collector Definitions 4-25 Configuring a Metric Collector 4-26 Watches and Notifications 4-28 Configuring a Watch 4-29 Watch Alarms 4-31 Configuring a JMS Notification 4-32 Configuring an Email Notification 4-33 Harvester WLST: Example 4-34 Watch WLST: Example 4-35 WLDF Sample Framework 4-36 Section Summary 4-37 Practice 4-1 Harvesting Diagnostic Metrics 4-38 Road Map 4-39 New Monitoring Dashboard 4-40 Viewing the Dashboard 4-41 Monitoring Dashboard Interface 4-42 Views 4-43 Built-In Views 4-44 Metric Browser 4-45 Creating a Custom View 4-46 Anatomy of a Chart 4-47 Chart and Graph Properties 4-48 Chart Styles 4-49 Current and Historical Data 4-50 Section Summary 4-51 Practice 4-2 Monitoring Diagnostic Metrics 4-52 Road Map 4-53 Subsystem Debugging 4-54 Console Debug Scopes 4-55 Debug Scopes: Examples 4-56 Debug Logging 4-57 WLST Debugging: Examples 4-58 Section Summary 4-59 Quiz 4-60 Summary 4-64 5 Diagnostic Instrumentation Objectives 5-2 Road Map 5-3 Instrumentation Scenarios 5-4 v
Instrumentation Architecture 5-5 Monitor Actions 5-6 Application-Scoped Modules 5-8 WLS Monitor Library 5-9 Deployment Plan Review 5-11 WLDF Deployment Plans 5-12 WLDF Deployment Plan: Example 5-13 WLDF Hot Swap 5-14 Configuring a System-Scoped Monitor 5-15 Configuring an Application-Scoped Monitor 5-17 Aspect-Oriented Programming (AOP) Concepts 5-18 Custom Monitors 5-19 Instrumentation WLST: Example 5-20 Instrumentation and Request Performance 5-21 Section Summary 5-22 Practice 5-1 Configuring and Monitoring Diagnostic Events 5-23 Road Map 5-24 Request Context ID 5-25 Viewing Context IDs 5-26 Request Dying 5-27 Available Dyes 5-28 Configuring a Dye Injection Monitor 5-29 Event Filtering 5-30 Configuring Dye Masks 5-31 Event Throttling 5-32 Configuring Throttle Properties 5-33 Section Summary 5-34 Quiz 5-35 Summary 5-38 Practice 5-2 Tracing a Client Request 5-39 6 JVM Diagnostics Objectives 6-2 Road Map 6-3 Basic Java Concepts 6-4 Java Virtual Machine (JVM): Review 6-5 Oracle JVM Support 6-6 JVM Recommendations 6-7 JVM Memory 6-8 Garbage Collection 6-9 Sun HotSpot Garbage Collection 6-10 vi
Garbage Collection (GC) Types 6-11 Setting WLS JVM Arguments 6-12 Basic Sun JVM Arguments 6-13 JRockit Garbage Collection 6-14 Basic JRockit JVM Arguments 6-15 Out of Memory 6-16 Out-of-Memory Response 6-17 Memory Leak 6-18 JVM Crash 6-19 JVM Error Log 6-20 Section Summary 6-21 Road Map 6-22 JVM Tool Varieties 6-23 Java Stack Trace 6-24 Java Thread Dump: Overview 6-25 Thread Dump Signal 6-26 JVM Crash Actions 6-27 Verbose GC 6-28 Sun JVM Profiler Agent 6-29 Sun JVM Diagnostic Tools: Overview 6-30 Sun Diagnostic Tools: Examples 6-31 JVisualVM 6-33 Using JVisualVM 6-34 Section Summary 6-36 Practice 6-1 Troubleshooting a Running JVM 6-37 Road Map 6-38 Console JVM Monitoring 6-39 JVM WLST: Example 6-40 WLS Low Memory Detection 6-41 Configuring Low Memory Detection 6-42 Section Summary 6-43 Road Map 6-44 JRockit Diagnostic Tools: Overview 6-45 JRockit Diagnostic Tools: Examples 6-46 Management Communication 6-47 JRockit Mission Control (JRMC) 6-48 JRockit Discovery Protocol (JDP) 6-49 JVM Browser 6-50 Management Console: Features 6-51 Management Console: General > Overview 6-52 Management Console: Runtime > Threads 6-53 vii
Management Console: MBeans > Triggers 6-54 JRockit Flight Recorder (JFR) 6-55 Integration of JRockit Flight Recorder and WLDF 6-56 Starting the Flight Recorder from JRMC 6-57 Flight Recorder Output 6-58 General > Overview 6-59 Memory: Object Statistics 6-60 Code > Overview 6-61 Memory Leak Detector (Memleak): Features 6-62 Memleak: Trend Tab 6-63 Memleak: Type Graph 6-64 Section Summary 6-65 Quiz 6-66 Summary 6-70 Practice 6-2 Troubleshooting Applications on JRockit 6-71 7 Troubleshooting Java Applications Objectives 7-2 Java Exception-Handling Concepts 7-3 Exception Chains 7-4 Class Not Found Errors 7-5 Class Cast Errors 7-6 Classpath: Review 7-7 WebLogic Start Script: Review 7-8 Viewing the WLS Classpath 7-9 Manifest Files and the Classpath 7-10 Domain Libraries 7-11 Java Class Loaders 7-12 Searching Class Loaders 7-13 Searching Class Loaders: Example 7-14 Default WLS Class Loader Hierarchy 7-15 Java EE Packaging: Review 7-16 Prefer Web Application Classes 7-17 Prefer Enterprise Application Classes 7-18 Client Library Errors 7-19 Null Pointer Errors 7-20 Stack Overflow Errors 7-21 Too Many Open Files Errors 7-22 Quiz 7-23 Summary 7-25 Practice 7-1 Investigating Classpath Problems 7-26 viii
8 Troubleshooting Servers Objectives 8-2 Road Map 8-3 WLS Message Catalog: Review 8-4 Server Startup Errors 8-5 Boot Identity Errors 8-6 WLS Native Libraries 8-7 Setting the Native Library Path 8-8 Causes of Unresponsive Servers 8-9 WLS Threading Architecture 8-10 Execute Thread State 8-11 Work Managers 8-12 Work Manager Architecture 8-13 Creating a Work Manager 8-14 Creating and Using a Request Class 8-15 Assigning Work Managers to Applications 8-16 Monitoring a Server Thread Pool 8-17 Monitoring Individual Server Threads 8-18 Server Monitoring: WLST Examples 8-19 Server WLDF Image Contents 8-20 Java Deadlock Concepts 8-21 Thread Analysis 8-22 Lock Chains 8-23 Stuck Thread Detection 8-24 Overload Protection 8-25 Configuring Overload Protection 8-26 Section Summary 8-27 Practice 8-1 Investigating Server Problems 8-28 Road Map 8-29 WLS Deployment: Review 8-30 Deployment Errors 8-32 Application Staging 8-33 Deployment Memory Errors 8-34 Shared Library: Review 8-35 Library Errors 8-36 Deployment Debug Flags 8-37 Application Error Handling 8-38 Application Monitoring: Review 8-39 Application Monitoring: WLST Examples 8-40 Section Summary 8-41 ix
Quiz 8-42 Summary 8-45 9 Troubleshooting JDBC Objectives 9-2 JDBC: Review 9-3 Data Sources: Review 9-4 JDBC Management: WLST Examples 9-5 JDBC Runtime Attributes 9-6 JDBC Monitoring: WLST Examples 9-7 JDBC WLDF Image Contents 9-8 JDBC WLDF Monitor: Review 9-9 Data Source Diagnostic Profiling 9-10 Configuring Diagnostic Profiling 9-11 JDBC Debug Flags 9-12 Other JDBC Debugging Tools 9-13 Common Configuration Errors 9-14 Configuration Error Examples 9-15 Insufficient Connection Errors 9-16 Connection Leaks 9-17 Database Cursor Considerations 9-18 Common Connection Errors 9-19 Statement Timeout 9-20 Data Sources and Database Availability 9-21 Retry Frequency and Login Timeout 9-22 Connection Testing: Review 9-23 Testing Trusted Connections 9-24 Firewall Considerations 9-25 Multi Data Source: Overview 9-26 Multi Data Source: Architecture 9-27 Java Persistence API (JPA): Overview 9-28 JPA Configuration: Overview 9-29 Troubleshooting JPA: Overview 9-30 Quiz 9-31 Summary 9-34 Practice 9-1 Investigating JDBC Problems 9-35 10 Troubleshooting JMS Objectives 10-2 JMS: Review 10-3 WebLogic JMS Configuration: Review 10-5 x
JMS Transactions: Review 10-7 JMS Management: Overview 10-8 Console JMS Management 10-9 JMS Management: WLST Examples 10-10 JMS Runtime MBean Hierarchy 10-11 JMS Monitoring: WLST Examples 10-12 JMS Diagnostic Image Contents 10-13 JMS Message Logging 10-14 Configuring JMS Logging 10-15 JMS Debug Flags 10-16 Message Type Considerations 10-17 Common Configuration Errors 10-18 JMS Client Libraries 10-20 Out-of-Memory Errors and Quotas 10-21 Configuring a JMS Server Quota 10-22 Creating a Destination Quota 10-23 Message Paging 10-24 Too Many Pending Messages 10-25 Quota Blocking Policies 10-26 Thresholds and Flow Control 10-27 Configuring Thresholds 10-28 Tuning Flow Control 10-29 Lost Messages 10-30 Time to Live (TTL) 10-31 Expiration Policies 10-32 Delivery Mode 10-33 Message Redelivery 10-34 Time to Deliver (TTD) 10-35 Durable Subscriber Review 10-36 Monitoring and Managing Subscriptions 10-37 Duplicate Messages 10-38 Poison Messages 10-39 Consumer Acknowledgement Modes 10-40 Messages Out of Sequence 10-41 Unit of Order (UOO): Overview 10-42 Unit of Work (UOW): Overview 10-43 Message-Driven Beans (MDBs): Review 10-44 MDB Capabilities 10-45 MDB Runtime Attributes 10-46 MDB Diagnostics and Debugging 10-47 Quiz 10-48 xi
Summary 10-51 Practice 10-1 Investigating JMS Problems 10-52 11 Troubleshooting Security Objectives 11-2 Road Map 11-3 Secure Sockets Layer (SSL): Review 11-4 SSL Communication: Review 11-5 WebLogic SSL Scenarios 11-6 Proxy Server SSL Scenarios 11-8 Keystore: Review to here 11-9 Trust Keystores 11-10 Keytool: Review 11-11 WebLogic SSL Support 11-13 SSL Configuration: Review 11-14 Restarting SSL 11-16 SSL Debug Flags 11-17 SSL Handshake Trace 11-18 Other SSL Traces 11-19 Invalid Format or Cipher Errors 11-20 Certificate Validation Errors 11-21 Host Name Verification Errors 11-22 Certificate Chains 11-23 WLS Chain Validation Utility 11-24 Missing Constraint or Policy Errors 11-25 Section Summary 11-26 Practice 11-1 Investigating SSL Problems 11-27 Road Map 11-28 Security Realm: Review 11-29 Security Provider Stores 11-30 Some Security Providers 11-31 Embedded LDAP: Review 11-32 Embedded LDAP Backups 11-33 Embedded LDAP Synchronization Issues 11-34 Viewing Embedded LDAP Contents 11-35 LDAP Concepts 11-36 LDAP Structure 11-37 LDAP Search Operations 11-38 Resetting Admin Password in Embedded LDAP 11-39 Database Store Cache Synchronization Issues 11-40 Auditing Provider 11-41 xii
Security Audit Events 11-42 Configuring the Auditing Provider 11-43 Realm Debug Flags 11-44 Typical Authentication Trace 11-45 Typical Role Mapping Trace 11-47 Typical Authorization Trace 11-48 LDAP Trace Log 11-49 Authentication Provider Control Flags 11-50 External LDAP Authentication Providers 11-52 LDAP Provider Configuration: Overview 11-53 Common LDAP Issues 11-56 Section Summary 11-57 Quiz 11-58 Summary 11-62 Practice 11-2 Investigating Security Realm Problems 11-63 12 Troubleshooting Node Manager Objectives 12-2 Node Manager (NM): Review 12-3 Node Manager Types: Review 12-4 Node Manager Configuration: Review 12-5 Basic Java Node Manager Properties 12-6 Java Node Manager Logging 12-7 Java Node Manager Availability 12-8 Basic Script Node Manager Interface 12-9 Node Manager Server Start Parameters 12-11 Configuring Server Start Parameters 12-13 Monitoring Node Managers 12-14 Node Manager: WLST Examples 12-15 Common Configuration Errors 12-16 Generating Template Properties for Java NM 12-17 Node Manager Authentication 12-18 Configuring Node Manager Credentials 12-19 Node Manager Trusted Domains 12-20 Machine Enrollment 12-21 Server Boot Identity 12-22 Configuring Node Manager SSL 12-23 Quiz 12-24 Summary 12-26 Practice 12-1 Investigating Node Manager Problems 12-27 xiii
13 Troubleshooting Clusters Objectives 13-2 Road Map 13-3 Cluster Review 13-4 Proxy Plug-in Review 13-5 Obtaining and Using Plug-Ins 13-6 Oracle HTTP Server (OHS) Review 13-7 Oracle Process Manager and Notification Server (OPMN) Review 13-9 OPMNCTL Examples 13-10 OHS Logs 13-11 Plug-in Configuration Review 13-12 Basic Plug-in Parameters 13-13 Proxy Connection Architecture 13-14 Dynamic Server List 13-16 Connection Parameters 13-17 Common Connectivity Issues 13-18 Proxy SSL Issues 13-19 Proxy Debug Page 13-20 Proxy Debug Log 13-22 Typical Proxy Trace 13-23 Section Summary 13-24 Practice 13-1 Investigating Proxy Problems 13-25 Road Map 13-26 Cluster Communication Review 13-27 Unicast Architecture 13-28 Session Management Review 13-29 Session Persistence Review 13-30 In-Memory Replication Review 13-31 Cluster Monitoring WLST Examples 13-32 Session Monitoring WLST Examples 13-33 Session Monitoring Attribute 13-34 Session Instrumentation 13-35 Cluster Debug Flags 13-36 Typical Cluster Heartbeat Trace 13-37 Typical Replication Trace: Primary 13-38 Typical Replication Trace: Secondary 13-39 Common Replication Issues 13-40 HttpSession API Overview 13-41 Serialization Overview 13-42 Serialization Debug Messages 13-43 Section Summary 13-44 xiv
Quiz 13-45 Lesson Summary 13-48 Practice 13-2 Investigating Cluster Replication Problems 13-49 Appendix A: WebLogic SNMP Simple Network Management Protocol (SNMP) A-2 SNMP Architecture A-3 Object Identifier (OID) A-4 Management Information Base (MIB) A-5 WLS MIB and OIDs A-6 Common SNMP Message Types A-7 WLS SNMP Architecture A-8 Creating an SNMP Agent A-10 Configuring an SNMP Agent A-11 SNMP Channels A-12 WLS SNMP Notifications A-13 Creating Trap Monitors A-14 Creating Trap Destinations A-15 SNMP Security A-16 Configuring Agent Security A-17 Configuring SNMP V3 Credentials A-18 Configuring Trap Destination Security A-19 WLS SNMP Utility A-20 xv
xvi