Oracle Fusion Middleware: Performance & Diagnostics solutions on WebLogic

Transcription

1 2018 Michel Schildmeijer Oracle Fusion Middleware: Performance & Diagnostics solutions on WebLogic nloug Tech Experience The Cloud is next Amersfoort, 7 June 2018 By Michel Schildmeijer,

2 Me.. Michel Schildmeijer, almost 52 years Solutions Architect Author of 2 books Specialties: WebLogic 6 to 12, OSB, Tuxedo, SOA, Architect and design

3 Contents Troubleshoot Patterns & Methods A slight view under the hood Hints, tips & tricks

4 Troubleshoot patterns & methods

5 Troubleshoot overview System errors, messages lost, bad performance Where do I begin?? What method should I follow?

6 Troubleshoot Methods Using these troubleshooting methods helps you to: A structured way of detecting issues Get behind the cause of the problems that occur Methods to follow: Use of Operating system tools Use of JDK tools Use of WebLogic Server tools Use of FMW Diagnostic tools Use of Enterprise Manager ( Cloud Control )(?) Use of OMC : Application Performance Management, Log Analytics(?)

7 Troubleshoot Patterns Patterns to identify a Hang Hangs related to the WebLogic Server, Java or some subset of resources Generic Server Hang Pattern JDBC Hang Pattern EJB & RMI Hang Pattern JMS Hang Pattern

8 Server Hang Server doesn t respond to new requests Requests time out. Requests take longer and longer to process (may be on the way to a hang). A server crash is not usually a symptom of a hung server but final state

9 JDBC Hang DriverManager.getConnection() in code. SQL takes unexpectedly long time JDBC connection pool hangs Slow network causes to slow down or hang. A deadlock all execute threads to hang & wait RefreshMinutes or TestFrequencySeconds in conn pool causes hang Long response JDBC conn pool shrinking and No Sufficient DB resources

10 EJB RMI Hang Issues over RMI with remote JVM Serialization Cluster issues JNDI lookup fails at weblogic.jndi.internal.wlcontextimpl.lookup(wlcontextimpl.java: 341)

11 Increasing pending messages Sent by a producer Received by a consumer Lagging async client JMS Hang Durable subscribers (to avoid missing placed messages in a topic)

12 WebLogic Troubleshoot overview O/S tools: sar: system activity mpstat: per-processor statistics vmstat: virtual memory statistics netstat: network statistics iostat: Input/Output statistics top

13 WebLogic Troubleshoot overview Start with a simple check it seems so obvious but: Is there enough storage Is there enough RAM available Does the system do a lot of paging

14 JVM Troubleshoot

15 The JavaVirtualMachine Java Runtime Java Source code.java JDK Compiler RuntimeCompiler JVM Java bytecode

16 JVM from code to machine A Java application runs a piece of code Code enters the JVM It enters several stages: Operations Data structure Transformation ( to bytecode) Java code Operat ons Data Structu re JVM Transformati on

17 Understanding Java threads JVM process consists Java and JVM internal threads JVM threads: Garbage Collect, Optimizer and Finalizer Each thread has a stack to store runtime data Overflow will happen if this stack will store more than its size Each O/S has its default stacksize Can be extended with -Xss

18 JVM from code to machine Runtime behaviour: o JVM runs as an OS process o Execute work through threading model (concurrency) o Garbage Collector Each Weblogic Server (instance) runs inside a separate JVM (instance).

19 JVM from code to machine Crash can occur during Compile process Cause: invalid native code (C, C++, Libs) Internal Code Garbage Collect ( Segmentation Fault) Cause: GC Fault leading into bad memory access Long running GC can result in paging or CPU overload Application code Like 3 rd Party drivers (JDBC, MQ libraries)

20 What to look for in java thread dumps: Low Memory Detector (Daemon thread to detect and report low memory conditions) CompilerThreadn (Daemon thread/s to compile & optimize byte code) Signal Dispatcher JVM troubleshoot (Daemon thread to respond OS signals)

21 JVM troubleshoot Finalizer Daemon thread to handle objects in finalizer queue Reference Handler Daemon thread to handle objects in reference queue VM Thread Main thread from JVM GC task thread#n Thread/s to perform garbage collection

22 Liveset and fragmentation JVM troubleshoot Heap gets fragmented when small & large objects are mixed store with various lifetimes. May cause longer GC Garbage collect time and behavior

23 Java HotSpot Java Mission Control JConsole JVisualVM JVM tools

24 Java Mission Control provides: Console and command line (jcmd) Live JVM statistics Graphs and metrics about local or remote systems Basic JMX interaction with MBeans Runtime data An in-depth look at live metrics, like memory, CPU and stack trace on specific threads Profile methods or exceptions & execute commands Remote or local Java Mission Control -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=3614 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false

25 Java Mission Control -XX:+UnlockCommercialFeatures jcmd [PID] VM.unlock_commercial_features

26 Diagnostic Volume must be set higher than the default (Low) Java Mission Control

27 Java Troubleshoot Some well known errors Class not found missing or incorrect Classpath set during startup (server start params or setdomainenv) or adding domain libs NullPointers A value is expected but it s not there (null): Poorly written Bad error handling Missing config Stack Overflow Recursive procedures, a method with no exit Too Many Open Files Ulimit settings

28 WebLogic Server Troubleshoot

29 Server Hangs Server applies to: Managed server or AdminServer Partial or total (OutOfMemory) Lack of resources/deadlocks

30 Symptoms The server does not respond to new requests. Requests time out. Requests take longer and longer to process (may be on the way to a hang). A server crash is not usually a symptom of a hung server but may follow.

31 Deadlock Lock Wait Resource A Thread 1 Thread 2 Deadlock Wait Resource B Lock

32 Possible Causes All threads waiting for RJVM, RMI responses. Garbage Collection taking too much time. Long Running JDBC calls or, connection leaks, deadlocks JVM hang during code optimization JSP compilation under heavy load.

33 Tools/Methods WebLogic Server generic: WebLogic Diagnostics Framework Logs WLST JVM specific : Java Mission Control, JStat, JConsole

34 Tools/Methods Force a JVM dump by: kill -3 <pid> stack in STDOUT Usually in <domain>/servers/logs/<serverinstance>/<serverinstance.out> See more in the next slides about WebLogic Threads

35 WebLogic Threads

36 WebLogic taking threaddumps A threaddump is a snapshot of the state of a WebLogic Server Instance Different ways to take a threaddump: kill -3 <pid> the O/S way WebLogic Console WLST(WebLogic Scripting Tool) cd ('Servers') ls() cd ('AdminServer') ls() threaddump()

37 WebLogic Thread Architecture Socket Threads Default Queue Execute Threads Application

38 WebLogic Threads Active threads: Are either processing or waiting for requests Can be tagged with additional attributes such as Idle or Hogging Standby threads are not used to process requests unless throughput is needed. Workmanagers can control threads of specific parts

39 WebLogic Stuck Threads Running for a long time Deadlocked Infinite loop Overloaded server WebLogic: Periodically checked how long threads are running Logs a warning if a thread becomes stuck Creates additional threads to handle new requests Sets its state to Failed after specified nr of threads become stuck Individual work managers can detect stuck threads & reject new requests.

40 Stuck Thread possible actions

41 The sense of taking threaddumps Hanging threads will be diagnosed as STUCK Default time to get STUCK is 600 secs Thread has not completed it s work

42 The sense of taking threaddumps Thread Dumps can give you: View of the state of application server threads at that instant in time Information about potential issues such as Hot spots within code which seem to be called often Portions of code where the application seems to be hung Locking and thread synchronization issues in an application

43 The sense of taking threaddumps Take several dumps to investigate various states Threads high in stack can be treated as idle, mostly waiting for some work to do Blocked or Wait on socketaccept() listener thread waiting for socket connections

44 The sense of taking threaddumps WebLogic specific: Ignore threads in ExecuteThread.waitForRequest() Muxer and processsocket threads are for housekeeping and listening A Muxer which is blocked should not be waiting for a lock by a non Muxer

45 WebLogic taking threaddumps In the AdminConsole

46 Understanding Locks Are meant to synchronize between threads Lock contention is a thread waiting for another to be released Fat locks: Have a history of contention (several threads try to take the lock simultaneously), or waiting for notification Thin locks: Without any contention. Recursive locks: A lock on a thread several times without having been released.

47 WebLogic JDBC Troubleshoot

48 Wrong or missing drivers in Classpath Connection errors by: Usernames/passwords JDBC additonal errors Connection leaks by bad apps Set Inactive timeout to a value (default is 0) Insufficient connection parameters Max Capacity, Reserve Time Out(-1) Invalid JDBC URLS weblogic.common.resourceexception: Could not create pool connection. The DBMS driver exception wa

49 WebLogic JDBC mechanism Application JNDI Lookup 1. Connection 2. Statement 3. SQL 4. Results 5. Close JDBC Driver Pool Conn1 Conn2 Database

50 Using dumppool and WLDF capture JDBC Profiling

51 Debug Flags DebugJDBCConn - Trace all connections. DebugJDBCSQL - Trace all JDBC API calls DebugJDBCInternal - low-level internal data source activities. DebugJTAJDBC - Trace transaction management

52 Connection Leaks Bad implemented applications can starve connections and data Cache or hog connections instead of releasing them. Do not explicitly release connections using close(). Automatically reclaim with Inactive Connection Timeout

53 WebLogic JMS Troubleshoot

54 Common JMS issues Incorrect targeting Connection factories to subdeployments instead of to entire servers Targeting distributed destinations to entire servers instead to subdeployments Using multiple subdeployments within the same module Configure JMS migration without a cluster Convert a single-server domain to a cluster

55 Common JMS issues - OOM OOM in case of large amount of pending messages JMS Quota can prevent that Message paging messages in virtual memory(file or db) Flow control

56 Common JMS issues Lost messages Expired messages Set a TimeToLive & Exp policy JMS system failure and nonpersistent messages Set a delivery and time to delivery mode Insufficient retry settings Message delay settings Nondurable topic subscriptions

57 MDBs Applications used to automatically consume JMS Often used in SOA Suite applications ( SOA Suite, OSB ) Manages itself ( pooling ) A few debug flags or use WLDF to diagnose

58 WebLogic Log and debug methods

59 WebLogic Log and Debug options Logging services are important for diagnostics WebLogic produces output by using a Message Catalog Using the standard java.util.logging Can be extended using log4j modules All kinds of events, like startup/shutdown, appl.deployments, subsystem failures are written to logs

60 WebLogic Log framework

61 WebLogic Log and Debug options Several severity levels like: TRACE, DEBUG,INFO.. Important to set the proper level For FMW specific components, ODL-HANDLER can be used

62 WebLogic Log and Debug options Subsystem logs can be from: HTTP Transaction Manager JDBC JMS server WebLogic Auditing provider FMW Logs

63 WebLogic debug options Debug Setting debug options in the Admin Console Or use D options in startup script -Dweblogic.debug.DebugJDBCSQL=true Prints information about all JDBC methods

64 WebLogic debug options Or use WLST serverconfig() or domainconfig() debug = getmbean('/servers/mserver/serverdebug/mserver') debug.setdebugjdbcinternal(true)

65 WebLogic Log4j OpenSource Log engine from Apache Enable the LogMBean.isLog4jLoggingEnabled attribute In WLST: cmo.setlog4jloggingenabled(true) Add Log4j classes, WL_HOME/server/lib/wllog4j.jar & log4j.jar to server CLASSPATH

66 WebLogic Log4j Enable log4j in AdminConsole

67 Oracle diagnostic Logging (ODL) As an extension on the default logging handlers Comes with FMW installation, not WebLogic default Logging view and config in Fusion Middleware Control Can be set to several trace levels Output in <servername>-diagnostic.log

68 Oracle diagnostic Logging Runtime

69 Oracle diagnostic Logging Edit or create your own handler

70 Oracle Diagnostic Logging Edit or create your own handler

71 WebLogic Diagnostics Framework

72 WebLogic Diagnostic Framework Included with WebLogic Watch & Notification Data Collectors Data Providers Archive Accessor Monitoring Dashboard Instrumentation Image Capture Define Watches to monitor runtime conditions and notify in case of failure. Collect configuration and runtime data for analysis. Capture images to preserve system state in failure situation.

73 WLDF Framework for monitoring and diagnostics Consolidated access for MBean data, instrumentation, logging, debugging Source for monitoring data Server & application instrumentation notifications based on the output : log, , SNMP, JMX, JMS Request dyeing and tracing Data archiving for later access WLDF Dashboard

74 WLDF Key Features Gathering input from JMX MBeans WebLogic subsystems and application code can be instrumented to provide events and watches Notifications can be defined for events and watches Console extension for monitoring of current and historical data Data archiving

75 WLDF Can be configured on filesystem or Database Configure with Data retirement

76 WLDF Create Diagnostic Module Collect Metrics. A collection includes: The MBean type to query The specific MBean instance names to query (all instances, by default) The MBean attributes to collect (all attributes, by default) How often to gather data Create Watches & Notifications

77 Diagnostic Frameworks & Diagnostics Tools in FMW WLDF WebLogic Diagnostic Framework DFW Diagnostic Framework o Dynamic Monitoring Service (DMS) o Oracle Diagnostic Logging (ODL) o Incident Packaging System (IPS or ADRCI) Additional Tools o Selective Tracing

78 WebLogic Diagnostic Framework Oracle Fusion

79 Diagnostic Tools for SOA and FMW Predefined Watches and Diagnostic Dumps Detect, diagnose and resolve problems Use WLS and SOA MBeans and DMS Metrics. Detect critical failures and collect diagnostic dumps containing relevant diagnostic information like logs, metrics, server images, Create incidents in the ADR (Automatic Diagnostic Repository)

80 Integration of WLDF, DFW and Diagnostic Tools for SOA and FMW Monitor System Predefined Watches Predefined Diagnostic Dumps Critical Failure Collect Diagnostic Dumps Create Incident Collect Collect Data using EM or

81 Dynamic Monitoring Service DMS for runtime metrics applications and components Can be monitored through WLDF Watches to trigger data collections through DFW.

82 Dynamic Monitoring Service (DMS) Direct accessible through Give pretty much raw unstructured results Better option is to use FMW Console

83 DMS in FMW Console

84 Diagnostic Framework (DFW) collect a pre-configured set of information at exactly the right time through a triggering mechanism. Triggered from Oracle Diagnostic Logging events or WebLogic Diagnostic Framework Watches. Information can also be collected manually using WLST

85 Oracle Diagnostic Logging (ODL) ODL is the primary logging for FMW The ODL log entries are in XML format

86 Selective Tracing (ST) Selective Tracing is a facility available through Enterprise Manager to limit the scope of trace logging. This allows for debugging of a production system without overloading the system with logging activity.

87 Selective Tracing (ST)

88 Selective Tracing (ST)

89 Performance

90 Performance Terms and Definitions Performance : How system s & applications response time and throughput are affected by adding load. Capacity : Maximum threshold a system is under a given set of conditions. Scalability : how does a system responds to increasing load by adding additional resources. Key indicators are : Response time Throughput

91 Performance objectives Locate & minimize bottlenecks Gather info about Application(s) Concurrent number of users and interfaces. Number and size of app requests. Amount of data and its consistency. Determining your target CPU utilization Tune your Database(s) (customers or repos) Tune WebLogic Server Performance Parameters Tune Your JVM Tune the Operating System Tuning WebLogic Persistent Store

92 Tune WebLogic Server Investigate and analyze! Always consider before implementing : What am I changing Has the bottleneck gone away? Is there a new bottleneck?

93 Native I/O and muxers Enable by default Will use a O/S threadpool instead from borrowing from WebLogic default Exec.Q Native muxers use platform-specific native binaries to read data from sockets Non native or Java sockets : Uses pure Java to read data from sockets. Blocks until there is data to be read from a socket(can cause issues on server level)

94 WebLogic Thread Pool Represents nr of simultaneous operations performed by apps that use the execute Q Application Execute Request WebLogic Socket Muxer WebLogic Kernel assigns Execute Req to ExecuteThread 1 WebLogic Kernel handles self tuning ThreadPool, Workmanagers, allocations ExecuteThread 1 handles the request (for example executes a JDBC request) The app now handles the thread itself and gives back result

95 WebLogic Thread Count Different execute Q s in 11g for applications and administration work weblogic.kernel.default, RMI, HTTP 1 default Queue In 12c Single threadpool Self tuning Work managers Prioritize by rules Different constraints regarding min max response capacity -Dweblogic.threadpool.MinPoolSize=5 -Dweblogic.threadpool.MaxPoolSize=5

96 Applications and Execute Queues Default All apps use all execute queues Configuring multiple execute queues can provide additional control for applications. By using multiple execute queues, selected applications have access to a fixed number of execute threads, regardless of the load on WebLogic

97 Threads as SocketReaders ThreadPoolPercentSocketReaders Default set to 33 ( range is 1-99) Allocating threads as readers can increase accepting requests

98 Piece of memory for read and write data to sockets (network layer) A pool of chunks is maintained For applications with large amounts of data Parameters are: weblogic.chunksize Size of a chunk (in bytes). Set to network's maximum transfer unit (MTU), after subtract it from any Ethernet or TCP header sizes weblogic.utils.io.chunkpoolsize Sets the maximum size of the chunk pool. Monitor the CPU profile or use a memory/ heap profiler for call stacks invoking the constructor weblogic.utils.io.chunk. weblogic.partitionsize Sets the number of pool partitions used (default is 4). Partitioning the thread pool spreads the potential for contention over more than one partition To be set in the setdomainenv.sh WebLogic ChunkSize

99 WebLogic Backlog Specifies how many TCP connections can be buffered. These are on the TCP stack but not yet accepted by an app. Default is 50 Maximum depends on O/S

100 When to tune? WebLogic Backlog When getting connection refused in high throughput apps (SOA composites) If tuning, increase by 25% of it s default until errors dissapear

101 JDBC Connectionpool tuning Prepared Statement Cache Keeps compiled SQL statements in memory Set to LRU A data source with 10 connections deployed on 2 servers, with the Statement Cache to 10 (default), 200 cursors can be used on the database for these cached statements MaxCapacity Max of physical connections Db drivers can limit connections Number should equal number of client sessions LLR instead of XA Decrease 2PC overhead Disable wrapping datatypes (debug, track, transaction control of blb, clob)

102 In some cases it takes ages after startup to continue Generate a stackup Tuning Startup

103 Tuning Startup This happens when using default seed generator: Use of dev/urandom during startup in stead of random Specifiy in startup properties - Djava.security.egd=file:///dev/urandom Or in $JAVA_HOME/jre/lib/security/java.security securerandom.source=file:/dev/./urandom

104 Tuning Startup SOA Keep MDS and Dehydration store clean MDS delete older label deployments Dehydration store purge strategy

105 Configure Transaction settings: BPEL SyncMaxWaitTime BPEL EJBS JTA DB Tuning other SOA

106 Tuning AdminServer EM SOA : Discovery cache age to discovery wait time to Set the cache results to true Dumpinterval to with a maximum of 75 MB Enable Large Repository Increase the collector configuration settings for DMS: Use an interval of 300 secs with a remove cycle of 3 And use an interval of 120 secs with a remove cycle of 2, set this to default Restrict the amount of displayed instance information to 12 hours

107 Tuning the JVM Most aspects of JVM tuning relate to: Sufficient memory heap An efficient garbage collection scheme Classloading Compile/interpret

108 Most important to focus on: Garbage collection scheme Compaction/Defrag Free up memory Memory heap Tuning the JVM Is a runtime repository for live objects, dead objects and free memory. No more than 75% of total RAM The JVM heap sets frequency & duration GC Large heap size : full garbage collection slower and less frequent Small heap size full garbage collection is faster & higher frequency

109 Goal is: Tuning the JVM Minimize Garbage Collect Set heapsize that GC does no happen all the time

110 Investigate with verbosegc option How often does GC run How long does it take? Tuning the JVM Full garbage collection should not take longer than 3 to 5 seconds. Lower heap if major GC time is greater.

111 Set Xms and Xmx parameters to specify initial and Max HeapSize Xmx is the Java Object Heap, not the System Heap > which is larger Values equal will cause no overhead of heap management PermSize and MaxPermSize Be aware: For stored classes, methods Tuning the JVM Java methods, thread stacks, native handles are in memory separate from heap

112 The Cloud is next

113

114 Any Questions? Let s keep in touch!! The ACE Community mschildmeijer@qualogy.com

115 Any Questions? Let s keep nl.linkedin.com/in/mschldmr