A Hitchhiker s Guide to Troubleshooting IBM Connections

Size: px

Start display at page:

Download "A Hitchhiker s Guide to Troubleshooting IBM Connections"

Loren Charles
6 years ago
Views:

1 Bob Knieff IBM Connections Software Support L2 A Hitchhiker s Guide to Troubleshooting IBM Connections Open Mic Webcast November 10, 2015

2 Agenda A Hitchhiker s Guide to Troubleshooting IBM Connections - Different types of logs, where are they located? - How to read systemout logs - Applying traces - Fiddler and Firebug traces - Troubleshooting - Potpourri - Q & A with Panel - Liz Hecht, Bob Knieff, Jason Oranie, Charlie Price 2

3 Connecting Connections there are many integration points Basic WebSphere Architecture plus: Plugins Search Mobile Portal Cognos FileNet IBM Docs Mail Integration TAM / SiteMinder SPNEGO / SAML 3

Different Types of Logs, Where are They Because Connections connects to everything, and because Connections is a collection of applications that run on the WebSphere platform, there

4 Different Types of Logs, Where are They Because Connections connects to everything, and because Connections is a collection of applications that run on the WebSphere platform, there are a variety of logs in a variety of locations: - WebSphere - Connections Install - Connections Migration - HTTP server - TDI - Congnos - FileNet - wsadmin - Database - Mobile - Fiddler 4

5 Where are the logs? Know your Topology Is the installation: - Small - Medium - Large These are not just generic terms but actually are significant in the installation: Small Deployment One server Medium Deployment Large Deployment Three servers Cluster1: activities, profiles, communities, forums and metrics Cluster2: files, wikis, bookmarks, and blogs Infracluster: common, homepage, news, search, mobile and moderation Each application runs on a server Topology example of a Medium deployment on 2 physical servers: bknieff45md mcmora45md Connections_server1 (node) - Cluster1 - Cluster1 - Cluster2 - Cluster2 - Infracluster - Infracluster DMGR Network share Cognos FileNet Connections_server2 (node) DB2 HTTP TDI 5

6 Where are the logs? - WebSphere WebSphere logs: - AppServer simply distinguishes the WebSphere Application Server from other WebSphere deployments that may be under the WebSphere folder: - The first layer of logs only pertain to the high level install and are seldom used - Most all of Connections log activity pertains to the logs within the profiles directory. 6

7 Where are the logs - WebSphere WebSphere logs: Again, most all of our log activity pertains to the logs within the oddly named profiles directory. ( Profiles reminds me more of templates however it really is an implementation of WebSphere Application Server. There can be multiple profiles on the same AppServer installation.) 7 - For instance, on my lab server: - AppSrv01 Profile for Connections server - AppSrv02 Profile for Cognos server - Dmgr01 Profile for Deployment Manager

8 Where are the logs - WebSphere WebSphere logs: - AppSrv01 The Profile - iccluster_server1 The actual Connections server JVM. Systemerr System level logs that can sometimes have key information, but not usually Systemout WebSphere application server log this is what we use 95% of the time. Traces If captured, would also be located here Multiply these logs by 2, 3, 4 or more for clustered environments! 8

9 Where are the logs Installation and CR Updates Connections 4.5 and 5.0 installation logs: C:\IBM\Connections\logs While Connections runs on WebSphere, there are additional resources that are wrapped around WebSphere. This data is stored in the \ibm\connections folder It includes such things as: - Indexing - Customization folders - Files - attachments - Logs: Install / CR updates Install NOTE: As of Connections 5.0: -- CRs are installed using the IBM Installation Manager. -- ifixes are still installed using the Update Installer. 9

10 Where are the logs - Installation Connections installation logs: Beware, sometimes the developer is just reporting that the installer successfully reported the installation error: 10

$C:\IBM\Connections\version\log 5.0 - ifixes 4.$

11 Where are the logs ifix install logs Connections update logs: C:\IBM\Connections\version\log ifixes ifixes / CR updates 11

$browse these files: C:\IBM\Connections\version 5.0 - Current 4.$

12 Where are the logs Current release level information To find the current release level you can browse these files: C:\IBM\Connections\version Current 4.5 All (need to look for latest) 12

$C:\Users\Administrator\lcWizard\log\dbWizard -$

13 Where are the logs DB2 DB wizard logs: - C:\Users\Administrator\lcWizard\log\dbWizard - /home/db2inst1/lcwizard/log/dbwizard (Linux) 13

14 Where are the logs TDISOL TDISOL logs C:\IBM\TDI\V7.1.1\tdisol\TDI\logs What is TDISOL? - TDISOL is where solutions are housed for TDI - In the Connections vernacular, this is where LDAP to Profiles processes are run 14

15 Where are the logs TDISOL TDISOL Files C:\IBM\TDI\V7.1.1\tdisol\TDI\ Here is where various files are used for TDISOL processing 15

16 Where are the logs HTTP Server HTTP Server logs: C:\IBM\HTTPServer\logs - access - error As all Connections application to applications interactions flow through the HHTP server, there are times when these logs will help to determine routing failures. 16

17 Where are the logs HTTP Server These logs can be useful for tracking calls to various applications. - Remember, every application to application is actually a request to the HTTP server to find out the location of the application. - Connections uses LotusConnections-Config.xml to get the URL for the application, then contacts the HTTP server to get to that location via the WebSphere plugin - The HTTP return code is second from the end on the right 17

$Where are the logs - Cognos Cognos logs: Install C:\install\CognosConfig\$ $log Run time C:\IBM\WAS\AppServer\profiles\AppSrv02\logs\$

18 Where are the logs - Cognos Cognos logs: Install C:\install\CognosConfig\ cognos-setup.log cognos-configure.log Run time C:\IBM\WAS\AppServer\profiles\AppSrv02\logs\ cognos_server\systemout.log C:\IBM\CognosTF\ logs\cogserver.log metricsmodel\trxschelog.log (Good for fast reference of system) 18

19 Where are the logs - FileNet CCM Filenet logs: C:\IBM\WAS\AppServer\profiles\AppSrv01\FileNet\CL_connections_server\ p8_server_error.log p8_server_trace.log pesvr_system.log pesvr_trace.log Connections 4.5 FileNet CPE - Content Platform Engine CCM = Connection Content Manager >>> <<< FNCS = FileNet Collaboration Services

20 Where are the logs - wsadmin The WebSphere administrative (wsadmin) scripting program is a non-graphical command interpreter environment enabling you to run administrative operations in a scripting language. Connections uses this tool to perform various maintenance and update functions. Wsadmin logs: (typically not logged, the information just displays in the command window) wsadmin>execfile("communitiesadmin.py") WASX7015E: Exception running command: "execfile("communitiesadmin.py")"; exception information: com.ibm.bsf.bsfexception: exception from Jython: Traceback (innermost last): File "<input>", line 1, in? IOError: File not found - communitiesadmin.py (No such file or directory) wsadmin> In this case, customer was running the wsadmin command from the Appserver rather than from the Dmgr and therefore the file was not present. Wsadmin tracing: Tracing operations using the wsadmin scripting tool (this will help to capture wsadmin errors, not functional errors) Right before you enter the communities.admin command, run the following (bolded) command to add tracing: wsadmin>admincontrol.trace('com.ibm.*=all') wsadmin>execfile("communitiesadmin.py") wsadmin>communitiesservice.movecommunitytosubcommunity... This will create "wsadmin.traceout" log under the \IBM\Websphere\Appserver\profiles\AppSrv01\logs 20

How to read Systemout logs - Structure WebSphere Server Startup Displays: WebSphere release WebSphere server name PID Process ID Operating System File location of the server Tracing applied Check to

21 How to read Systemout logs - Structure WebSphere Server Startup Displays: WebSphere release WebSphere server name PID Process ID Operating System File location of the server Tracing applied Check to see if this is the startup of the server that you want! Does not display: Release of Connections This is because Connections is a collection of applications that run on WebSphere server. We can however find this from the systemout.log that includes the application startups. Stay tuned. 21

22 Systemout.log Structure Sample SystemOut.log [14/08/11 07:06:54:204 BST] a ManagerAdmin I TRAS0017I: The startup trace state is *=info. [14/08/11 07:06:54:683 BST] a ManagerAdmin I TRAS0111I: The message IDs that are in use are deprecated [14/08/11 07:06:55:606 BST] a FileRepositor A ADMR0010I: Document cells/server1_cell/nodes/server1_manag [14/08/11 07:06:57:823 BST] a ThreadPoolMgr W WSVR0626W: The ThreadPool setting on the ObjectRequestBro [21/05/10 10:02:56:240 BST] TCPPort E TCPC0003E: TCP Channel TCP_5 initialization failed. The Time Stamp: The first part of the log message in sample code is [15/08/11 03:49:59:585 BST]. It is the time stamp when the message was written. The time stamp is formatted using the locale of the process and it is 24 hour time stamp with milli-second precision Thread ID: The next part in the log message is / c, which represents the thread id. The thread ID is an eight-character hexadecimal value that is generated from the hash code of the thread that issued the message Short name: The short name is the abbreviated name of the component that issued the message. This name is typically the class name of a WAS component and would be some other identifier for the application. here ThreadPoolMgr, FileRepositor, TCPPort etc Event Type: The event type is a one character field that indicates the type of the message. The possible values are * F- Fatal message * E- Error message * W- Warning message * A- Audit message * I- Informational message * C- Configuration message * D- detail message * O- Messages that are written directly to System.out by an application or server component * R- Messages that are written directly to System.err by the user application or internal component. * Z- Place holder to indicate type was not recognized # Message identifier (TCPC0003E / WSVR0626W / ADMR0010I): The message identifier is a string that is nine characters in length and is in the form CCCC1234X. The first four characters indicate the WAS component that issues the message. The next four characters indicate the specific message that component is issuing. The last character indicates the severity of the message. Its value is either I- informational, W- Warning or E error. # Message: The message is the data that is logged to the SystemOut.log by the component From Oracle: How to Analyze JVM logs See also Reference: Log file format in the following: 22

23 How to Read and Analyse the Systemout Log Tools: Notepad++ notepad-plus-plus.org/ You may have your own favorite - You may need to access multiple logs in a clustered environment - Some of our experts say start at the top some say go right to the bottom - Make sure you are looking at what you think you are looking at! (Some logs have months of data, others have minutes) - Systemout is not always at the beginning of a reboot, check to see "application started in systemout. From Notepad++ you can find all in current document and see all applications started in a systemout.log: 23

How to Read and Analyse the Systemout Log Searching: - Get counts of errors - See historical trends of errors - e error Best trouble-shooting tip I learned

24 How to Read and Analyse the Systemout Log Searching: - Get counts of errors - See historical trends of errors - e error Best trouble-shooting tip I learned last year, notice the spaces before and after the e - w warning Will still need to review the log but this can help rise to 10,000 feet and see the context. 24

25 How to Read and Analyse the Systemout Log Search for e 1. Database schema not updated to match release of Connections: 2. DB2 not started: 25

How To Read the Logs Stack Trace There is little information about reading stack traces from a support perspective and unfortunately healthy servers may also have stack traces, so it can be hard to

26 How To Read the Logs Stack Trace There is little information about reading stack traces from a support perspective and unfortunately healthy servers may also have stack traces, so it can be hard to interpret which ones are serious and which are not. Basically: The first line will tell you key information about the error The lines in-between are all pointing back to what called them The last time is the ultimate source Look for the caused by line if available Unfortunately, many healthy servers also have stack traces, so it can be hard to interpret which ones are serious and which are not 26Here is a hipster s explanation of how to read a stack trace:

27 WebSphere Tracing Applying tracing: Log on to the WebSphere Application Server (WAS) integrated solutions console using an administrator ID Go to -- Troubleshooting -> Logs and Trace -> Logging and Tracing -> CL_connections_server -> Diagnostic Trace Server -> Change Log Detail Levels -> Add the following string and click Apply -> OK *=info:com.ibm.lconn.moderation.*=all: (sample trace) - This is dynamic - It will not let you save a bad trace - Turn off all but one server or collect all logs 27

28 Fiddler Traces Instructions for capturing a Fiddler trace: Install Fiddler Download and install Fiddler (3rd party tool - at ) on the workstation If you are using SSL/https, please be sure that Fiddler is set to capture HTTPS (SSL). This is done by going to Tools - Fiddler Options - HTTPS tab - then select both "Capture HTTPS CONNECTs" and "Decrypt HTTPS traffic" (You will get a security/certificate warning - you will have to decide if you select Yes or No, however if you select No, we will not get a useful trace.) Using Fiddler Clearing the trace is easy just use Ctrl + x Useful for removing extraneous information prior to recreating a problem with the smallest trace possible. Please shut down other apps/browsers that add extraneous data to the trace Note that Fiddler time is in your local browser s time keep in mind when trying to find time in the systemout.logs as they may be using a different time zone. Use File Save All Sessions to save the trace. By default the traces are stored in: My Documents\Fiddler2 Please note the approximate line number of when the issue at hand occurs. 28

29 Fiddler Traces 29

30 Fiddler Traces 30

31 Fiddler Traces 31

32 Firebug Traces 32

33 Troubleshooting Example WebSphere Trace End user sees: Administrator sees this in SystemOut.log: 33

34 Troubleshooting Example WebSphere Trace Administrator provides this Fiddler trace 34

35 Troubleshooting Example WebSphere Trace Is this a Forum or Moderation issue???? I add trace strings for both Forum and Moderation: *=info: com.ibm.lconn.forum.*=all: com.ibm.forum.*=all: com.ibm.lconn.moderation.*=all 35

36 Troubleshooting Example WebSphere Trace 36

37 Troubleshooting Example - Fiddler Trace 37

38 Troubleshooting Example - Fiddler Trace As per the lcc.xml file, opensocial is on port 9080 rather than 9085: 38

39 Potpurri How to determine WebSphere Application Server WAS server version level: See 1. Go to C:\IBM\WAS\AppServer\bin 2. type in versioninfo -maintenancepackages How to determine IBM Connections server version level: - Go to \IBM\Connections\version\ directory - How To Determine What Fix-pack, CR and/or ifix Is Installed On IBM Connections The above link also lists the dates of various Connections release dates 39

is 1 meg): Java virtual machine (JVM) log settings http://www- 01.ibm.

40 Potpurri How to increase the log size and number in WebSphere Application Server Customer needs to increase logging to retain additional information for future touble-shooting (default setting is 1 meg): Java virtual machine (JVM) log settings 01.ibm.com/support/knowledgecenter/SSAW57_8.5.5/com.ibm. websphere.nd.doc/ae/utrb_jvmlogs.html?cp=ssaw57_8.5.5%2 F &lang=en 40

41 Potpurri SIB messaging bus queue depth: Use the following path in ISC to see if there is a backlog in the messaging bus: (keep this handy!) Service Integration > Buses > ConnectionsBus > Destinations > connections.events > Publication points > connections.events@cl.000-connectionsbus [Runtime] > Subscriptions 41

42 Potpurri How to determine what server ports go to which application server Application servers > CL_connections_server > Ports ISC > System Administration -> Deployment Manager -> Ports - ISC port = WC_adminhost_secure - SOAP port = SOAP_CONNECTOR_ADDRESS 42

43 Potpurri How to determine what server an application runs on: Go to Enterprise Applications > <app> > Target specific application status 43

44 Potpurri Clearing Cache - Shut down the Connections server, the DMGR, and the nodeagent. - Clear the cache in these locations: C:\IBM\WebSphere\AppServer\profiles\AppSrv01\temp C:\IBM\WebSphere\AppServer\profiles\AppSrv01\wstemp C:\IBM\WebSphere\AppServer\profiles\Dmgr01\temp C:\IBM\WebSphere\AppServer\profiles\Dmgr01\wstemp - Start the DMGR the node agent and Connections - Clear the cache on the browser 44

45 Potpurri - What happens if the HTTP Server is not available? If the HTTP Server is not available, or the address is wrong, the following will display: The browser is stating that the address is not available. This is a DNS, HTTP server, network issue and Connections is not involved with this error. 45

46 Potpurri - What happens if the Connections server is not available? If the Connections server is down or not accessible, this error from the HTTP Server will display: This also happens if one of the settings in the plugin is not correct Or if the server does not trust the ssl certificates 46

47 A Hitchhiker s Guide to Troubleshooting IBM Connections Q & A with L2 Support Panel: Liz Hecht Bob Knieff Jason Oranie Charlie Price 47

48 Questions 48

Architecture & Deployment

Architecture & Deployment IBM Connections 5.0 Workshop Author: Paul Godby IBM Ecosystem Development Duration: 30 minutes 2014 IBM Corporation Agenda IBM Connections lications Prerequisite Software Deployment