White Paper: ELK stack configuration for OpenEdge BPM

Transcription

1 White Paper: ELK stack configuration for OpenEdge BPM

2

3 Copyright 2017 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. These materials and all Progress software products are copyrighted and all rights are reserved by Progress Software Corporation. The information in these materials is subject to change without notice, and Progress Software Corporation assumes no responsibility for any errors that may appear therein. The references in these materials to specific platforms supported are subject to change. Business Making Progress, Corticon, DataDirect (and design), DataDirect Cloud, DataDirect Connect, DataDirect Connect64, DataDirect XML Converters, DataDirect XQuery, Deliver More Than Expected, Icenium, Kendo UI, Making Software Work Together, NativeScript, OpenEdge, Powered by Progress, Progress, Progress Software Developers Network, Rollbase, SequeLink, Sitefinity (and Design), SpeedScript, Stylus Studio, TeamPulse, Telerik, Telerik (and Design), Test Studio, and WebSpeed are registered trademarks of Progress Software Corporation or one of its affiliates or subsidiaries in the U.S. and/or other countries. AccelEvent, Analytics360, AppsAlive, AppServer, Arcade, BravePoint, BusinessEdge, DataDirect Spy, DataDirect SupportLink, DevCraft, DigitalFactory, Fiddler, Future Proof, High Performance Integration, JustCode, JustDecompile, JustMock, JustTrace, OpenAccess, ProDataSet, Progress Arcade, Progress Profiles, Progress Results, Progress RFID, Progress Software, ProVision, PSE Pro, SectorAlliance, Sitefinity, SmartBrowser, SmartComponent, SmartDataBrowser, SmartDataObjects, SmartDataView, SmartDialog, SmartFolder, SmartFrame, SmartObjects, SmartPanel, SmartQuery, SmartViewer, SmartWindow, WebClient, and Who Makes Progress are trademarks or service marks of Progress Software Corporation and/or its subsidiaries or affiliates in the U.S. and other countries. Java is a registered trademark of Oracle and/or its affiliates. Any other marks contained herein may be trademarks of their respective owners. Please refer to the Release Notes applicable to the particular Progress product release for any third-party acknowledgements required to be provided in the documentation associated with the Progress product. The Release Notes can be found in the OpenEdge installation directory and online at: For the latest documentation updates see OpenEdge Product Documentation on Progress Communities: ( openedge-product-documentation-overview.aspx). June 2017 Last updated with new content: Release Updated: 2017/06/19 3

4 Copyright 4

5 Contents Table of Contents Chapter 1: Introduction...7 Supported versions...8 Chapter 2: Understanding OpenEdge BPM logging...9 Component specific logs...9 Container specific logs...10 Script debug logs...11 Adapter logs...11 External logs...11 Chapter 3: Configuring ELK stack for OpenEdge BPM...13 Prerequisites...14 FORMAT 1 (Component and Container specific logs)...14 FORMAT 2 (Adapter logs)...14 FORMAT 3 (AppServer logs)...14 Logstash configuration...15 Logstash input...15 Logstash filters...15 Logstash output...19 Exploring Elasticsearch with Kibana...19 Chapter 4: Use cases...25 Use case 1 (Tracking user login and logout events)...25 Use case 2 (Troubleshooting OpenEdge Adapter)

6 Contents 6

7 1 Introduction This document describes how to configure the ELK (Elasticsearch, Logstash, and Kibana) stack for optimizing log management in OpenEdge BPM. To gain better visibility and control of various logs scattered across different locations in different machines, ELK stack provides a scalable and real-time solutions to easily troubleshoot different failures. The ELK stack consists of three open source products: Elasticsearch A NoSQL database based on the Lucene search engine. Logstash A log pipeline tool that accepts inputs from various sources, executes different transformations, and exports the data to various targets (along with Elasticsearch). Kibana A visualization layer that works on top of Elasticsearch. 7

8 Chapter 1: Introduction The following figure shows the ELK stack flow for monitoring OpenEdge BPM and AppServer logs: Note: This document is not intended to provide complete information about ELK stack components. For more information about these components, refer to For details, see the following topics: Supported versions Supported versions The steps to configure the ELK stack in this document are specific to the following product versions and may not be applicable for other versions: Product OpenEdge Elasticsearch Logstash Kibana Supported versions (Latest version as of 5th June, 2017) You can download the latest version of ELK from 8

9 2 Understanding OpenEdge BPM logging OpenEdge BPM executes various business processes which involve different components of BPM and occasionally interacts with external systems, thereby generating many logs which help you to monitor and troubleshoot the runtime behavior of these processes. OpenEdge BPM logs are categorized based on their usage. For details, see the following topics: Component specific logs Container specific logs Script debug logs Adapter logs External logs Component specific logs Component specific logs help you to troubleshoot any issues specific to the BP Server and the deployed applications. 9

10 Chapter 2: Understanding OpenEdge BPM logging The following table describes the logs specific to each OpenEdge BPM Server component: Log file oebps.log bpserver.log dbconfigtool.log bpmportal.log bpmprocessstore.log bpmevents.log dms.log oebpswebsecurity.log mconsole.log bpmwebflow.log webservice.log distcache.log Description The root logger category of OpenEdge BPM components which contains logs from most of the components and other third-party libraries such as Spring, Axis, Apache tools, etc. Logs generated by BPM Process runtime component. Logs generated by Database Configuration Tool. Logs generated by BPM Web Portal. Logs generated by BPM Process Store component. Any failures of BPM Process Store are logged here. This primarily effects the historical process instance data from the Management tab of BPM Portal. Logs generated by BPM Events (BizPulse) engine. Any failures of BPM Events (BizPulse) rules are logged here. Logs generated by Document Management System. Logs generated by Web Security component of BPM Web Portal. Any exceptions from XSS/XSRF filters are logged here. Logs generated by Management Console table populator. Logs generated by BPM WebFlow (BizSolo) applications. Any issues with custom JSP tags with "bizsolo" prefix are logged here. Logs generated by BP Server Web Services deployed on Portal. Logs generated by distributed cache framework. Container specific logs Container specific logs allow you to inspect and troubleshoot sever container specific issues such as database transaction failures, JMS failures, servers failed to start, etc. These logs are available in the following locations: $DLC\oebpm\jboss\server\ejbServer\log $DLC\oebpm\jboss\server\portalServer\log 10

11 Script debug logs Script debug logs Script debug logs are generated when JSTools.writeLog() writes debug messages from pre and post scripts, and custom adapters to bpserver.log file. Adapter logs Adapter logs are generated during the execution of a Managed Adapter. They are appended to their respective log files, and are created with the adapter workstep name $OEBPS_HOME\ebmsapps\<PROCESS_NAME>\logs\<ADAPTER_WS_NAME>.log. External logs External logs are generated by third-party systems such as External WebServices, Corticon, OpenEdge AppServer, PASOE, etc. These must be inspected in case of any failure in their respective Managed Adapters configured in BPM. 11

12 Chapter 2: Understanding OpenEdge BPM logging 12

13 Configuring ELK stack for OpenEdge BPM 3 This section describes how to set up ELK stack to monitor various logs created by OpenEdge BPM. For details, see the following topics: Prerequisites Logstash configuration Exploring Elasticsearch with Kibana 13

14 Chapter 3: Configuring ELK stack for OpenEdge BPM Prerequisites Apply the configuration changes described in the following sections to OpenEdge BPM logs and other logs generated by external systems. FORMAT 1 (Component and Container specific logs) In the oebpslog.conf and oebpswebsecuritylog.conf files located in the $OEBPS_HOME\conf directory, consider the following: 1. Set the ConversionPattern attribute for every logger category to %d{iso8601 %p %c - %m%n. This gives a standard date format across all Component specific logs. For example: log4j.appender.<logger_category>.layout.conversionpattern=%d{iso8601 %p %c - %m%n. Where: %d{iso8601 yyyy-mm-dd<date format> %p Priority %c Component %m%n Message appended with platform specific line separator 2. Set the log level as required. For more information, refer to Business Process Portal Administrators Guide. FORMAT 2 (Adapter logs) Adapter logs have a fixed format and cannot be modified. For example: Mon May 29 14:41:32 IST 2017 [MESSAGE] [ #424] [44] Adapter invoked. FORMAT 3 (AppServer logs) Configure the log formats of the external systems to similar formats. Consider the example of the default bpsbroker1 log for OpenEdge Classic AppServer. This log has the following format: [2017/03/31@18:42: ] [3] [AdminServer] Reading environment from file: C:\Progress\OpenEdge\fathom.init.params 14

15 Logstash configuration Logstash configuration Logstash uses a configuration file which defines which log files to inspect (Logstash Input), how to parse different log formats (Logstash Filters), and the location of Elasticsearch database to post each log entry (Logstash Output). The following sections briefly describe how to configure Logstash specific to OpenEdge BPM logs. For more information about Logstash configuration, see Logstash input To analyze various logs from log files and identify which log files to inspect, use the file input plugin of Logstash to configure different log files. Consider the following example of the configuration of the file input: input { file { path => "$OEBPS_HOME/server/logs/*.log" start_position => "beginning" type => "oebpm_logs" codec => multiline { pattern => "^%{TIMESTAMP_ISO8601" negate => true what => "previous" Where: path Path to one or more files to inspect. type Attribute used to uniquely identify a group of log files. This can be used to apply filters specific to a type of logs. codec Encodes or decodes before the filters are applied to the log entry. In this example, multiline codec is used to differentiate multi-line log messages. By default, every new line in a log file is considered as a single log entry. Using multiline codec, any log message spanning across multiple lines is configured in a way that anything which does not start with a timestamp is considered as a part of previous log entry. Logstash filters Filters are a crucial part of Logstash configuration. The configuration must be able to parse the timestamp for each log entry appropriately which in turn allows you to run time-based queries on Elasticsearch using Kibana. 15

16 Chapter 3: Configuring ELK stack for OpenEdge BPM To parse different log formats, consider the following: 1. Logstash parses each log entry using GROK expressions. You must identify the different log formats you want to inspect, and then create a GROK expression for each one. 2. The following are three different GROK expressions provided for three different formats described in Prerequisites on page 14. a. GROK 1 for FORMAT 1 (Component and Container specific logs) %{TIMESTAMP_ISO8601:timestamp\s+%{LOGLEVEL:bpm_loglevel\s+%{WORD:bpm_category\s+%{GREEDYDATA b. GROK 2 for FORMAT 2 (Adapter logs) %{ADAPTER_LOG_DATEFORMAT:timestamp \[%{WORD:priority\] \[%{NOTSPACE:piName\] %{GREEDYDATA:message c. GROK 3 for FORMAT 3 (AppServer logs) \[%{BPSBROKER_TIMESTAMP:timestamp\]\s+%{GREEDYDATA 3. When you inspect various log files with different formats, you may find a non-standard (custom) date format. In such cases, add a custom GROK, specific to that format, to the <LOGSTASH_HOME>\vendor\bundle\jruby\1.9\gems\logstash-patterns-core-4.1.0\patterns\grok-patterns file. For example: For Mon May 29 14:41:32 IST 2017 date format, the custom GROK can be ADAPTER_LOG_DATEFORMAT %{DAY %{MONTH %{MONTHDAY %{HOUR:%{MINUTE:%{SECOND %{WORD %{YEAR For 2017/03/31@18:42: date format, the custom GROK can be BPSBROKER_TIMESTAMP %{YEAR/%{MONTHNUM/%{MONTHDAY@%{HOUR:%{MINUTE:%{SECOND\.\d{3%{ISO8601_TIMEZONE To create or verify your GROK expression, you can use the tool available at 4. When you have the appropriate GROK expression ready, add the grok section under the filer configuration as shown in the following code: 16

17 Logstash configuration filter { # filter for bpsbroker AppServer logs if [type] == "bpsbroker" { grok { match => { "message" => "\[%{BPSBROKER_TIMESTAMP:timestamp\]\s+%{GREEDYDATA" date { match => ["timestamp","yy/mm/dd@hh:mm:ss.sssz"] # filter for BPM Adapter logs else if [type] == "adapter_log" { grok { match => { "message" => "%{ADAPTER_LOG_DATEFORMAT:timestamp \[%{WORD:priority\] \[%{NOTSPACE:piName\] %{GREEDYDATA:message " date { match => ["timestamp", "yy/mm/dd@hh:mm:ss.sssz"] # filter for BPM component & container specific logs else { grok { match => { "message" => "%{TIMESTAMP_ISO8601:timestamp \s+%{loglevel:bpm_loglevel \s+%{bpm_log_category:bpm_category \s+%{greedydata" date { match => ["timestamp", "ISO8601"] 5. You must set the timestamp for each log entry before it is posted to Elasticsearch using the date plugin. 17

18 Chapter 3: Configuring ELK stack for OpenEdge BPM 6. Even when Logstash fails to parse a log entry using the given GROK expression, it is still included in the output with a _grokparsfailure tag appended to the Elasticsearch record. 7. Adapter logs for OpenEdge adapter contain logs from BPM and logs from 1PAdapters in the same file, where each log follows different formats. Following is an example of OpenEdge Adapter log entries with two different formats: Tue Jun 06 12:16:01 IST 2017 [MESSAGE] [WSAdapter#274] [189] Executing OE Managed Adapter Workstep Activity 1 Tue Jun 06 12:16:01 IST 2017 [-1] [WSAdapter#274] [189] Tue Jun 06 12:16:01 IST 2017 [TRACE] [WSAdapter#274] [189] Created OECafAdapter :16: [INFO][OEAdapter] Managed adapter running in BizLogic :16: [INFO][OECafAdapter] Invoking CAF adapter :16: [INFO][OECafAdapter] Using Cached CAF Context... Following is a sample filter to handle such scenarios: filter { grok { # Match different formats with their corresponding GROK expressions # An array of even numbered entries where each checked for a match, till a match found match => [ "message", "%{ADAPTER_LOG_DATEFORMAT:timestamp \s+\[%{notspace:priority\] \s+\[%{notspace:piname\] \s+\[%{notspace:threadid\] \s+%{greedydata", "message", "%{1PADAPTER_DATEFORMAT:timestamp \s+\[%{notspace:priority\]\[%{notspace:component\] \s+%{greedydata" ] # Three letter timezones are not supported by default, mutate with the ZZZ format # Otherwise the date parsing will be failed. mutate { gsub => ["timestamp", "IST", "+05:30"] # Parse timestamp for different date formats date { match => [ "timestamp", "E MMM dd HH:mm:ss ZZ yyyy", "yyyy-mm-dd HH:mm:ss.SSS" ] 18

19 Exploring Elasticsearch with Kibana Logstash output To define the location of Elasticsearch database for posting each log entry, consider the following: 1. The output configuration of Logstash defines the destination where the filtered log entries must be written. 2. Before configuring Elasticsearch output plugin, it is recommended to configure a file output for verifying if the input and filter sections are configured properly. output { file { path => "c:/temp/log-stash-all.json" 3. To launch Logstash with a specific configuration, save the Logstash configuration in the C:\Temp\oebpm-logstash.conf file and run this command: $LOGSTASH_HOME\bin>logstash -f C:\Temp\oebpm-logstash.conf 4. The filtered log entries from all the log files are stored in the log-stash-all.json file. Each entry has a JSON object which lists the source file, variables configured in GROK express, tags attaches, type, and timestamp. For a log entry, :51:28,155 INFO OEBPS - SQL script file <archiveschedulerdrop.sql> executed successfully the following JSON is generated: "Jun 02nd 2017, 1, _id: AVv3gTGX0oo0B9iv, _type: "oebpm_logs", bpm_category: "OEBPS", bpm_loglevel: "INFO", host: "nbhydskothuri10", message: "SQL script file <archiveschedulerdrop.sql> executed successfully", path: "C:/Progress/OpenEdge/oebpm/server/logs/oebps.log", type: "oebpm_logs", timestamp: "Jun 02nd 2017, 16:51:28.155" 5. When there is a failure during parsing, check for additional tags that get attached such as _grokparsefailure, _dateparsefailure, etc. in each entry. Also, check for field value which is a default timestamp for the output entry if the parsing is performed properly. Exploring Elasticsearch with Kibana Once the Logstash output is verified with the file output plugin and no errors are found, you can start other components of the stack Elasticsearch and Kibana. 19

20 Chapter 3: Configuring ELK stack for OpenEdge BPM Consider the following: 1. You can configure the Elasticsearch output plugin for posting the log entries to Elasticsearch only if no errors are found after writing the output to the file. 2. Before posting the log entries to Elasticsearch database, start elastic search ibn localhost with the following command: $ELASTICSEARCH_HOME\bin>elasticsearch This starts the Elasticsearch database on localhost at port Add the Elasticsearch output to Logstash configuration and re-launch Logstash. output { elasticsearch { hosts => ["localhost:9200"] index => "oebpm" Elasticsearch requires an index name (for example, oebpm) to group the incoming log entries together. If this is not specified, Logstash creates an index in the logstash-timestamp format. 4. Start Kibana using the following command: $KIBANA_HOME\bin>kibana This starts the Kibana server on localhost at port Open Kibana in your favorite web browser with URL 5. When you launch Kibana for the first time and if no indexes are created, the following configuration window appears: 20

21 Exploring Elasticsearch with Kibana 21

22 Chapter 3: Configuring ELK stack for OpenEdge BPM 6. If you already provided the index name as oebpm and it is created in Elasticsearch database successfully, Kibana detects it and enables other fields. 7. Click Create to start browsing the logs (Elasticsearch records) from the Discover page. 22

23 Exploring Elasticsearch with Kibana For more details about how to use Kibana interface, refer to 23

24 Chapter 3: Configuring ELK stack for OpenEdge BPM 24

25 4 Use cases For details, see the following topics: Use case 1 (Tracking user login and logout events) Use case 2 (Troubleshooting OpenEdge Adapter) Use case 1 (Tracking user login and logout events) This use case demonstrates how to audit BPM user login and logout events using custom GROK patterns to scan Business Process Portal logs. 1. Business Process Portal generates Audit Logs to inspect various process specific events, and you can inspect user login and logout events only by using these logs. 2. But now, you can query the login and logout event of BPM users using the Logstash configuration with the following GROK patterns: grok { match => { # temp_qstring is a placeholder for quoted strings "message" => "%{TIMESTAMP_ISO8601:timestamp \s+%{loglevel:bpm_loglevel \s+%{bpm_log_category:bpm_category \s+[^\"]*%{space(?:%{qs:temp_qstring)?%{space%{greedydata" if [bpm_category] == "Portal" { # Log patterns when a user logged in. if [message] =~ ".*dologin.*6119.*" { 25

26 Chapter 4: Use cases mutate { add_tag => "bpm_user_login" # Capture the quoted string in a field named bpm_user add_field => {"bpm_user" => "%{temp_qstring" else if [message] =~ ".*logoutuser.*6134.*" { mutate { add_tag => "bpm_user_logout" add_field => {"bpm_user" => "%{temp_qstring" mutate { # Drop the placeholder field temp_qstring remove_field => ["temp_qstring"] gsub => ["bpm_user", "\"", ""] 3. When the configuration is successfully parsed and posted to Elasticsearch, you can run the following queries from Kibana: bpm_category:"portal" AND (tags:"bpm_user_login" OR tags:"bpm_user_logout") 26

27 Use case 1 (Tracking user login and logout events) 4. The query lists all the BPM user s login and logout events with time stamps. Further, to query the details for a user, run the following query including the user specific condition: bpm_category:"portal" AND (tags:"bpm_user_login" OR tags:"bpm_user_logout") AND bpm_user:"skothuri" 27

28 Chapter 4: Use cases Use case 2 (Troubleshooting OpenEdge Adapter) This use case demonstrates how the ELK setup helps you to identify the root cause of an OEAdapter failure using time-based filtering and custom queries. 1. Consider an OEAdapter workstep which is suspended due to an unknown internal error. 2. In the Home module of Business Process Portal, click My Instances > Process Status Viewer > Audit History. 3. Note the time stamp when the OEAdapter workstep is activated. In this example, the time stamp is 12th June 12, :31 PM as shown in the following figure: 4. Navigate to Kibana s Discover page and perform the following: a) Using BPM audit logs, you can observe that the adapter has been activated and suspended at :31. Set the time filter based on the time frame starting from :31:00 to a small buffer till 17:32:59:

29 Use case 2 (Troubleshooting OpenEdge Adapter) b) In the query field, available at the bottom of the page, query using the type, tags, and fields to filter the logs for more specific components. In this example, the query type:"oe_adapter_log" AND priority:"error" AND tags:"abl_runtime_error" searches for OEAdapter logs with loglevel set to ERROR and has a tag abl_runtime_error. This query checks if there is a failure when the AppServer executes the ABL procedure. For more details, refer to the sample Logstash configuration available at 5. If you find that there are entries tagged as abl_runtime_error, check the OpenEdge AppServer broker logs for BPM (bpsbroker1). 6. To get the bpsbroker logs for the specified time frame, run the following query in the query field: 29

30 Chapter 4: Use cases 7. As you browse through the 16 log entries, you can locate an application log which reads Input received as ERROR logged from a procedure MutateInput.p at line no. 22 (as it is programmed in ABL procedure). 30