End-user experience monitoring with FMS, FxM and FxV

Size: px

Start display at page:

Download "End-user experience monitoring with FMS, FxM and FxV"

Sherilyn Robinson
5 years ago
Views:

1 End-user experience monitoring with FMS, FxM and FxV Aniello Human September 2011 Abstract: This document will demonstrate a method of integrating data from the Foglight Experience Monitor (FxM) and Foglight Experience Viewer (FxV) with the Foglight Management Server (FMS) to be used in creating end-user experience dashboards, rules and drilldown capabilities in a reusable way. Intermediate to advanced experience is assumed with all three technologies.

2 END-USER EXPERIENCE MONITORING WITH FMS, FXM AND FXV... 1 OVERVIEW... 3 DASHBOARD DETAILS... 4 Response Time... 4 Component Details... 4 Long Running Transactions... 5 Success Ratio Disk... 5 Java traces CREATE FXV HITFILTERS FOR TRANSACTIONS CREATE FXM APPLICATION COMPONENTS CONFIGURING THRESHOLDS KEY PRINCIPLES IN DESIGN ADVANCED TOPIC: AUTOMATIC JAVA TRACES FOR TRANSACTIONS CONFIGURING A RULE TO COLLECT JAVA TRACES CONCLUSION AND FUTURE WORK... 19

Overview In this example we ll be looking at a typical shopping cart-like web application and a way of configuring the FxM, FxV and FMS to monitor the end-user experience.

3 Overview In this example we ll be looking at a typical shopping cart-like web application and a way of configuring the FxM, FxV and FMS to monitor the end-user experience. The dashboards were built using an excellent Portlet Library created by Philipp von dem Bussche- Hünnefeld. The Portlet Library is a bunch of custom WCF views that help to quickly create drag-and-drop dashboards from a number of different topology types. The one we ll be using accepts an FxMApplicationResult as input. If you have enough WCF experience you can build your own FxMApplicationResult portlets. The focus, however, of this document is on a method that makes integrating the FxM, FxV and FMS components uniform and therefore reusable. The main components on the dashboard are shown below. They are: Portlets - A type of WCF view used to drag in FxMApplicationResult objects, Success Ratio Disk Ratio of successful HTTP requests with a popup that can drill into the FxV, and Application Details A popup giving more detail on the response time and errors for an FxMApplicationResult, also with FxV drill-down.

4 Dashboard details This section provides a quick tour of the details behind each of the portlets. The intent is to give the reader an idea of how these portlets are integrated with the FxM and FxV data. Response Time The main metric in each of the portlets in the breakdown view is the pagedownloadtime average. This metric reflects the average end-user experience in terms of response time. The dashed line in the chart is a derived metric which calculates pagedownloadtime for the past 30 days and is used as a baseline for the response time. Component Details When clicking on any of the charts in the breakdown view, the Application Details popup appears. It displays key metrics from an FxM application

component as well as extra derived metrics, like session and timeout count for the last 30 days. These derived metrics give better context to key metrics.

The disk shows the ratio of successful requests to the total number of requests with client and server errors indicating a request failure.

5 component as well as extra derived metrics, like session and timeout count for the last 30 days. These derived metrics give better context to key metrics. Long Running Transactions Clicking on Long Running Transactions drills into the FxV and searches for any sessions which match the specific component e.g. Search Customer transactions that took longer than their defined threshold: Success Ratio Disk The disk below each of the charts is called a success ratio disk. The disk shows the ratio of successful requests to the total number of requests with client and server errors indicating a request failure. The color of the disk will change from green to red depending on the success ratio. Clicking on the disk will present a popup showing the result codes. The result code data comes from the FxV. Clicking on any of these result codes would drill into the FxV and search for the matching hitfilter.

Java traces When the end-user experience starts to degrade it is also useful to have data available from the back-end systems to discover any areas contributing to slow response.

6 Java traces When the end-user experience starts to degrade it is also useful to have data available from the back-end systems to discover any areas contributing to slow response. As such, we collect automatic Java traces on requests when the average pagedownloadtimemean exceeds the warning threshold. By the time performance is significantly bad we have a collection of Java traces available to interrogate. With Java cartridge version it is also possible to collect HTTP parameters and therefore to link this data back to FxV sessions. In our case we want to go the other way around: If the end-user experience degrades, collect traces and hopefully some of the poor end-to-end experiences will have been traced (and can be matched via JSESSION ID).

Create FxV hitfilters for transactions The first step is to identify the core transactions in your

7 We will now present a general pattern which can be followed to get the above demonstrated integration working. 1. Create FxV hitfilters for transactions The first step is to identify the core transactions in your application. Things like Search customer, Add to basket, Create customer, Checkout etc. are good candidates. Once you have identified these in the FxV you can build hitfilters on the matching conditions.

8 One of the principles we follow is to name our hitfilters with a prefix which indicates the application ( DWeb in this case). This is useful if you have multiple web applications sending traffic to the same FxM/FxV appliances and want to keep them separate; and also because we rely exclusively on convention over configuration to make things like the drill-down features of the dashboards be generic. Each of our hitfilters has a metric which reports to the FMS the HTTP response code data for each transaction. This metric is used in the success ratio disk of the portlet to show the count for each HTTP response code and also gives a hook for us to use when we drill-down into the FxV when searching for a transaction with a given response code. It s also useful to create FxV links in your alerts which provide a URL that support staff can use to replay a session. This is done by using the template metric AppName TransName HTTP $0 which will create a unique metric for each response code.

Once data starts coming in you should see that each of your transactions have created a

9 Repeat this process until you have created a hitfilter for each of the transactions you want to see on your dashboards. Once data starts coming in you should see that each of your transactions have created a template metric AppName TransName HTTP $0 in the metric list of the FxV. In your FMS you will have a separate metric for each HTTP response code under the FxVStateMetrics.

10 2. Create FxM Application Components For each of the transactions we also need to create an FxM application component. These are the types which the portlet expects as input to create the dashboard views. Each application component must have exactly the same name as the FxV hitfilter as well as the same hitfilter conditions. If your hitfilter makes use of HTTP response content then this will not work. However, most transactions can be identified on the HTTP header and query parameters. The reason for having the same name and conditions is so that drill-down from the FMS can be geneirc. When you have an FxM component of name DWeb Search Customer and you want to replay a session, your portlet can create an iframe drill-down to the FxV with a search on hitfilters with name DWeb Search Customer, if the convention over configuration principle was adopted. This is the key to making your components reusable. Note how the FxM application component has the same name as the FxV hitfilter. It also has the same conditions in the regular expression:

11 After configuring all your transactions, your FxV hitfilters and FxM application components should have the same names. You now need to configure the FxM to send data from these application components to the FMS as metrics:

12 Application components can now be dragged into the dashboard from the FxM Application Components we just created:

13 3. Configuring thresholds In our design, we make use of the same registry variable in our rules and thresholds. This has the advantage of creating a good sense of context for dashboards. If the threshold band is green you know no alerts fired on that rule. If it is red you know that a Fatal has probably fired. In order to get threshold bands in dashboards correspond to thresholds in multiple-severity rules we make use of three registry variables. AppName-PageEndToEndTime- {Warning,Critical,Fatal} The registry variable allows you to specify a global default for each of your transactions on a warning, critical and fatal level. It also allows you to override the threshold for a specific transaction with

scoped registry values. As shown in the image, Search Customer has a critical threshold of 7 seconds which is different from the default critical value and the Checkout transaction.

14 scoped registry values. As shown in the image, Search Customer has a critical threshold of 7 seconds which is different from the default critical value and the Checkout transaction. These scoped values apply to rules and also the threshold bands in dashboards. We then use this registry variable when creating the Threshold for the pagedownloadtimemean metric, which is the main metric shown in the portlet charts, and our rule:

15 The end result is a dashboard which has deep integration with FxM and FxV for drill-down purposes. The visual appearance of the dashboard is also neatly linked to the rules that fire on it through a registry variable that is shared by the severity-level rules and the threshold-severities. Success ratio disks let us know if users are having trouble other than poor response. Baselines on the dashboards let us know how the current performance compares to the past 30 days: 4. Key principles in design The key principles in this design are: 1. FxM application component names and conditions must match FxV hitfilter names and conditions 2. FxV hitfilters must have an HTTP Response Code $0 metric From these 2 principles it becomes easy to have generic portlets that can do drill-down, generate drill-down links in s and add more advanced things like automatic Java traces on transactions when performance goes south.

5. Advanced topic: Automatic Java traces for transactions One thing we could do with this design is to automatically take Java traces on the transactions when performance degrades.

16 5. Advanced topic: Automatic Java traces for transactions One thing we could do with this design is to automatically take Java traces on the transactions when performance degrades. For example, suppose that a Search Customer transaction starts trending up and you notice this from the dashboard view. When it gets to the Fatal band you would have received an SMS that the end-user experience is terrible. You suspect it might be the frontend Java server doing something like a T3 call to another Java server that is taking long to respond. You don t know. If you were to start tracing by the time you received the Fatal you are doing reactive monitoring. We want to be proactive, so we want to take a trace when we hit the Warning threshold. To do this we need to configure a rule that looks at the response time of the Java requests in question and if it s too high, kick off a trace. 6. Configuring a rule to collect Java traces What we need is a rule action that will collect a trace, once again, using exactly the same condition that our FxM application component and FxV hitfilter conditions are configured as. This is easy from the JavaEE dashboard:

17 But we need to do this from a rule action to make it automated:

18 The obvious problem here is that we don t know what the FxV hitfilter condition or FxV Application Component regular expressions are. Fortunately the FxM has a handy SQL database which is well documented in the FxM help and we can use this to get to the regular expressions used to configure the application component. There are two ways that are fairly out-of-the-box to get to these expressions. The one way is to use write a persistable script that would query the SQL database and get the expressions based on e.g. scope.name if your Java trace rule is scoped on a FxMApplicationResult.

A second way of doing this would be to create a GenericSQLAgent that queries the FxM database and brings back the regular expressions for each application component.

19 A second way of doing this would be to create a GenericSQLAgent that queries the FxM database and brings back the regular expressions for each application component. You can now use the regular expressions built for each FxM application component to configure the requestname in your JavaEETracesAction. 7. Conclusion and Future Work We this document will give FxM/FxV users a few interesting ideas and principles which have helped us build a very useful set of enduser monitoring components. Our hope is that Quest will further extend the out-of-the-box integration of the FxM and FxV appliances with the FMS. In this design there are many steps which are still tedious, and yet completely unnecessary from a configuration point of view. Portlets like those built by Philipp von dem Bussche-Hünnefeld should be stock-standard with any FMS deployment. The FxM and FxV application component and hitfilters need not be so disparate. The Java trace capability should have a pro-active element like we have shown is possible and Java traces should automatically be get FxM (and perhaps FxV) regular expressions / filter conditions without the FMS administrator having to dig into the FxM SQL database.

Using WinTask to Extend ehealth Application Monitoring

Using WinTask to Extend ehealth Application Monitoring WinTask is a task-automation tool that can record and play back user actions for any Windows-based application. It captures all the user activity