How To Automate Validation of Tivoli Common Reporting Cognos-based reports

Transcription

1 Tivoli Common Reporting Automation Tests November 2012 How To Automate Validation of Tivoli Common Reporting Cognos-based reports Document version 1.0 Antonio Sgro, Agent Architect for SAPM Stefano Borghetti, IBM Tivoli Monitoring OS Agents Reporting Lead Copyright International Business Machines Corporation All rights reserved. US Government Users Restricted Rights Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.

2 CONTENTS Contents...ii 1.Introduction The report test main flow A New Approach The Architecture How it works Conclusions...8 ii

3 1 Introduction One of the most challenging and expensive activities in the test automation area is the validation of reporting solutions because it involves a set of tasks to be executed that requires a human-gui interaction. Due to the above, it is mostly impossible to have an automation system that is able to produce and validate reports completely avoiding the human interaction. In this article, we introduce an automation attempt in order to validate Tivoli Common Reporting COGNOS-based reports. The goal is to reduce, or avoid (there is possible), human involvement in testing process. 2 The report test main flow The typical flow in testing a report can be summarized in three main macro steps: 1. Generate the report: for each scenario, the user has to manually execute the report providing a specific set of parameter; this means that to generate multiple reports the user has to repeat the same execution action multiple times only changing the set of parameters. 2. Save the generated report: generally for tracking purposes, it is useful to maintain the generated report and store somewhere in a repository. 3. Validate the report: for each generated report, the user has to open it and validate it in terms of look and feel correctness and the data reliability. (1) Generation (2) Save (3) Validation A set of obvious disadvantages can be recognized in the above test flow if it is entirely manually executed: 1. Generate and store reports is a long and tedious work 2. The tester has to open all generated reports to check them even if there are cases where the correctness or a problem is obvious, for example: i. An exception during report generation, in this case COGNOS engine produces a blank PDF/HTML, so the test fails

4 The query return an empty set, in this case typically the report contains and error message like No data, so the test fails In a regression test context, if a report is equal to a correct report generated in the previous test, then we can consider it correct too (success). 3 A New Approach The objectives of the approach we are going to describe are listed below: Full automation of reports generation Full automation of reports storage Automatically recognize specific success and failure cases: i. Recognize blank reports Recognize empty set message inside reports Recognize matching between two or more consecutive generated reports (baseline comparison approach) Give a GUI that helps user in: i. Opening saved reports Mark reports as success or fail Defining a baseline for user validated reports See which report is a blank report, is a no data report or match with the well generated report in baseline, without open it 4 The Architecture The following figure describes the used architecture for the solution: 4

5 Tivoli Common Reporting S-Cat Legacy DB 3 CLI 4 Test Management GUI 5 Test Launcher 2 Test Control Module 1 6 What is showed in yellow background represents components belonging to the prototype that has been implemented following the described approach. The key components are: s-cat. It is an automation tool that allows to run tests scenarios on remote systems. Legacy DB. It contains three main tables: i. The table of test cases: it contains the test cases definitions; a command represents the execution of a specific report with a specific set of input parameters. The table of scenarios: it contains scenarios definitions, a scenario can be considered as a logic group of one or more test cases The table of the test cases results: it contains tests execution results Report Tests Launcher. It is a Java program that runs and validates each test contained in the scenarios mapped in the legacy DB. Test Management GUI. It is a PHP/Dojo web application that uses PHP, Dojo and REST technologies. It allows users to speed up the validation of the test cases results. It is designed in order to minimize the interaction asking the user to make activities only when really needed. A more specific description of this component is in the main flow section, hereafter a very brief list of its main features: i. Shows test cases results in tabular form. Figure 2 Allow to open or download PDF/HTML reports. Allow to manually change the status of the test case result (Success, Wait, Fail). Allow to mark a test case result as baseline for future execution comparison. 5

6 5 How it works The Figure 2 diagram represents the main flow of this solution including the interaction between components and the functionality that each of them implements in order to reach the final goal. There is an activity that is not represented. It is the environment setup that the user should perform once at the beginning and that enable the solution to start multiple test based on that. This activity can be summarized using the following macro steps: 1. The user creates one or more test case inside the legacy DB. Each test case represents all the information required execute a report and get a PDF or HTML file as results. It includes information like report name, input parameter, output format and so on. 2. The user creates a scenario inside the legacy DB and associate it to the test cases created in the previous steps. When the scenario is saved it is automatically synchronized with the list of s-cat scenario so that a new related scenario is created also in the last one.. After these preliminary activities, hereafter there is the description of the main flow described in the picture: 1. The user access s-cat and reserves the test environment. Then it run the scenario previously defined against the Tivoli Common Reporting system. 2. The Test Control Model of the s-cat system invokes The Report Test Launcher on the TCR system passing the scenario name. 3. The Report Test Launcher accesses the legacy DB and retrieves the list of the test cases associated to the passed (and obviously invoked) scenario. 4. The Report Test Launcher, for each retrieved test case: i. Build the TCR command to execute in order to produce a report Use the TCR Command Line Interface and run the command built in the previous step. Check the command result and produce a failure in case of errors. In this solution performs two different type of analysis of the produced report (PDF or HTML): Calculate the MD5 check sum on the produced file. This is because the solution allows defining a report as a baseline to compare it in future running. In other words it allow the user to mark it as the correct report and future new execution are expected to produce the same one. This type of approach is particularly useful in regression scenario where the goal is to verify that nothing is broken. Calculating the MD5 make the comparison between the current file and the baseline faster. Convert the report to image; more precisely it produces a binary raster bit-stream. In this form the Report Test Launcher is able to search for specific sub-images that can brings to specific error scenario. In this prototype we implemented the recognition of a report is empty because of an empty result set of the associated queries. In a similar automation context this approach is useful because some errors can be detected directly without the need to check them manually. 5. The Report Test Launcher accesses the Legacy DB and inserts the result of the executed test cases (including the produced report in PDF or HTML) in a devoted table. 6. The user accesses the Test Management GUI (see below picture). With this GUI he is able to easily and quickly perform the following actions: 6

7 i. See all the results in a unique view. Directly verify the failed testcases because of an error in the command execution or because of a sub-images indicating an error was detected. Open and check only the report that need to be validated. For example in a regression test, there is no need to open reports that already results equals to the baseline, the testcase can be considered in success because this is what is expected. Mark the test case result as the new baseline. For example in a regression test, the expected result is that reports do not change but, in case one of them is different, the user can open if and check the reason. It could be because a problem (he marks the test failed) or because a minor change (i.e. a defect fix). In the second case the user can put the test case result in success and mark it as new baseline for future tests. 6 Conclusions The article introduces a new method in order to automate verification tests for Tivoli Common Reporting reports. Even if in this context it is not possible to automate completely the overall process, the article describes how to heavily reduce manual intervention introducing a consistent gain of time and effort and also provides a valid example about how a similar approach can also help improving the test coverage that will allow increasing products quality. 7