Keysight TAP Training Lab 2 TAP Results Viewer

Transcription

1 Keysight TAP Training Lab 2 TAP Results Viewer

2 Notices DFARS/Restricted Rights Notice If software is for use in the performance of a U.S. Government prime contract or subcontract, Software is delivered and licensed as "Commercial computer software" as defined in DFAR (June 1995), or as a "commercial item" as defined in FAR 2.101(a) or as "Restricted computer software" as defined in FAR (June 1987) or any equivalent agency regulation or contract clause. Use, duplication or disclosure of Software is subject to Keysight Technologies standard commercial license terms, and non-dod Departments and Agencies of the U.S. Government will receive no greater than Restricted Rights as defined in FAR (c)(1-2) (June 1987). U.S. Government users will receive no greater than Limited Rights as defined in FAR (June 1987) or DFAR (b) (2) (November 1995), as applicable in any technical data. Warranty THE MATERIAL CONTAINED IN THIS DOCUMENT IS PROVIDED "AS IS," AND IS SUBJECT TO BEING CHANGED, WITHOUT NOTICE, IN FUTURE EDITIONS. FURTHER, TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW, KEYSIGHT DISCLAIMS ALL WARRANTIES, EITHER EXPRESS OR IMPLIED WITH REGARD TO THIS MANUAL AND ANY INFORMATION CONTAINED HEREIN, INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. KEYSIGHT SHALL NOT BE LIABLE FOR ERRORS OR FOR INCIDENTAL OR CONSEQUENTIAL DAMAGES IN CONNECTION WITH THE FURNISHING, USE, OR PERFORMANCE OF THIS DOCUMENT OR ANY INFORMATION Technology Licenses The hardware and/or software described in this document are furnished under a license and may be used or copied only in accordance with the terms of such license. Keysight Technologies, Inc. 2015, 2016, 2017 Version 7.3, April 2017 Keysight Technologies, Inc. 900 South Taft Avenue, Loveland, CO, USA Keysight TAP Training Lab 2 Results Viewer Page 2

3 Contents Overview... 4 Lab Objectives... 4 Prerequisites... 4 Installing the TAP Demonstration Plugin... 5 Configuration of Result Listeners... 6 Result Listeners... 9 Results Viewer DataSplit Data Filtering Multiple Charts Templates Test Plan Run Explorer Generate Data Explore the Explorer Compare Test Plans Tags in Explorer Search in Explorer Limit Set Editor Keysight TAP Training Lab 2 Results Viewer Page 3

4 Overview Lab Objectives This lab will help you understand: How TAP handles results and how they can be formatted and stored for later use How and when the TAP Results Viewer can be used to visualize and analyze the results How to use the TAP Test Plan Run Explorer to get an overview, find/view test plan runs, and apply limits to results Prerequisites This lab requires the following software. For instructions on installing TAP, see the TAP GUI Training Lab document. Test Automation Platform (TAP), the latest version downloaded from Test Automation Platform Developer s System at TAP is installed in either: C:\Program Files\Keysight\TAP8 (64-bit systems) C:\Program Files (x86)\keysight\tap8 (32-bit systems) In this document, the TAP install directory is referred as TAP_PATH. TAP license Upon first start, TAP prompts you to request a free trial license. Keysight License Manager (KLM) KLM is installed with TAP. You will use it to install your license. Keysight TAP Training Lab 2 Results Viewer Page 4

5 Installing the TAP Demonstration Plugin In this section we will install the TAP Demonstration Plugin. This plugin contains steps, DUTs and Instruments which are useful for demonstration purposes. 1. Select Windows Start > Keysight Test Automation Platform to start TAP GUI (or double-click C:\Program Files <(x86)>\keysight\tap8\keysight.tap.gui.exe). 2. Select Tools > Package Manager to open TAP Package Manager. 3. Installed plugins are identified with a checkmark ( ) next to package name. To install the demo package: a. Click Demonstration to select the package. b. Click Install. Note: A warning appears if the TAP GUI needs to be closed and reopened. If it does, close TAP GUI while the demonstration plugin is being installed. 4. Close TAP Package Manager. Keysight TAP Training Lab 2 Results Viewer Page 5

6 Configuration of Result Listeners A TAP Result Listener is a resource used to store data created by Test Plan execution. TAP includes multiple result listeners. Result Listeners are plugins, so new/custom listeners can be created and used in the GUI. Multiple Result Listeners can be configured at the same time and will process the data from TAP in parallel. Note that this happens in a separate thread so it does not impact the execution speed of the Test Plan. In order to use the Test Plan Run Explorer and Results Viewer, a Database Result Listener is needed. Follow these steps to set up the Result Listeners for this lab. 1. Open the TAP GUI if it is not open already. 2. Select Settings > Results to open the Results tab in the Settings window. The settings show the Log Result Listener, which is enabled by default: 3. Click the + button in the lower left corner of the Settings window. Keysight TAP Training Lab 2 Results Viewer Page 6

7 4. The Add New Result Listener window displays groups of supported result listeners. In this window: 5. Click the blue arrows ( )to expand the groups. 6. Click Add next to the following result listeners: Database: SQLite (use default settings) Text: CSV (use default settings) 7. Notice that the result listeners are added in the Resource Bar of TAP GUI: 8. Close the Add New Result Listener window. Keysight TAP Training Lab 2 Results Viewer Page 7

8 9. In the Settings window: a. Navigate to the GUI tab. b. Disable the Show Results Viewer setting if it is enabled. When disabled, the TAP Results Viewer does not open when each test plan completes. 10. Close the Settings window. Keysight TAP Training Lab 2 Results Viewer Page 8

9 Result Listeners To get familiar with Result Listeners, we will work with the Data Gen for Results Viewer test plan. 1. Select File > Open and and navigate to TAP_PATH\DataGenForResultsViewer.TapPlan (TAP_PATH is typically C:\Program Files\Keysight\TAP8 or C:\Program Files (x86)\keysight\tap8). The test plan opens: 2. Let s examine the four steps in the test plan. Click on each step to inspect its Step Settings: Ramp step generates a ramp of double results. Sine1 step generates a single sine wave of double results. Sine1 has these settings: Sine2 step has same configuration as Sine1, except it only generates 30 points. Sine3 step also has same configuration as Sine1, except noise is enabled for sine wave. Keysight TAP Training Lab 2 Results Viewer Page 9

10 3. Diversion: Dynamic Steps. Let s make some temporary changes to the plan to learn about dynamic step names. a. Select the Ramp step in Test Plan. Notice the settings of this step: We will use the name of first setting Point Count to dynamically rename of test step. b. Rename the step. In the Test Plan panel, double-click the step name (or select the step and press F2). Change the name from Ramp to exactly the following: Ramp ({Point Count} points) c. Notice that {Point Count} is replaced with 50, the actual point count, in the new name: d. Now, change value of Point Count in the step settings to 60. This change is reflected in step name in Test Plan Panel. Double-clicking the step name shows the dynamic name: e. Select File > Open and re-load the original DataGenForResultsViewer.TapPlan file. Do not save the changes you made. 4. Follow these steps to generate and examine data: a. Run Test Plan by clicking the Run button or pressing F5. b. In Windows Explorer, open the Results directory (typically TAP_PATH\Results). You should find a number of files with.txt and.csv suffixes. For example: c. Examine the TXT file results. The.txt files were generated by the Log Result Listener. These files are designed to log the execution of a Test Plan, and are useful for debugging and time Keysight TAP Training Lab 2 Results Viewer Page 10

11 optimization. They are not useful for looking at results detail besides the verdict. An example: d. Examine the CSV file results. The.csv files include the numerical results from a test plan run. The file can be inspected in a text editor, but also imported into Excel. This examples shows a.cvs file imported into Excel: The SQLite results are based on a lightweight database. It consists of a single file (stored on local machine) that encapsulates the full data set. It has a much more flexible interface compared to the basic result listeners, which allows us to use TAP Results Viewer to view the data. The next section is dedicated to the TAP Results Viewer. Results Viewer The TAP Results Viewer is a separate executable available from the TAP GUI Tools menu or by double-clicking Keysight.Tap.ResultsViewer.exe in the TAP installation folder. It allows you to view, filter, label, and export the Test Plan results that are stored in a database Result Listener. These results include measurement values, verdicts, and log data. Follow these steps to get familiar with TAP Results Viewer: 1. Click Tools > Results Viewer to open the TAP Results Viewer, which displays the results from the last test plan execution: Keysight TAP Training Lab 2 Results Viewer Page 11

12 By default, the Results Viewer shows one chart for each result type published during Test Plan execution. 2. Notice that the RampResults chart is selected (as indicated by a blue border). The settings to the right of the chart are valid for this chart. Each chart has its own settings. This enables charts to be configured individually (as we will see later). The settings are grouped into four groups for customization of chart/data: Chart Settings: Overall settings Legend: Chart legend settings Series: Formatting (such as filter and split) of data series Chart Data: Visualization of charts 3. Experiment with the different Chart Type for the RampResults plot by going to Chart Settings > Chart Type and selecting other types: XY Scatter: Displays data as X/Y points. XY Line: Displays data with straight lines connecting X/Y data points. Column: Displays data as a series of vertical columns. Histogram: Displays a graphical representation of the distribution of data. Data Table: Displays data is listed in table. Data Table (Full): Displays data and all possible chart data parameters are listed in a table. 4. Set Chart Settings > Chart Type to Data Table (Full): The table shows all data stored in the database for this test plan run. The first columns show data and metadata for the test plan run, including test plan name, run ID and test plan verdict: Keysight TAP Training Lab 2 Results Viewer Page 12

13 The table also contains data and metadata for test step run. Scrolling to the right in the table above to see the results published by the step: TAP Results Viewer is powerful for visualizing results in different ways. The following sections provide insight to this. DataSplit Perform these step to get familiar with splitting of data: 1. Scroll to SineResults and click on the chart to select it. Notice that the chart type is XY-Scatter plot: Keysight TAP Training Lab 2 Results Viewer Page 13

14 2. In Chart Settings > Chart Type, change the chart type to XY-Line. You should see something like this: Notice that the results from Sine1 (50 points, noise disabled), Sine2 (30 points, noise disabled), and Sine3 (50 points, noise enabled) are treated as one data set. Hence the graphs are connected (as indicated above with the dashed red lines). The solution to this is to split the data on something unique to each step. 3. Under Series, click Add a Modifier and select Split: Keysight TAP Training Lab 2 Results Viewer Page 14

15 4. In the Split field, select Step > Name. The Series settings looks like this: The chart should now look like this: It is possible to split a data set on any of the data associated with it in the database. Hence the split can be set up for any of the columns seen for Data Table (Full) in one of the previous steps. Keysight TAP Training Lab 2 Results Viewer Page 15

16 Data Filtering Let s get familiar with filtering of data by looking at the Result Set that has noise enabled: 1. Add a filter to the settings. Click Add a Modifier and select Filter. 2. In the Filter fields, select: Step > Results > Noise Enabled for the filter True as the value 3. Now the chart should only show the noisy data series (Sine3): 4. Under Chart Settings, change the Title to SineResults Noise. The chart title changes. Keysight TAP Training Lab 2 Results Viewer Page 16

17 Multiple Charts As previously mentioned, by default the Results Viewer shows one chart for each result type/name published during test plan execution. It is possible to add and remove charts from the viewer. In the following steps we will customize the view so that: SineResults has two charts: SineResults Noise Enabled SineResults Noise Disabled RampResults does not display a chart 1. Scroll to the top of the chart area so that RampResults is visible. Click the X to delete the chart: 2. Now you only see the SineResults Noise Enabled chart. Click the + button to duplicate the chart: Two SineResults Noise Enabled charts are now shown. 3. Scroll down to the new chart and click on it to select it. Change the Series > Filter value to False: 4. Change Chart Settings > Title to SineResults Noise Disabled. Now you should see two charts: SineResults Noise Enabled and SineResults Noise Disabled. Templates It is possible to save custom chart settings in a template. This template can later be applied to charts in the Results Viewer. Follow these steps to save the current chart settings to a template: Keysight TAP Training Lab 2 Results Viewer Page 17

18 1. Select Template > Save As to save the template. 2. Enter DataGenForResultsViewer for the template name. The.TapReport file extension will be added to the name. Note: If the template name is the same as the test plan name, the template is automatically applied when Results Viewer opens after the test plan runs. 3. Close Results Viewer. 4. In the GUI, run the test plan again by clicking the Run button or pressing F5. 5. Select Tools > Results Viewer. Notice that you see two SineResults charts: SineResults Noise Enabled SineResults Noise Disbled 6. Close Results Viewer. Keysight TAP Training Lab 2 Results Viewer Page 18

19 Test Plan Run Explorer TAP Test Plan Run Explorer is a separate executable available from the TAP GUI Tools menu or by double-clicking Keysight.Tap.RunExplorer.exe in the TAP installation folder. TAP Test Plan Run Explorer extracts test plan execution data from a database. It allows you to: Open the log file or test plan associated with a run Plot results using the TAP Results Viewer Search the database(s), export results, compare test plans, compare logs using the TAP Timing Analyzer Apply limit sets to results Generate Data Perform these steps to generate data for the following exercises: 1. Open the DataGenForResultsViewer test plan (If not already open). 2. Click on Sine2 step in test plan. Enable Limit Checking > Enabled in step settings: 3. Run test plan by clicking the Run button or pressing F5. The verdict of the test plan is Pass. 4. Run test plan again. Same verdict as above. 5. Click on Sine2 step in test plan. Set Limit Checking > Upper Limit to 25 in step settings: 6. Run test plan again. The verdict of the test plan is now Fail. Keysight TAP Training Lab 2 Results Viewer Page 19

20 Explore the Explorer Now it is time to dive into the TAP Test Plan Run Explorer. 1. Click Tools > Test Plan Run Explorer. Notice that when Show is set to Runs, the Explorer shows a list of runs. For each run, the run ID, test plan name and test plan verdict are shown. Any tags (explained later) applied to a test plan run are also shown. 2. Now close the TAP GUI that is running in the background. If you are prompted to save test plan changes, click No. Remember that Test Plan Run Explorer is a standalone application, so it will remain open even after closing the TAP GUI. 3. Now set Show to Results. A list of results is shown: This shows all result types in database and the number of rows in each result type. 4. Set Show to Runs again. 5. Double-click the newest test plan run. The Explorer now shows result types for that run: Keysight TAP Training Lab 2 Results Viewer Page 20

21 The result names and number of rows in each result type are shown. Notice that a filter is applied to only display results in test plans where Run ID = 5. We will cover Test Plan Run Explorer filtering in more detail later. 6. Double-click on SineResults to plot the results using TAP Results Viewer. Notice that the viewer is launched. Since you are already an expert in the TAP Results Viewer, close it by selecting File > Exit or clicking the X in the window. 7. Click on the newest test plan run and notice that buttons are enabled: You can click the following icons: Open the Test Plan of the selected run in TAP GUI Open the results of the selected run in TAP Results Viewer Open the Export Results dialog: Keysight TAP Training Lab 2 Results Viewer Page 21

22 It is possible to export results using any result listener that stores the data to a single file. This is the case for the Log, CSV and SQLite result listeners included with TAP. It is possible to configure each export format. 8. Click Cancel to close the dialog. Now you should see the TAP Test Plan Run Explorer again. 9. Click the icon to open TAP Timing Analyzer, which also is a standalone tool/executable. It is a very powerful tool that can help users: Optimize test plans Locate bottlenecks in execution Identify areas where parallel execution could be done The Timing Analyzer visualizes the test plan run with respect to the timing. This is based on the log file of the test plan run. The TAP Timing Analyzer looks like this: In the Timing Analyzer: Each element of the Test Plan is shown as source on the left. The Time View window on the right shows time sequencially, with a visual break down of how long each action took and when it occurred. The Log Panel shows the log used for the timing analysis. If you click on a log entry in this panel, the entry gets highlighted in the Time View. Keysight TAP Training Lab 2 Results Viewer Page 22

23 The example above is very basic, but hopefully helps you understand the benefit of TAP Timing Analyzer. Now you are aware of this amazing tool! Close the Timing Analyzer by clicking the X. Note: It is also possible to compare two test plan runs in Timing Analyzer with the Compare Logs Selection Mode function ( Compare Test Plans ). We will not cover this during this lab. You can compare test plans in Test Plan Run Explorer. You can configure TAP to use your favorite comparison tool. Make sure to have a compare tool installed prior to proceeding. If you do not have a favorite compare tool, Diffuse is a free tool used in the steps below, and vsdiffmerge.exe is installed on your system if you have Visual Studio. Note: You can also do basic compares without a compare tool. In this case, skip to Step From Test Plan Run Explorer click Settings. 2. From the Settings dialog, browse to the Compare Tool Path for the application you want to use. 3. In Compare Tool Arguments, enter any arguments to be used when launching the tool (if necessary). Use <pathx> and <namex> (replacing X with a number starting at 1) to get the data of the selected test plans. For example, the DiffUse compare tool supports labeling each file with -L, as well as pointing to the path of each file to compare. Use <name1> and <name2>, to insert the names of the selected files. Use <path1> and <path2> to insert the paths to the selected files. 4. Close the dialog window. Keysight TAP Training Lab 2 Results Viewer Page 23

24 5. Select the two newest test plans by holding Ctrl and clicking on test plans: 6. Select Test Plan > Compare Two Test Plans. One of three popups will appear: a. The test plans are identical: b. The test plans are not identical and a compare tool has not been configured: c. The test plans are not identical and a compare tool has been set up. The configured tool shows the differences. In the following example, DiffUse is used: Keysight TAP Training Lab 2 Results Viewer Page 24

25 Tags in Explorer In this section we will explore using tags in TAP Test Plan Run Explorer: 1. Right-click on newest test plan run in TAP Test Plan Run Explorer and select Add Tag 2. In the Add Tag dialog, enter DUT Rev2, then click Ok. 3. You can see in test plan list that the DUT Rev2 tag has been added to the results of this test plan execution: 4. It is possible to apply tags to multiple test plan runs at once: a. Hold Ctrl and click on two test plan runs that are not already tagged DUT Rev2. A grey background identifies the selected plans. b. Right-click when these two are selected and select Add Tag. c. Enter DUT Rev1, then click Ok. The result: Keysight TAP Training Lab 2 Results Viewer Page 25

26 5. It also is possible to apply multiple tags to one test plan run: a. Right-click the second-from-top test plan run and select Add Tag. b. Enter New RF, then click Ok. The result: We suspect that the DUT with New RF is defect but we don't have a proof yet. The theory needs further investigation. We need to inspect the results for these three Test Plan runs. 6. Now we will view the results for these three test plan runs. Select the three test plan runs by holding Ctrl while clicking on each of the runs: 7. Right-click above the selected area and select Plot Using Template. Select the template created in earlier steps (DataGenForResultsViewer.TapReport) and click Open. The SineResults Noise Enabled chart is visible: Keysight TAP Training Lab 2 Results Viewer Page 26

27 8. Notice that data from the three test plan runs is treated as one data set. Hence the graphs are 'connected' as highlighted with the red lines in screenshots above. This can be resolved by splitting the data: a. Click on the SineResults Noise Enabled chart, so that it is becomes active. b. Under Series, click Add a Modifier and select Split. c. In the Split field, select Plan > Tags. Note that the same result can be achieved by splitting on Plan > Run ID, but in that case the informative tag would not be visible in the chart s legend. 9. Now the data is split on Plan > Tag. Notice the legends: 10. Assume the chart above proves your theory to your colleagues and you want to show them Right-click the chart and select Copy as PNG to capture proof. It is also possible to export all charts in Results Viewer as PNGs or docx file with the File > Export command. 11. Close the TAP Results Viewer by clicking File > Exit or clicking the X in the window. Keysight TAP Training Lab 2 Results Viewer Page 27

28 Search in Explorer This section describes the search feature of TAP Test Plan Run Explorer. The search feature works for both test plan runs and results. Follow these steps. 1. Notice the Search field in TAP Test Plan Run Explorer 2. We want to see only test plan runs with verdict set. Click in the Search field and enter plan.verdict!=notset: 3. Now we want to narrow this down to only show DUT Rev1. Click in Search filed and append DUT Rev1 to the previous search term: 4. If we only want to see specific plan runs we can search by Run ID. In the Search field, enter run.id = {1, 4}: Keysight TAP Training Lab 2 Results Viewer Page 28

29 Limit Set Editor The Limit Set Editor lets you define limit sets. These limit sets can be added to existing result sets. Follow these steps to get familiar with the Limit Set Editor. 1. In TAP Test Plan Run Explorer, select Results > Limit Sets start the Limit Set Editor: 2. Click the icon to create a new limit set. Enter SineLimits as name of new limit set, then click Add: 3. Enter these values in Limit Set Editor (as shown in screenshot above): Result Name: SineResults Column Name: Value Lower Limit: 10 Upper Limit: Now the SineLimits limit set looks like this: 5. Click X to save the limit set and close Limit Set Editor. Keysight TAP Training Lab 2 Results Viewer Page 29

30 6. The limits can be applied to any test plan run that contains results of SineResults type. We apply them to test plan run with ID=4. Click in the Search field and enter plan.id = {4} using {SineLimits}: 7. Double-click on SineResults row to launch TAP Results Viewer: Notice the red areas, which indicate the limits. You can hide the limits by disabling Chart Data > Show Limits in settings. You also could have opened the results using the template by clicking Template > Open and selecting DataGenForResultsViewer.TapReport. Keysight TAP Training Lab 2 Results Viewer Page 30

31 8. Set Chart Settings > Chart Type to Data Table (Full). Scroll to the right-most columns in table: Notice that the limit set has been applied to the result set. For each data point you will also see an upper/lower margin and a verdict (if data point is within limit set). 9. Close Results Viewer. This completes the exercises on TAP Results Viewer. We hope you see the power and flexibility of TAP Results Viewer. We also hope you are keen to learn more in the following labs and maybe even start using TAP on your own! Keysight TAP Training Lab 2 Results Viewer Page 31