Teradata Profiler. Plug-in for Eclipse User Guide

Transcription

1 Teradata Profiler Plug-in for Eclipse User Guide Release 15.0 B A June 2014

2 The product or products described in this book are licensed products of Teradata Corporation or its affiliates. Teradata, BYNET, DBC/1012, DecisionCast, DecisionFlow, DecisionPoint, Eye logo design, InfoWise, Meta Warehouse, MyCommerce, SeeChain, SeeCommerce, SeeRisk, Teradata Warehouse Miner, Teradata Source Experts, WebAnalyst, and You ve Never Seen Your Business Like This Before are trademarks or registered trademarks of Teradata Corporation or its affiliates. Adaptec and SCSISelect are trademarks or registered trademarks of Adaptec, Inc. AMD Opteron and Opteron are trademarks of Advanced Micro Devices, Inc. EMC, PowerPath, SRDF, and Symmetrix are registered trademarks of EMC Corporation. GoldenGate is a trademark of GoldenGate Software, Inc. Hewlett-Packard and HP are registered trademarks of Hewlett-Packard Company. Intel, Pentium, and XEON are registered trademarks of Intel Corporation. IBM, CICS, DB2, MVS, RACF, Tivoli, and VM are registered trademarks of International Business Machines Corporation. Linux is a registered trademark of Linus Torvalds. Microsoft, Active Directory, Windows, Windows NT, Windows Server, Windows Vista, Visual Studio and Excel are either registered trademarks or trademarks of Microsoft Corporation in the United States or other countries. Novell and SUSE are registered trademarks of Novell, Inc., in the United States and other countries. QLogic and SANbox trademarks or registered trademarks of QLogic Corporation. SAS, SAS/C and Enterprise Miner are trademarks or registered trademarks of SAS Institute Inc. SPSS is a registered trademark of SPSS Inc. STATISTICA and StatSoft are trademarks or registered trademarks of StatSoft, Inc. SPARC is a registered trademarks of SPARC International, Inc. This product includes software developed by Spring Security Project ( Sun Microsystems, Solaris, Sun, and Sun Java are trademarks or registered trademarks of Sun Microsystems, Inc., in the United States and other countries. Symantec, NetBackup, and VERITAS are trademarks or registered trademarks of Symantec Corporation or its affiliates in the United States and other countries. Unicode is a collective membership mark and a service mark of Unicode, Inc. UNIX is a registered trademark of The Open Group in the United States and other countries. Woodstox XML Processor Copyright (c) , Woodstox Project ( Other product and company names mentioned herein may be the trademarks of their respective owners. THE INFORMATION CONTAINED IN THIS DOCUMENT IS PROVIDED ON AN AS-IS BASIS, WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NON-INFRINGEMENT. SOME JURISDICTIONS DO NOT ALLOW THE EXCLUSION OF IMPLIED WARRANTIES, SO THE ABOVE EXCLUSION MAY NOT APPLY TO YOU. IN NO EVENT WILL TERADATA CORPORATION BE LIABLE FOR ANY INDIRECT, DIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES, INCLUDING LOST PROFITS OR LOST SAVINGS, EVEN IF EXPRESSLY ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. The information contained in this document may contain references or cross-references to features, functions, products, or services that are not announced or available in your country. Such references do not imply that Teradata Corporation intends to announce such features, functions, products, or services in your country. Please consult your local Teradata Corporation representative for those features, functions, products, or services available in your country. Information contained in this document may contain technical inaccuracies or typographical errors. Information may be changed or updated without notice. Teradata Corporation may also make improvements or changes in the products or services described in this information at any time without notice. To maintain the quality of our products and services, we would like your comments on the accuracy, clarity, organization, and value of this document. Please teradata-books@lists.teradata.com Any comments or materials (collectively referred to as Feedback ) sent to Teradata Corporation will be deemed non-confidential. Teradata Corporation will have no obligation of any kind with respect to Feedback and will be free to use, reproduce, disclose, exhibit, display, transform, create derivative works of, and distribute the Feedback and derivative works thereof without limitation on a royalty-free basis. Further, Teradata Corporation will be free to use any ideas, concepts, know-how, or techniques contained in such Feedback for any purpose whatsoever, including developing, manufacturing, or marketing products or services incorporating Feedback. Copyright by Teradata Corporation. All Rights Reserved.

3 Preface Purpose This volume introduces the features of the Teradata Profiler Plug-in for Eclipse, a component of Teradata Warehouse Miner and Teradata Master Data Manager. Audience This volume is written for users of the Teradata Profiler Plug-in for Eclipse. Knowledge of Teradata Warehouse Miner, Teradata Master Data Manager or Teradata SQL is not required. Revision Record The following table lists a history of releases where this guide has been revised: Release Date Description TWM /31/13 Maintenance Release How This Manual Is Organized This manual is organized and presents information as follows: Chapter 1: Dependencies describes the prerequisites for using the product. Chapter 2: Installing Teradata Profiler Plug-in for Eclipse describes how to install the product. Chapter 3: Using Teradata Profiler Plug-in for Eclipse describes the features of the product and how to use them. Conventions Used In This Manual The following typographical conventions are used in this guide: Plug-in for Eclipse User Guide iii

4 Preface Related Documents Convention Italic Monospace ALL CAPS Bold GUI Item Description Titles (esp. screen names/titles) New terms for emphasis Code sample Output Acronyms Important term or concept Screen item and/or esp. something you will click on or highlight in following a procedure. Related Documents Related Teradata documentation and other sources of information are available from: Additional technical information on data warehousing and other topics is available from: Support Information Services, support and training information is available from: iv Plug-in for Eclipse User Guide

5 Table of Contents Preface iii Purpose iii Audience iii Revision Record iii How This Manual Is Organized iii Conventions Used In This Manual iii Related Documents iv Support Information iv Chapter 1: Dependencies Software Dependencies Database Requirements Chapter 2: Installing Teradata Profiler Plug-in for Eclipse... 3 Install Teradata Profiler Plug-in for Eclipse Profiler Logging Chapter 3: Using Teradata Profiler Plug-in for Eclipse Overview Project Explorer Add a Database Connection Create the Metadata Database Configure the Metadata Database Update Profiler Plug-in for Eclipse Connect to a Database Managing Folders, Projects and Folders Projects Plug-in for Eclipse User Guide v

6 Table of Contents Values Statistics Frequency Histogram Text Field Analyzer Drill Through Appendix A: References vi Plug-in for Eclipse User Guide

7 List of Figures Figure 1: Data Source Explorer Add Connection to Profiler menu Figure 2: Profiler Project Explorer Add Connection Figure 3: Data Source Explorer Create Metadata Database menu Figure 4: Profile Project/Analysis Explorer Create Metadata Database menu Figure 5: Data Source Explorer Configure Metadata Database menu Figure 6: Profile Project/Analysis Explorer Configure Metadata Database menu Figure 7: Add New Folder window Figure 8: Add New Project window Figure 9: Add New Analysis wizard Figure 10: Data Source Explorer Ad-Hoc Profile Figure 11: Values analysis View Figure 12: Statistics analysis View Figure 13: Frequency analysis View Figure 14: Frequency with Statistics Graph View Figure 15: Histogram analysis View Figure 16: Histogram with Overlay Graph View Figure 17: Histogram with Statistics Graph View Figure 18: Text Field Analyzer analysis View Figure 19: Text Field Analyzer SQL Plug-in for Eclipse User Guide vii

8 List of Figures viii Plug-in for Eclipse User Guide

9 List of Tables List of Tables Table 1: Teradata Profiler Plug-in for Eclipse 15.0 Dependent Products Table 2: Loggin when: Table 3: Results Returned by Analysis Table 4: Results Returned by Statistics Table 5: Results Returned by Frequency Table 6: Results Returned by Histogram Table 7: Results Returned by Analysis Plug-in for Eclipse User Guide ix

10 List of Tables x Plug-in for Eclipse User Guide

11 Chapter 1: Dependencies Software Dependencies CHAPTER 1 Dependencies What s In This Chapter 1 Software Dependencies on page 1 2 Database Requirements on page 2 Software Dependencies The table below lists dependent software products for Teradata Profiler Plug-in for Eclipse These products should be installed prior to installing the Teradata Profiler Plug-in for Eclipse. The following list describes acceptable versions of Eclipse (either may be used): Eclipse IDE for Java EE Developers (Indigo, 3.7) Eclipse IDE for Java EE Developers (Juno, 4.2 SP2) Eclipse IDE for Java EE Developers (Kepler, 4.3) Table 1: Teradata Profiler Plug-in for Eclipse 15.0 Dependent Products Dependent Product Java Runtime Environment (JRE 6 or higher) Eclipse IDE for Java EE Developers (Indigo) Eclipse IDE for Java EE Developers (Juno) Eclipse IDE for Java EE Developers (Kepler) Teradata Plug-in for Eclipse 14.2 or higher Download Location Search for the Indigo release download Search for the Juno release download Search for the Juno release download Plug-in for Eclipse User Guide 1

12 Chapter 1: Dependencies Database Requirements Database Requirements The Teradata Profiler Plug-in for Eclipse depends on the Teradata Database. The following are the Teradata Database requirements: Teradata Database 14.0, or A Teradata User and Metadata database (can be the same) A minimum of 10MB of permanent space in the Metadata database The following permissions are required for the Teradata user: GRANT EXECUTE PROCEDURE ON sqlj TO [user]; GRANT CREATE EXTERNAL PROCEDURE ON [metadata_db] TO [user] WITH GRANT OPTION; GRANT EXECUTE PROCEDURE ON [metadata_db] TO [user]; 2 Plug-in for Eclipse User Guide

13 Chapter 2: Installing Teradata Profiler Plug-in for Eclipse Install Teradata Profiler Plug-in for Eclipse CHAPTER 2 Installing Teradata Profiler Plug-in for Eclipse What s In This Chapter 1 Install Teradata Profiler Plug-in for Eclipse on page 3 2 Profiler Logging on page 3 Install Teradata Profiler Plug-in for Eclipse Once the dependent software has been downloaded, Teradata Profiler Plug-in for Eclipse can be installed and configured. Follow this procedure to install Teradata Profiler Plug-in for Eclipse 15.0 using the Eclipse Update Site install procedure. To install Teradata Profiler Plug-in for Eclipse 15.0: 1 From the Eclipse main menu, click Help and then click Install New Software. 2 Click the Add... button. 3 Type Profiler Local Install in the Name field and click on the Local button to select the ProfilerLocalInstall folder from your installation media as the Location. 4 Click OK to add the site. 5 Select the Teradata Profiler Plug-in for Eclipse component, and then click Next. 6 When the Feature License Agreement appears, select I accept the terms in the license agreement, and then click Next. A list of features appears with the feature version, size, and Eclipse install location. 7 Verify that the information is correct, and then click Finish to continue the installation. 8 A warning message appears regarding unsigned content. Click OK to continue. 9 When a message appears asking if you want to restart Eclipse, click Yes. Profiler Logging The Profiler helps the users by storing important information into log files for error detail or informational purposes. It creates 2 separate files for each case: profiler_error.log and profiler_info.html. Both of them are created on the user workspace ".metadata/twm/" directory. The "TWM/" folder is created if it does not exist. Plug-in for Eclipse User Guide 3

14 Chapter 2: Installing Teradata Profiler Plug-in for Eclipse Profiler Logging Table 2: Loggin when: Type Operations INFO CRUD operation on a Folder, Project or Analysis. Execution or Saving of Analysis (Values, Stats, etc). ERROR CRUD operation fails on a Folder, Project or Analysis. Execution or Saving fails of Analysis (Values, Stats, etc). Drillthrough fails. Metadata creation or libraries update fails. 4 Plug-in for Eclipse User Guide

15 Overview CHAPTER 3 Using Teradata Profiler Plug-in for Eclipse What s In This Chapter 1 Overview on page 5 2 Project Explorer on page 6 3 Folders on page 12 4 Projects on page 13 5 on page 15 Overview The descriptive statistics available with Teradata Profiler plug-in provide a variety of functions to statistically analyze and explore data tables in a Teradata database. Such descriptive statistical analysis is valuable for several reasons. First, it can provide business insight in its own right. Second, it uncovers data quality issues which, if not corrected or compensated for, would jeopardize the accuracy of any models that are based on the data. Next, it isolates the data that should be used in building analytic models. For example, outlying values should sometimes be excluded from a model; in other cases, these values might be required to solve a particular business problem. Further, some processes used in analytic modeling may require a certain type of distribution of data. The Profiler s descriptive statistical analyses can determine the suitability of various data elements for model input and can suggest transformations that may be required for these data elements. In the Profiler s Descriptive Statistics analyses, NULL values are handled through the generated SQL s aggregate functions. This SQL ignores the NULL value and adjusts the number of observations in its calculation. This effectively provides a listwise deletion of NULL values. The following are the descriptive statistical functions currently available in Teradata Profiler: Values analysis Count the number of values of various kinds for a given column or columns, including: a b c d Number of Rows Rows with Non-NULL Values Rows with NULL Values Unique Values Plug-in for Eclipse User Guide 5

16 Project Explorer e Rows with Value 0 f g Rows with a Positive Value Rows with a Negative Value h Rows Containing Blank Values Statistical Analysis Determine any of the following descriptive statistics for numeric column(s): a b c d e f g h i j k l Minimum Value Maximum Value Mean Value Standard Deviation Skewness Kurtosis Standard Mean Error Coefficient of Variance Variance Sum Uncorrected Sums of squares Corrected Sums of squares m Values Count Frequency Compute frequency of column values. Optionally, compute frequency of values for columns in a single column list, and optionally generate simple statistics for any other column within a table. Histogram Determine the distribution of a numeric column(s) giving counts with optional overlay counts and statistics. Project Explorer The Profile Project Explorer allows you to work with and manage Profile Projects. Make sure that the Eclipse Profile Project Explorer Perspective is opened. Eclipse Perspectives provide an alternative presentation of the data, as well as an additional way to navigate information in the Eclipse workbench. The Profile Project Explorer Perspective provides a tree view of the database connections, folders, projects and analyses for running analyses and viewing results. To open the Profile Project Explorer Perspective, click the Window > Open Perspective > Other... option in the Eclipse menu. Double click on the Data Profiling item. You will notice the Profile Project Explorer tab in the workspace. The Project Explorer window provides the following functions: 6 Plug-in for Eclipse User Guide

17 Project Explorer Add a Database Connection Add a Database Connection on page 7 Create the Metadata Database on page 8 Configure the Metadata Database on page 10 Connect to a Database on page 11 Managing Folders, Projects and on page 12 A Profiler connection to a Teradata database is needed to start profiling data. To add a connection from the Data Source Explorer: 1 Select a Database Connection within the Data Source Explorer. 2 Right click on the selected database. 3 Select the Profiler > Add Connection to Profiler menu item. 4 The selected database connection will be added to the Profile Project Explorer window in Teradata Profiler. Figure 1: Data Source Explorer Add Connection to Profiler menu To add a connection from the Profiler Project Explorer: 1 Click on the Connection icon within the menu bar of the Profiler Project Explorer window. 2 Select the database connection from the Connection popup window. (The list of database connections comes from the Data Source Explorer. If there are no connections defined in the Data Source Explorer, you will be prompted to add a Connection.) 3 Click on the Add button. Plug-in for Eclipse User Guide 7

18 Project Explorer 4 The selected database connection will be added to the Profile Project Explorer window in Teradata Profiler. Figure 2: Profiler Project Explorer Add Connection To Delete a Connection from the Profile Project Explorer: 1 Select a Database Connection within the Profile Project Explorer. 2 Right click on the selected database. 3 Select the Delete menu item. 4 Click on the Yes button. Create the Metadata Database A Profiler connection requires a metadata database in order to store the Project and Analysis information and to install the Procedure Library. You must create the metadata database tables for each connection the first time you connect. If you create the metadata database tables again for a connection, the old metadata database information (including all projects and analyses) will be overwritten and lost. However, you will be warned if this is attempted. To create the Metadata Database tables for a Connection from the Data Source Explorer: 1 Select a Database Connection within the Data Source Explorer. 2 Right click on the selected database. 3 Select the Profiler > Create Metadata Tables menu item. 4 Enter the name for the Metadata Database in the popup window (it must have all of the requirements listed in Chapter 1: Dependencies ). 5 Click on the OK button. 6 If there are no issues, the Metadata Database tables will be created and the executable Procedure Library will be installed in the selected database connection. 8 Plug-in for Eclipse User Guide

19 Project Explorer 7 The Profile Project Explorer window in Teradata Profiler will be opened. Figure 3: Data Source Explorer Create Metadata Database menu To create the Metadata Database tables for a Connection from the Profile Project Explorer: 1 Select a Database Connection within the Profile Project Explorer. 2 Right click on the selected database. 3 Select the Create Metadata Tables menu item. 4 Enter the name for the Metadata Database in the popup window (you must have all of the requirements listed in Chapter 1: Dependencies on the database entered). 5 Click on the OK button. 6 If there are no issues, the Metadata Database tables will be created and the Procedure Library installed in the selected database connection. Figure 4: Profile Project/Analysis Explorer Create Metadata Database menu Plug-in for Eclipse User Guide 9

20 Project Explorer Configure the Metadata Database If a Teradata Profiler Metadata Database already exists for a Connection, but has not been configured for this instance of the Teradata Profiler Plug-in for Eclipse, then you can configure the Metadata Database for that connection without recreating the Metadata Database and destroying the contents. To configure the Metadata Database tables for a Connection from the Data Source Explorer: 1 Select a Database Connection within the Data Source Explorer. 2 Right click on the selected database. 3 Select the Profiler > Configure Metadata Database menu item. 4 Enter the name for the Metadata Database in the popup window. 5 Click on the OK button. 6 The Metadata Database will be configured for the selected database connection. Figure 5: Data Source Explorer Configure Metadata Database menu To configure the Metadata Database tables for a Connection from the Profile Project Explorer: 1 Select a Database Connection within the Profile Project Explorer. 2 Right click on the selected database. 3 Select the Configure Metadata Database menu item. 4 Enter the name for the Metadata Database in the popup window. 5 Click on the OK button. 6 The Metadata Database will be configured for the selected database connection. 10 Plug-in for Eclipse User Guide

21 Project Explorer Figure 6: Profile Project/Analysis Explorer Configure Metadata Database menu Update Profiler Plug-in for Eclipse Connect to a Database If an update becomes available to the Profiler Plug-in you can update it directly from Eclipse. You will either update from a URL such as the Teradata Download site or from a local directory if the files are downloaded to your system. To perform an update to the Profiler Plug-in for Eclipse: 1 From the Eclipse main menu, click Help and then click About Eclipse. 2 Click the Installation Details button. 3 Select the Teradata Profiler from the Installed Software tab. 4 Click on the Update button. 5 Follow the prompts answering the questions in the same manner as for the initial installation. 6 When a message appears asking if you want to restart Eclipse, click Yes. 7 Upon restart of Eclipse, you may be prompted to Add a Connection. Select a connection to Add or just Close the dialog. 8 After Eclipse restarts you will need to Configure the Metadata Database again. This will only point the Profiler Plug-in to the correct Metadata Database (not recreate the tables). Be sure to keep track of which database you use for each connection. 9 Once a Connection has been made in the Profile Project Explorer, right click on the Connection and choose the Update Libraries menu item. This will update the Procedure Library with any new Procedures. In order to profile the data in a database you must first connect to the database. To connect to a database, double click on the Database Connection in the Profile Project Explorer or right click on the desired Database Connection in either the Data Source Explorer or the Profile Project Explorer and select Connect. If you have not saved your password with the connection, Plug-in for Eclipse User Guide 11

22 Managing Folders, Projects and you will be prompted to enter the password. Note that the user name appears at the top of the prompt popup for the password. To Disconnect from a Connection, right click on the Database Connection in the Profile Project Explorer and select Disconnect. Managing Folders, Projects and Within the Profile Project Explorer you can create and manage Profile Folders, Projects and. The Explorer s base level contains the Connections you have created to databases. Within each connection Folders, Projects and can be created in a hierarchical manner. This hierarchy of Folders, Projects and is saved in the Connections Metadata Database Tables and thus is not visible until you connect to the database. Folders Folders allow you to group and manage your Projects and within a Connection. You can create Folders under a Connection or under other Folders. To add a Folder: 1 Right click on a Connection or Folder in the Profile Project Explorer. 2 Select Add Folder from the popup menu. 3 Enter the desired Folder name in the popup window. 4 Click on OK. Figure 7: Add New Folder window To delete a Folder: 1 Right click on a Folder in the Profile Project Explorer. 2 Select Delete from the popup menu. 3 Click on Yes. To change the name of a Folder: 12 Plug-in for Eclipse User Guide

23 Projects 1 Right click on a Folder in the Profile Project Explorer. 2 Select Modify from the popup menu. 3 Enter the desired Folder name in the popup window. 4 Click on OK. To add a Project or Analysis to a Folder: 1 Right click on a Folder in the Profile Project Explorer. 2 Select Add Project or Add Analysis from the popup menu. 3 Fill in the details for the Project or Analysis (see Project and Analysis information below). 4 Click on Finish. To move a Folder or Project into a Folder: 1 Select the object to move. 2 Drag and drop the object onto the Folder you want to move into. 3 All objects below the selected object will be moved. Projects Projects allow you to group and manage your within a Connection. You can create Projects under a Connection or under Folders. must be created in Projects. To add a Project: 1 Right click on a Connection or Folder in the Profile Project Explorer. 2 Select Add Project from the popup menu. 3 Enter the desired Project name in the popup window. 4 Optionally add a description for the Project. 5 Click on Finish. Projects can also be added from the Add Analysis wizard when adding an analysis from a Connection or Folder. Plug-in for Eclipse User Guide 13

24 Projects Figure 8: Add New Project window To delete a Project: 1 Right click on a Project in the Profile Project Explorer. 2 Select Delete from the popup menu. 3 Click on Yes. To change the name of a Project: 1 Right click on a Project in the Profile Project Explorer. 2 Select Modify from the popup menu. 3 Enter the desired Project name in the popup window. 4 Optionally you may also modify the Description. 5 Click on Finish. To add an Analysis to a Project: 1 Right click on a Project in the Profile Project Explorer. 2 Select Add Analysis from the popup menu. 3 Fill in the details for the Analysis (see Analysis information below). 4 Click on Finish. To move a Project to a Folder: 14 Plug-in for Eclipse User Guide

25 1 Select the Project to move in the Profile Project Explorer. 2 Drag and drop the Project onto the Folder you want to move into. The descriptive statistical analyses available with Teradata Profiler provide a variety of functions to analyze and explore a Teradata database. It should be noted that in the case of the Descriptive Statistics, NULL values are handled through the generated SQL s aggregate functions. In this case, SQL ignores the NULL value and adjusts the number of observations in its calculation. This effectively provides a listwise deletion of NULL values. must be contained in a Project in Profiler. You can add an Analysis directly in a Project or you can add an Analysis at the Connection or Folder level. However, if you add the analysis at the Connection or Folder level you will be required to first select or add a Project within the Analysis wizard. To add an Analysis from a Connection: 1 Right click on a Connection in the Profile Project Explorer. 2 Select Add Analysis from the popup menu. 3 Optionally select the Folder in which the Analysis will be created. 4 Click on Next. 5 Select the Project in which the Analysis will be created or add a new Project. 6 Click on Next. 7 Select the type of Analysis to create (Values, Statistics, Frequency or Histogram). 8 Enter the desired Analysis name. 9 Optionally enter a Description for the Analysis. 10 Click on Finish. Plug-in for Eclipse User Guide 15

26 Figure 9: Add New Analysis wizard To add an Analysis from the Data Source Explorer: 1 Right click on a Table, Column or multiple selections of Columns. 2 Select Profiler > Values (Statistics/Frequency/Histogram) Ad-Hoc Profile from the popup menu. 3 The analysis will automatically run and open in the Profiler perspective. 4 Optionally modify and/or save the analysis in Profiler (the analysis is ad-hoc and is not automatically saved in the Profile Project Explorer). 16 Plug-in for Eclipse User Guide

27 Figure 10: Data Source Explorer Ad-Hoc Profile To delete an Analysis: 1 Right click on an Analysis in the Profile Project Explorer. 2 Select Delete from the popup menu. 3 Click on Yes. To change the name of an Analysis: 1 Right click on an Analysis in the Profile Project Explorer. 2 Select Modify from the popup menu. 3 Enter the desired Name in the Analysis window. 4 Optionally modify the Description. 5 Click on Finish. The following are the descriptive statistical functions currently available in Teradata Profiler. Values The following topics discuss the Values analysis. Values Overview on page 18 Values Input Options on page 18 Values Results on page 20 Plug-in for Eclipse User Guide 17

28 Values Graph on page 21 Values Overview Values analysis is often useful as the first type of analysis to perform on data which is relatively unknown to the analyst. It helps determine the nature and overall quality of the data. For example, this type of analysis reports whether the data is categorical or continuously numeric, how many null values it contains, etc. Values analysis can readily be applied to any type of character or numeric data, even date fields. Given a table name and the name of a column, the Values analysis provides a count of the number of rows, rows with non-null values, rows with null values, rows with value 0, rows with a positive value, rows with a negative value, and the number of rows containing blanks in the given column. Note that for a column of nonnumeric type, the zero, positive and negative counts will always be zero (for example, 000 is not counted as 0). The Values analysis is parameterized by specifying the table and column(s) to analyze, the optional Group By column, as well as specifying the Output or Expert Options. See on page 15 for instructions on how to add a Values analysis. Values Input Options To open a Values analysis: 1 Right click on a Values analysis in the Profile Project Explorer. 2 Select Open from the popup menu. 3 The selected Values analysis will open in the Analysis View. 18 Plug-in for Eclipse User Guide

29 Figure 11: Values analysis View To configure the data Columns for a Values analysis: 1 Select a database from the database dropdown list. 2 Select a table from the table dropdown list. 3 Select the desired columns to profile from the Available Columns list (use the CTRL key to select multiple columns). To select optional Group By Columns: 1 Click on the Group By (Optional) tab to view the available Group By columns. 2 Select the desired columns to Group By from the list (use the CTRL key to select multiple columns). Note: If 2 or more Group By Columns are selected, no result graphs will be available. To configure the Expert Options for a Values analysis: 1 Click on the Expert Options icon in the Analysis Menu Bar. 2 Enter an optional Where Clause in the text box to restrict rows selected by the analysis. 3 Click on the OK button to apply the Where Clause to the Analysis. To configure the Output Options for a Values analysis: Plug-in for Eclipse User Guide 19

30 1 Click on the Output Options icon in the Analysis Menu Bar. 2 If you want to save the results of the analysis to the Teradata database, select the Store Analysis in Database check box. 3 Enter the Database and Table name of the results table to be created. 4 Click on the OK button. To run a Values analysis: 1 Click on the Run icon in the Analysis Menu Bar. 2 The results will appear in the Results View. To load the saved results of a Values analysis: 1 Click on the Load icon in the Analysis Menu Bar. 2 The saved results of the Analysis will appear in the Results View. Note: The Load icon will only be enabled if the analysis was executed after it was saved, in order to ensure that the results match the analysis. To save a Values analysis: 1 Click on the Save icon in the Analysis Menu Bar. 2 The Analysis definition will be saved to the Metadata Database for this Connection. Values Results Results data, if any, is displayed in a data grid in the Results View Tab. The following is a description of the results returned by the analysis, depending on the options selected. If an output table is created, the columns in bold below will comprise the Unique Primary Index (UPI) of the output table. Table 3: Results Returned by Analysis Name Type Definition COLUMN Group By Column Type Column(s) will be created only if the group parameter is specified. Multiple columns, if specified, will be created or displayed as the column name. These column(s) will contain the unique values that the group by column takes on. xdb VARCHAR (128) Database that the variable for the values operation resides in, as specified by the database parameter. xtbl VARCHAR (128) Table that the variable for the values operation resides in, as specified by the table parameter. xcol VARCHAR (128) Variable that the values operation will be run against, as specified by the column parameter. xtype VARCHAR (50) The data type of this variable. xcnt FLOAT The total number of occurrences of this variable. 20 Plug-in for Eclipse User Guide

31 Table 3: Results Returned by Analysis Name Type Definition xnull FLOAT Total number of rows where this variable takes on a null value. xunique FLOAT Total number of rows where this variable takes on a unique value. xblank FLOAT Total number of rows where this variable is blank. xzero FLOAT Total number of rows where this variable is equal to 0. xpos FLOAT Total number of rows where this variable has a positive value. xneg FLOAT Total number of rows where this variable has a negative value. The results data displayed in the data grid can be exported to Excel by clicking on the Export to Excel icon on the Results View menu bar. Values Graph To graph the results, select a row in the data grid in the Results View Tab and then click on the graph icon on the Results View menu bar or double click on a row in the data grid. The bar graph of the Zero, Null, Unique, Positive and Negative values results for the selected column will be displayed in the Graph View Tab. Placing the mouse over a graph plot will display the values for that plot. The graph can be printed by clicking on the print icon in the Graph View menu bar. Values Graph Drill Through Click on a graph bar to drill down into the graph values to see source data rows. Each value selected will open the drill down data grid with all of the values for that column pertaining to that value analysis. The drill through data displayed in the Drill Through data grid can be exported to Excel by clicking on the Export to Excel icon on the Drill Through View menu bar. The number of rows returned in a Drill Through can be very large. As such, the Drill Through data grid will limit the number of rows returned per page of results. The Drill Through View has the following controls to work with the paging of the data results: Page Row Limit ( ) select how many rows are displayed on a page. Jump to First Page ( ) Return to the first page of results. Go to the Previous Page ( ) Move to the previous page of results. Current Page ( ) Displays the current page or jump to the page entered. Total Pages ( ) the total number of pages. Go to the Next Page ( ) Move to the next page of results. Jump to Last Page ( ) Go to the last page of results. Total row count ( ) Total number of rows in the result set. Plug-in for Eclipse User Guide 21

32 Statistics The following topics discuss the Statistics analysis. Statistics Overview on page 22 Statistics Input Options on page 22 Statistics Results on page 27 Statistics Graph on page 28 Statistics Overview When dealing with numeric data columns, it is useful to have several statistical measures to understand the characteristics and properties of each of those numeric columns, to assess their quality, and to look for outlying values and other possible anomalies. Statistical analysis provides several common and not so common statistical measures for numeric data columns. Given a table name and the name(s) of numeric column(s), Statistical analysis determines descriptive statistics for each of the column(s). Univariate Statistics provided include the following: Count Minimum Maximum Mean Standard Deviation Skewness Kurtosis Standard Error Coefficient of Variance Variance Sum Uncorrected Sums of squares Corrected Sums of squares For columns of type DATE, statistics other than count, minimum, maximum and mean are calculated by first converting to the number of days since If any non-numeric columns are selected for input, you will receive a Teradata SQL Error 2621: Bad character in format or data of TABLE.COLUMN. The Statistical analysis is parameterized by specifying the table and column(s) to analyze, options unique to the Statistical analysis, as well as specifying the desired Output and Expert Options. See on page 15 for instructions on how to add a Statistics analysis. Statistics Input Options To open a Statistics analysis: 22 Plug-in for Eclipse User Guide

33 1 Right click on a Statistics analysis in the Profile Project Explorer. 2 Select Open from the popup menu. 3 The selected Statistics analysis will open in the Analysis View. Figure 12: Statistics analysis View To configure the data Columns for a Statistics analysis: 1 Select a database from the database dropdown list. 2 Select a table from the table dropdown list. 3 Select the desired columns to profile from the Available Columns list (use the CTRL key to select multiple columns). To select optional Group By Columns: 1 Click on the Group By (Optional) tab to view the available Group By columns. 2 Select the desired columns to Group By from the list (use the CTRL key to select multiple columns). Note: If two or more Group By Columns are selected, no result graphs will be available. To configure the Expert Options for a Statistics analysis: Plug-in for Eclipse User Guide 23

34 1 Click on the Expert Options icon in the Analysis Menu Bar. 2 Enter an optional Where Clause in the text box to restrict rows selected by the Analysis. 3 Click on the OK button to apply the Where Clause to the Analysis. To configure the Output Options for a Statistics analysis: 1 Click on the Output Options icon in the Analysis Menu Bar. 2 If you want to save the results of the Analysis to the Teradata database, select the Store Analysis in Database check box. 3 Enter the Database and Table name of the results table to be created. 4 Click on the OK button. To configure the Statistic Options for a Statistics analysis: 1 Click on the Statistic Options icon in the Analysis Menu Bar. 2 The Statistic Options dialog will open. 3 Select the desired options or the Select All/Deselect All button to select/deselect all of the options. 4 Click on the OK button. The following basic univariate statistics are individually selectable for the analysis. By default, the Number of Values, Minimum Value, Maximum Value, Mean Value and Standard Deviation are selected (and must be selected for graphs to be available). The Select All / Deselect All button can be used to enable or disable all options. Number of Values (required for graphs) A count of the total number of rows (observations) with values for the specified column. Minimum Value (required for graphs) The smallest value taken on by the column: fx MIN x Maximum Value (required for graphs) The largest value taken on by the column: fx Mean Value (required for graphs) The average value of the column: fx = = MAX x 1 = -- x n i 1 Where n is the total number of rows (observations) with values for the variable x. Standard Deviation (required for graphs) The standard deviation of the variable. The standard deviation is a measure of how widely values are dispersed from the average value (the mean), and is calculated as follows, based on the entire population (by default): n 24 Plug-in for Eclipse User Guide

35 fx = n x 2 i 1 n x n n i 2 Where n is the total number of rows (observations) with values for the variable x. Skewness The skewness of the variable is a characterization of the degree of asymmetry of a distribution around its mean. Positive skewness indicates a distribution with an asymmetric tail extending toward more positive values. Negative skewness indicates a distribution with an asymmetric tail extending toward more negative values. (Note that the measures for Skewness (and Kurtosis) that are provided by Teradata Profiler are also known as the Fisher g statistics, related to the momental skewness and kurtosis [D Agostino, Belanger, and D Agostino Jr.].) Skewness is calculated as follows, based on the entire population s (by default): fx n = n 1 n 2 i 1 x x s In the above equation, n is the total number of rows (observations) with values for the variable x. Note that skewness is undefined when either the standard deviation of the variable is equal to 0, or the number of occurrences is less than 3. Kurtosis The kurtosis of the variable is a characterization of the relative peakedness or flatness of a distribution compared with the normal distribution. Positive kurtosis indicates a relatively peaked distribution. Negative kurtosis indicates a relatively flat distribution.. (Note that the measures for Kurtosis (and Skewness) that are provided by Teradata Profiler are also known as the Fisher g statistics, related to the momental skewness and kurtosis [D Agostino, Belanger, and D Agostino Jr.].) Kurtosis is calculated as follows, based on the entire population s (by default): fx n n+ 1 = n 1 n 2 n 3 In the equation above, n is the total number of rows (observations) with values for the variable x. Note that kurtosis is undefined when either the standard deviation of the variable is equal to 0, or the number of occurrences is less than 4. Standard Error The standard error of the variable, calculated as the standard deviation divided by the square root of the number of occurrences. Standard error is calculated as follows, based on the entire population (by default): n i 1 n x x n s n 2 n 3 Plug-in for Eclipse User Guide 25

36 fx x2 n x n n i 1 = n Where n is the total number of rows (observations) with values for the variable x. Coefficient of Variance The coefficient of variance of the variable, calculated as 100 times the standard deviation divided by the mean. Coefficient of variance is calculated as follows, based on the entire population (by default): fx x2 n x n n i 1 = n 1 -- x n i 1 Where n is the total number of rows (observations) with values for the variable x. Note that coefficient of variance is undefined when the average of the variable is 0. Variance The variance of the variable, calculated as the square of the standard deviation. Variance is calculated as follows, based on the entire population (by default): fx x n n x n i 1 Where n is the total number of rows (observations) with values for the variable x. Sum The sum of the variable, calculated as: Where n is the total number of occurrences of this variable. Uncorrected Sums of squares The uncorrected sums of squares of the variable, calculated as: Where n is the total number of occurrences of this variable. = fx = n i 1 n x fx = x 2 i 1 26 Plug-in for Eclipse User Guide

37 Corrected Sums of squares The corrected sums of squares of the variable, calculated as: Where n is the total number of occurrences of this variable. To run a Statistics analysis: 1 Click on the Run icon in the Analysis Menu Bar. 2 The results will appear in the Results View. To load the saved results of a Statistics analysis: 1 Click on the Load icon in the Analysis Menu Bar. 2 The saved results of the Analysis will appear in the Results View. Note: The Load icon will only be enabled if the analysis was executed after it was saved, in order to ensure that the results match the analysis. To save a Statistics analysis: n n 2 fx x 2 1 = n -- x n i 1 1 Click on the Save icon in the Analysis Menu Bar. 2 The Analysis definition will be saved to the Metadata Database for this Connection. Statistics Results Results data, if any, is displayed in a data grid in the Results View Tab. The following is a description of the results returned by the analysis, depending on the options selected. If an output table is created, the columns in bold below will comprise the Unique Primary Index (UPI) of the output table. i 1 Table 4: Results Returned by Statistics Name Type Definition COLUMN Group By Column Type Column(s) will be created only if the Group By option is specified. Multiple columns, if specified, will be created or displayed as the column name. These column(s) will contain the unique values that the group by column takes on. Xdb VARCHAR (128) Database that the variable for the values operation resides in, as specified by the database parameter. xtbl VARCHAR (128) Table that the variable for the statistics operation resides in, as specified by the table parameter. xcol VARCHAR (128) Variable that the statistics operation will be run against, as specified by the column parameter. xcnt FLOAT The total number of occurrences of this variable. Plug-in for Eclipse User Guide 27

38 Table 4: Results Returned by Statistics Name Type Definition xmin FLOAT The minimum value of the variable. Created only if the Minimum Value option is selected. xmax FLOAT The maximum value of the variable. Created only if the Maximum Value option is selected. xmean FLOAT The arithmetic mean of the variable. Created only if the Mean Value option is selected. xstd FLOAT The standard deviation of the variable. Created only if the Standard Deviation option is selected. xskew FLOAT The skewness of the variable. Created only if the Skewness option is selected. xkurt FLOAT The kurtosis of the variable. Created only if the Kurtosis option is selected. xste FLOAT The standard error of the variable. Created only if the Standard Error option is selected. xcv FLOAT The coefficient of variance of the variable. Created only if the Coefficient of Variance option is selected. xvar FLOAT The variance of the variable. Created only if the Variance option is selected. xsum FLOAT The sum of the variable. Created only if the Sum option is selected. xuss FLOAT The uncorrected sums of squares of the variable. Created only if the Uncorrected Sums of squares option is selected. xcss FLOAT The corrected sums of squares of the variable. Created only if the Corrected Sums of squares option is selected. The results data displayed in the data grid can be exported to Excel by clicking on the Export to Excel icon on the Results View menu bar. Statistics Graph To graph the results, select a row in the data grid in the Results View Tab and then click on the graph icon on the Results View menu bar or double click on a row in the data grid. The Box and Whisker Plot graph of the Max, Mean + Standard Deviation, Mean, Mean Standard Deviation and Min values results for the selected column will be displayed in the Graph View Tab. Placing the mouse over a graph plot will display the values for that plot. The graph can be printed by clicking on the print icon in the Graph View menu bar. 28 Plug-in for Eclipse User Guide

39 Statistics Graph Drill Through Click on a graph plot to drill down into the graph values to see source data rows. Each value selected will open the drill down data grid with all of the values for that column pertaining to that Statistics analysis. The drill through data displayed in the Drill Through data grid can be exported to Excel by clicking on the Export to Excel icon on the Drill Through View menu bar. The number of rows returned in a Drill Through can be very large. As such, the Drill Through data grid will limit the number of rows returned per page of results. The Drill Through View has the following controls to work with the paging of the data results: Page Row Limit ( ) select how many rows are displayed on a page. Jump to First Page ( ) Return to the first page of results. Go to the Previous Page ( ) Move to the previous page of results. Current Page ( ) Displays the current page or jump to the page entered. Total Pages ( ) the total number of pages. Go to the Next Page ( ) Move to the next page of results. Jump to Last Page ( ) Go to the last page of results. Total row count ( ) Total number of rows in the result set. Frequency The following topics discuss the Frequency analysis. Frequency Overview on page 29 Frequency Input Options on page 30 Frequency Results on page 32 Frequency Graph on page 32 Frequency Overview Frequency analysis is designed to count the occurrence of individual data values in columns that contain categorical data. It can be useful in understanding the meaning of a particular data element, and it may point out the need to recode some of the data values found, either permanently or in the course of building an analytic data set. This function can also be useful in analyzing combinations of values occurring in two or more columns. Given a table name and the name of one or more columns, the Frequency analysis calculates the number of occurrences of each value of the column or columns individually or in combination. Additionally, the percentage of rows in the selected table are listed in descending order starting with the most frequently occurring value. Optionally you may request: Whether to calculate basic statistics (min, max, mean, standard deviation) on one or more columns. Whether to include only frequency values that occur a minimum percentage of the time. A WHERE clause, reducing the rows before aggregation. Plug-in for Eclipse User Guide 29

40 A HAVING clause, reducing the answer set after aggregating, which must refer to the requested column(s) xcnt or xpct. Note: BYTE types are supported only with the Select result option and individual column requests. The Frequency analysis is parameterized by specifying the table and column(s) to analyze, options unique to the Frequency analysis, as well as specifying the desired Output and Expert Options. See on page 15 for instructions on how to add a Frequency analysis. Frequency Input Options To open a Frequency analysis: 1 Right click on a Frequency analysis in the Profile Project Explorer. 2 Select Open from the popup menu. 3 The selected Frequency analysis will open in the Analysis View. Figure 13: Frequency analysis View To configure the data Columns for a Frequency analysis: 30 Plug-in for Eclipse User Guide

41 1 Select a database from the database dropdown list. 2 Select a table from the table dropdown list. 3 Select the desired columns to profile from the Available Columns list (use the CTRL key to select multiple columns). To select optional Statistic Columns: 1 Click on the Statistic tab to view the available Statistic columns. 2 Select the desired columns to calculate basic Statistics on from the list (use the CTRL key to select multiple columns). To configure the Expert Options for a Frequency analysis: 1 Click on the Expert Options icon in the Analysis Menu Bar. 2 Enter an optional Where Clause in the text box to restrict rows selected by the Analysis. 3 Enter an optional Having Clause in the text box to filter the output. 4 Click on the OK button to apply the Where Clause to the Analysis. To configure the Output Options for a Frequency analysis: 1 Click on the Output Options icon in the Analysis Menu Bar. 2 If you want to save the results of the Analysis to the Teradata database, select the Store Analysis in Database check box. 3 Enter the Database and Table name of the results table to be created. 4 Click on the OK button. To configure the Frequency Options for a Frequency analysis: 1 Click on the Frequency Options icon in the Analysis Menu Bar. 2 The Frequency Options dialog will open. 3 If desired, select the Only include percentage values that occur a minimum percentage of the time checkbox and enter a Minimum Percentage value. If checked, this option will show frequency only for those values that occur in at least this percentage of rows. Enter an integer or decimal value between 0 and 100. (The default value is 1 for 1 percent). 4 Click on the OK button. To run a Frequency analysis: 1 Click on the Run icon in the Analysis Menu Bar. 2 The results will appear in the Results View. To load the saved results of a Frequency analysis: 1 Click on the Load icon in the Analysis Menu Bar. 2 The saved results of the Analysis will appear in the Results View. Note: The Load icon will only be enabled if the analysis was executed after it was saved, in order to ensure that the results match the analysis. Plug-in for Eclipse User Guide 31

42 To save a Frequency analysis: 1 Click on the Save icon in the Analysis Menu Bar. 2 The Analysis definition will be saved to the Metadata Database for this Connection. Frequency Results Results data, if any, is displayed in a data grid in the Results View Tab. The following is a description of the results returned by the analysis, depending on the options selected. If an output table is created, the columns in bold below will comprise the Unique Primary Index (UPI) of the output table. Table 5: Results Returned by Frequency Name Type Definition xtbl VARCHAR (128) Table that the variable for the statistics operation resides in, as specified by the table parameter. xcol VARCHAR (128) Variable that the statistics operation will be run against, as specified by the column parameter. xval VARCHAR (256) Distinct values of the frequency variable. xcnt FLOAT Count of the number of rows that the frequency variable is equal to the distinct value specified in xval. xpct FLOAT Percentage of the total records where the frequency variable is equal to the distinct value specified in xval. xmin_col FLOAT If the Statistic Columns option is selected, then this column is created and represents the minimum value of the column specified for the given frequency value (xval). xmax_col FLOAT If the Statistic Columns option is selected, then this column is created and represents the maximum value of the column specified for the given frequency value (xval). xmean_col FLOAT If the Statistic Columns option is selected, then this column is created and represents the mean value of the column specified for the given frequency value (xval). xstd_col FLOAT If the Statistic Columns option is selected, then this column is created and represents the standard deviation of the column specified for the given frequency value (xval). The results data displayed in the data grid can be exported to Excel by clicking on the Export to Excel icon on the Results View menu bar. Frequency Graph To graph the results, select a row in the data grid in the Results View Tab and then click on the graph icon on the Results View menu bar or double click on a row in the data grid. The 32 Plug-in for Eclipse User Guide

43 Frequency Graph Options dialog box will open. The Frequency Graph has the following options available: Show Frequency Graph Radio button to show only the Frequency counts graph for the selected column values. Show Frequency and Stats Graph Radio button to show both the Frequency counts and Statistics for the selected column values on the graph. Select Frequency Column Drop down list containing the selected columns to analyze. Select Statistic Column Drop down list containing the selected columns from the Statistic (Optional) tab. Once the options are selected and the OK button clicked, the Frequency counts graph and, if selected, the Box and Whisker Plot graph of the Statistics (Max, Mean + Standard Deviation, Mean, Mean Standard Deviation and Min values) results for the selected column values will be displayed in the Graph View Tab. Placing the mouse over a graph plot will display the values for that plot. Figure 14: Frequency with Statistics Graph View The graph can be printed by clicking on the print icon in the Graph View menu bar. The options for the graph can be changed by clicking on the options icon in the Graph View menu bar. Plug-in for Eclipse User Guide 33

44 Frequency Graph Drill Through Click on a Frequency counts graph plot to drill down into the graph values to see source data rows. Each value selected will open the drill down data grid with all of the values for that column pertaining to that Frequency analysis. The Frequency drill through data displayed in the Drill Through data grid can be exported to Excel by clicking on the Export to Excel icon on the Drill Through View menu bar. The number of rows returned in a Drill Through can be very large. As such, the Drill Through data grid will limit the number of rows returned per page of results. The Drill Through View has the following controls to work with the paging of the data results: Page Row Limit ( ) select how many rows are displayed on a page. Jump to First Page ( ) Return to the first page of results. Go to the Previous Page ( ) Move to the previous page of results. Current Page ( ) Displays the current page or jump to the page entered. Total Pages ( ) the total number of pages. Go to the Next Page ( ) Move to the next page of results. Jump to Last Page ( ) Go to the last page of results. Total row count ( ) Total number of rows in the result set. Histogram The following topics discuss the Histogram analysis. Histogram Overview on page 34 Histogram Input Options on page 35 Histogram Results on page 38 Histogram Graph on page 39 Histogram Overview Histogram analysis is designed to study the distribution of continuous numeric or date values in a column by providing the data necessary to create a histogram graph. This type of analysis is sometimes also referred to as binning because it counts the occurrence of values in a series of numeric ranges called bins. The Histogram analysis provided in Teradata Profiler is particularly rich in functionality, providing a number of ways to define bins and allowing multi-dimensional binning, overlaying of categorical data, and the calculation of numeric statistics within bins. Given a table name, the name of one or more numeric columns, and either the desired number of equal sized data bins, the desired number of bins with a nearly equal number of values, a desired width, or the specific boundaries, the Histogram analysis separates the data to show its distributional properties. It does this by separating the data by bin number and gives counts and percentages over the requested rows. Percentages will always sum to 100%. Separate options are available to specify a number of equal sized data bins in which the analysis determines the minimum and maximum value, as well as a user specified minimum and maximum value. If the minimum and maximum are specified, all values less than the 34 Plug-in for Eclipse User Guide

45 minimum are put in to bin 0, while all values greater than the maximum are put in to bin N+1. The same is true when the boundary option is specified. The Histogram analysis optionally provides subtotals within each bin of the count, percentage within the bin and percentage overall for each value or combination of values of one or more overlaid columns. Another option is provided to collect simple statistics for a binned column or another column of numeric or date type within the table, providing the minimum, maximum, mean, and standard deviation. (When statistics are collected for a date type column, the standard deviation is given in units of days.) Part of the information returned for each bin is the beginning and ending range value delimiting the bin. In general, beginning range values are inclusive and ending range values are exclusive except for the last bin. One exception to this is that if equally populated bins are requested the ending range values are inclusive since they are actually the maximum values in the ranges. Another exception is that the beginning and ending values of bins of a date type column may be truncated to a whole date value when equal width or specified width bins are requested. When this happens to an ending range value, the value is inclusive, and when this happens to a beginning range value, the value is exclusive. Therefore, since it will probably not be obvious when truncation has occurred, beginning and ending date range values for equal width or specified width bins should be considered approximate. If multiple columns are requested, the select is repeated for each column. If the create table option is requested along with multiple columns, the create table occurs only once with the insert/select repeated. An optional WHERE clause may be used to reduce the range of bins or to reduce the rows to bin in some other way. See on page 15 for instructions on how to add a Histogram analysis. Histogram Input Options To open a Histogram analysis: 1 Right click on a Histogram analysis in the Profile Project Explorer. 2 Select Open from the popup menu. 3 The selected Histogram analysis will open in the Analysis View. Plug-in for Eclipse User Guide 35

46 Figure 15: Histogram analysis View To configure the data Columns for a Histogram analysis: 1 Select a database from the database dropdown list. 2 Select a table from the table dropdown list. 3 Select the desired columns to profile from the Available Columns list (use the CTRL key to select multiple columns). To select optional Overlay Columns: 1 Click on the Overlay tab to view the available Overlay columns. 2 Select the desired columns to subdivide each bin by (use the CTRL key to select multiple columns). An overlay column is typically a categorical variable with only a few values. If an overlay column is specified, frequencies within each bin are calculated for each value of that overlay column. A specific column can be used in either Overlay Columns or Statistic Columns, but not both. To select optional Statistic Columns: 36 Plug-in for Eclipse User Guide

47 1 Click on the Statistic tab to view the available Statistic columns. 2 Select the desired columns to calculate basic Statistics on from the list (you can use the CTRL key to select multiple columns). A specific column can be used in either Overlay Columns or Statistic Columns, but not both. To configure the Expert Options for a Histogram analysis: 1 Click on the Expert Options icon in the Analysis Menu Bar. 2 Enter an optional Where Clause in the text box to restrict rows selected by the Analysis. 3 Click on the OK button to apply the Where Clause to the Analysis. To configure the Output Options for a Histogram analysis: 1 Click on the Output Options icon in the Analysis Menu Bar. 2 If you want to save the results of the Analysis to the Teradata database, select the Store Analysis in Database check box. 3 Enter the Database and Table name of the results table to be created. 4 Click on the OK button. To configure the Histogram Options for a Histogram analysis: 1 Click on the Histogram Options icon in the Analysis Menu Bar. 2 The Histogram Options dialog will open. 3 Select the Bin Style radio button desired and enter the Bin Values for the selected columns in the table (click in the table and enter a new value). Bins Specify a number of equal sized data bins. By default 10 bins are derived for each column. Widths Specify the desired width of each bin. Quantiles Specify a number of bins to contain a nearly equal number of values. By default 10 bins are derived for each column. Boundaries Specify a list of the desired boundaries for each bin to start, with the final value indicating the end of the last bin. Note that bin 0 will be generated if necessary to contain data values less than the first boundary specified, and bin N+1 will be generated if necessary for those data values greater than the final boundary value. Bins with Boundaries Specify the number of desired equal sized data bins, along with minimum and maximum values. By default 10 bins are derived for each column. Note that bin 0 will be generated to contain data values less than the minimum specified, and bin N+1 will be generated for those data values greater than the maximum specified. 4 Click on the OK button. To run a Histogram analysis: Plug-in for Eclipse User Guide 37

48 1 Click on the Run icon in the Analysis Menu Bar. 2 The results will appear in the Results View. To load the saved results of a Histogram analysis: 1 Click on the Load icon in the Analysis Menu Bar. 2 The saved results of the Analysis will appear in the Results View. Note: The Load icon will only be enabled if the analysis was executed after it was saved, in order to ensure that the results match the analysis. To save a Histogram analysis: 1 Click on the Save icon in the Analysis Menu Bar. 2 The Analysis definition will be saved to the Metadata Database for this Connection. Histogram Results Results data, if any, is displayed in a data grid in the Results View Tab. The following is a description of the results returned by the analysis, depending on the options selected. If an output table is created, the columns in bold below will comprise the Unique Primary Index (UPI) of the output table. Table 6: Results Returned by Histogram Name Type Definition Xdb VARCHAR (128) Database that the variable for the values operation resides in, as specified by the database parameter. xtbl VARCHAR (128) Table that the variable for the statistics operation resides in, as specified by the table parameter. xcol VARCHAR (128) Variable that the statistics operation will be run against, as specified by the column parameter. xbin INTEGER An integer representing the bin number. xbeg FLOAT Value that represents the beginning boundary for the bin of the column specified. xend FLOAT Value that represents the ending boundary for the bin of the column specified. xcnt FLOAT Number of records in the bin. xpct FLOAT Percentage of total records that this bin represents. ovly_col Overlay Column Type Value of column to overlay within the bin. Only exists if Overlay Columns is specified. COL indicates the name or alias of the requested overlay column. This column will consist of unique categorical values that the bin is subdivided into. 38 Plug-in for Eclipse User Guide

49 Table 6: Results Returned by Histogram Name Type Definition xocnt FLOAT Number of records within the bin in which this overlay column value is present. Created only if Overlay Columns is specified. xobpct FLOAT Percentage of records within the bin in which this overlay column value is present. Created only if Overlay Columns is specified. xopct FLOAT Percentage of total records in which this overlay column value is present. Created only if Overlay Columns is specified. xmin_col FLOAT Minimum value of a requested stats column within the bin. Only exists if the Stats Columns option is specified. COL indicates the name or alias of the requested stats column. Multiple columns, if specified, will be created or displayed as a repeating group consisting of xmin_col, xmax_col, xmean_ COL and xstd_col for each COL name or alias (see xmax, xmean and xstd below). xmax_col FLOAT Maximum value of a requested stats column within the bin. Only exists if the Stats Columns option is specified. COL indicates the name or alias of the requested stats column. xmean_col FLOAT Mean (average) value of a requested stats column within the bin. Only exists if the Stats Columns option parameter is specified. COL indicates the name or alias of the requested overlay column. xstd_col FLOAT Standard deviation of a requested stats column within the bin. Only exists if the Stats Columns option parameter is specified. COL indicates the name or alias of the requested overlay column. The results data displayed in the data grid can be exported to Excel by clicking on the Export to Excel icon on the Results View menu bar. Histogram Graph To graph the results, select a row in the data grid in the Results View Tab and then click on the graph icon on the Results View menu bar or double click on a row in the data grid. The Histogram Graph Options dialog box will open. Select the Histogram Column from the Histogram Column drop down list and the Bins from the table to graph. The Histogram graph will display the Bin Range and Count for each selected Bin. Placing the mouse over a graph plot will display the values for that plot. If Overlay columns were selected, you will have the option to Add Overlay information to the graph and choose the Overlay Column. The Overlay option will add the frequencies within each bin that are calculated for each value of that overlay column. Plug-in for Eclipse User Guide 39

50 Figure 16: Histogram with Overlay Graph View If Statistic columns were selected, you will have the option to Add Statistic information to the graph and choose the Statistic Column. The Statistic option will add the Box and Whisker Plot graph of the Statistics (Max, Mean + Standard Deviation, Mean, Mean Standard Deviation and Min values) results for the selected column values. Figure 17: Histogram with Statistics Graph View 40 Plug-in for Eclipse User Guide

51 Text Field Analyzer The graph can be printed by clicking on the print icon in the Graph View menu bar. The options for the graph can be changed by clicking on the options icon in the Graph View menu bar. Histogram Graph Drill Through Click on a Histogram Bin Range graph plot to drill down into the graph values to see source data rows. Each value selected will open the drill down data grid with all of the values for that column pertaining to that Histogram analysis. The Histogram drill through data displayed in the Drill Through data grid can be exported to Excel by clicking on the Export to Excel icon on the Drill Through View menu bar. The number of rows returned in a Drill Through can be very large. As such, the Drill Through data grid will limit the number of rows returned per page of results. The Drill Through View has the following controls to work with the paging of the data results: Page Row Limit ( ) select how many rows are displayed on a page. Jump to First Page ( ) Return to the first page of results. Go to the Previous Page ( ) Move to the previous page of results. Current Page ( ) Displays the current page or jump to the page entered. Total Pages ( ) the total number of pages. Go to the Next Page ( ) Move to the next page of results. Jump to Last Page ( ) Go to the last page of results. Total row count ( ) Total number of rows in the result set. The following topics discuss the Text Field Analyzer analysis. Text Field Analyzer Overview on page 41 Text Field Analyzer Options on page 42 Text Field Analyzer Result on page 43 Text Field Analyzer SQL on page 44 Text Field Analyzer Overview When dealing with character data it is sometimes helpful to be able to examine this data and determine what actual data type the data could be stored in within the database. The Text Field Analyzer analysis can analyze character data and help distinguish whether the field is a numeric type, a date, a time, a timestamp, or character data. Text field analysis can readily be applied to any type of character data. Non character data types go unprocessed and are passed along to the output just as they are defined in the input table. Given a table name and the name of a column, the Text Field Analyzer analysis provides a series of tests to distinguish what the correct underlying type should be. The first test performed on the field is the MIN and the MAX test. The MIN and MAX values of a column are retrieved from the database and tested to determine what type the values are. The next test is a sample test which retrieves a small sample of data for each column and again accesses what type they should be. The final test is a test for fields that have already been determined Plug-in for Eclipse User Guide 41

52 to be numeric and it tries to classify them in a more specific category if possible. For instance, a field that is considered a FLOAT type after the first two tests might really be a DECIMAL type with 2 decimal places. In the final test, a date type will simply be validated to make sure all values in that column are truly dates. Text Field Analyzer Options To change a Text Field Analyzer analysis input options: 1 Click on the Input Options menu option. 2 The selected Text Field Analyzer analysis options will be pre-populated in the Analysis View. Figure 18: Text Field Analyzer analysis View To configure the data Columns for a Text Field Analyzer analysis: 1 Select the desired columns to profile from the Available Columns list by clicking on them to select or deselect each item. To configure the Text Field Analyzer Options for a Text Field Analyzer analysis: 1 Click on the Text Field Analyzer Options menu option in the Analysis Menu Bar. 2 Enter an optional Perform extended numeric analysis in the check box to check if the content of the column can be numeric. 42 Plug-in for Eclipse User Guide

53 3 Enter an optional Perform extended analysis of unicode character fields in the check box to check if the content is Unicode. To configure the Output Options for a Histogram analysis: 1 Click on the Output Options icon in the Analysis Menu Bar. 2 If you want to save the results of the Analysis to the Teradata database, select the Store Analysis in Database check box. 3 Enter the Database and Table name of the results table to be created. (It will appear on the created SQL) 4 Click on the OK button. To run a Text Field Analyzer analysis: 1 Click on the Run icon in the Analysis Menu Bar. 2 The results will appear in the Results View. To save a Text Field Analysis: 1 Click on the Save icon in the Analysis Menu Bar. 2 The Analysis definition will be saved to the Metadata Database for this Connection. Text Field Analyzer Result Results data, if any, is displayed in a data grid in the Results Table. The following is a description of the results returned by the analysis, depending on the options selected. Table 7: Results Returned by Analysis Name Type Definition Column Name VARCHAR (128) Variable that the text field analyzer operation will be run against, as specified by the column parameter. Initial Database Type VARCHAR (128) The type of the column in the current table. Min Max Type VARCHAR (128) Checks the min and max values of the column to see if it can be a numeric type. Sample Type VARCHAR (128) Does a random sample of the column values to see if it actually is a numeric type. Numeric Test Type VARCHAR (128) Tests all values to check if it can be converted into a numeric type. The results data displayed in the data grid can be exported to Excel by clicking on the Export to Excel icon on the Results View paging bar. Plug-in for Eclipse User Guide 43

54 Text Field Analyzer SQL To show the sql generated, click the Show SQL button on the paging bar of the Text Field Analyzer Table. Figure 19: Text Field Analyzer SQL Drill Through In each analysis, you can click the graph plot to drill down into the graph values to see source data rows. Each value selected will open the drill down data grid with all of the values for that column pertaining to that analysis. The drill through data displayed in the Drill Through data grid can be exported to Excel by clicking on the Export to Excel icon on the Drill Through View menu bar. 44 Plug-in for Eclipse User Guide