EDIT 2014 Users Manual

Transcription

1 EUROPEAN COMMISSION EUROSTAT Directorate B: Methodology; corporate statistical and IT Services Unit B-3: IT for statistical production EDIT 2014 Users Manual Date: Version: Commission européenne, 2920 Luxembourg, LUXEMBOURG Tel

2 Edit - TABLE OF CONTENTS EDIT 2014 Users Manual...1 Edit - TABLE OF CONTENTS...2 About this manual...6 Quick Start Guide...7 Chapter 1: Introduction to the Edit Data Validation System...8 Starting/Stopping the EDIT Server...9 Starting/Stopping the EDIT Client...9 Logging in Process Chapter 2: Legal Disclaimer Chapter 3: Homepage Online Help Language Interface Selection Change Password View News Run Validation Flow Chapter 4: User functionalities User Menu Presentation System User Workflow User Workflow Chapter 5: Data Management Dataset Import Process Dataset Import Process Errors Dataset Export Process Search Datasets Manage Owner Group Datasets Details Import SDMX file using SDMX import converter DB Export DB Import Import SDMX Codelist Import/Export Datasets Generate Graphics Archive/Restore Datasets Import dataset using Pre-editing Program EDIT 2014 Data Validation Users Guide Page 2/176

3 Chapter 6: Job Management Search Job Create Job View Job Results View Job Error Statistics View Job Error Reports View Detailed Statistics Report View Job Statisctics Pre-Editing Programs Selecting the Execution Queue Chapter 7: Programmer functionalities Programmer Menu Presentation Programmer Workflows Chapter 8: Formats Management Format Menu Import Format Format Details Search Format Add Format Edit Format Chapter 9: Program Management Import Program Search Programs Export Programs Export Program with Dependencies Program Editor Import EVE Program Import SDMX Data Structure Import GESMES Data Structure Import Program with Dependencies Chapter 10: Script Vertical Templates Breakdown Search Script Templates View Template Details Generate Vertical Breakdown Script Template Generation EDIT 2014 Data Validation Users Guide Page 3/176

4 Chapter 11: Templates Management Add Connection Template Search Connection Template Edit Connection Template Delete Connection Template Add Selection Template Search Selection Template Add Insertion Template Search Insertion Template Add Contact Key Contact Keys Search Add Contact Contacts Search Verify Contact Chapter 12: Automation Management Edit Definition Test Definition Search Dynamic Program Add Dynamic Program Test Dynamic Program Validation Flow Add Validation Flow Search Validation Flow Edit Validation Flow View Chapter 13: News Management Add News Publish News Unpublish News Edit News Search News Delete News View News Chapter 15: Administrator Functionalities Add Domain Search Domain EDIT 2014 Data Validation Users Guide Page 4/176

5 Edit Domain Add User Search User Edit User Delete User Add user Group Search User Group Edit User Group Delete User Group Interface Languages Add Queue Search Queues Set default queue Disclaimer Audit Search Chapter 16: Validation Flow Example Chapter 17: Advanced information on the EDIT system View Predefined Program Scripts Import Dataset CSV File Properties details Import Dataset FLR File Properties details Job Naming Rule Execution Queues EDIT Languages Interface Abbreviations and Acronyms EDIT 2014 Data Validation Users Guide Page 5/176

6 About this manual This User Manual contains all the information required to describe the functionality of the Editing Building Block, based on the User Interface of the application. Additional information regarding its Scripting Language Syntax along with detailed examples is described further within the latest Scripting Language document available. In order to install the application in a standalone mode Edit Installation Wizard has to be launched, which configures fully the application in its target environment. The EDIT application is Windows XP, Vista, 7 and 8 XP compatible. Multiple browsers are supported including IE and Firefox. Detailed installation instructions are provided with the application where applicable. EDIT 2014 Data Validation Users Guide Page 6/176

7 Quick Start Guide You can expect the application to be setup for you with some predefined formats and programs. The application can also include some datasets known as look-up tables since they are used to validate codes in the dataset you want to validate. So, if you have been provided with a preconfigured installation, this is what you can do: a. Login using the user/password provided. The application will open the Dashboard page that contains the main navigation bar. b. Select from the main page the validation flow to be executed. The Run validation flow page will be displayed, you have to provide the file that contains the data you want to validate as well as the name under which it will be saved in the system. Provide also the parameter set, if available in the validation program. Click the Start button, you will be redirected to the Validation flow progress page where you can see the progress of the validation flow: Status of the job, Started, Finished. Wait until the job is finished. You will be redirected to the Job Details page c. The Job details page is displayed where you can manage the input datasets, output datasets and error datasets. Note: This route plan assumes you have some previous knowledge of the data you want to validate and the corresponding validation program. EDIT 2014 Data Validation Users Guide Page 7/176

8 Chapter 1: Introduction to the Edit Data Validation System In this section you will gain knowledge for: System Introduction System basic overview and workflows presentation System Start-up/Stopping process System logging in/change password procedures System Introduction The Editing Build Block application is a generalized imputation system developed by EUROSTAT for cleansing data. The system intended usage is designed to be at the last stage of the processing of data, one step prior to data tabulation and data retrieval, with the aim to check and validate a number of fields. With its usage the users can extensively cover a given statistical function or method. The system is capable of being used for performing to cover at least the following needs, Metadata preparation. Review the variables in your data file and determine their valid values, labels, and measurement level. Identify combinations of variable values that are impossible but commonly miscoded. Define validation rules based on this information. Data validation. Run basic checks and checks against defined validation rules to identify invalid cases, variables, and data values. When invalid data are found, investigate and correct the cause. Model preparation. Identify potential statistical outliers that can cause problems for many predictive models. Some outliers are the result of invalid variable values that have not been identified. For more information, please check the latest EDIT Scripting Language. System Start Up/Stopping Procedure EDIT 2014 Data Validation Users Guide Page 8/176

9 Starting/Stopping the EDIT Server Note: This step applies only for the EDIT Standalone version. The EDIT system consists of two elements while in standalone mode, the User Interface and Server consisting of the engine of the system. For working with the EDIT system both have to be initialised: 1. Starting the Server 2. Starting the User Interface (Client). Note: While the Server is running you may start/stop the Client multiple times without affecting the behaviour of the application. The default EDIT Server path that the application is installed can be accessed through the Windows Start Programs EDIT and then select Start Server. (to stop it, use the option Stop Server) This will initialise a console window where the EDIT System will log all the actions taking place in the application. Alternatively, you can run the app_start.vbs script located at the installation directory, typically within the Program Files. Note: You should not close this window, as this would mean stopping the Server and the potential validation jobs running processes. Starting/Stopping the EDIT Client Note: This step applies only for the EDIT Standalone version. The default EDIT Client path that the application is installed can be accessed through the Windows Start Programs EDIT and then select Login. Note: While the sequence of starting the Client doesn t affect in any way the operation of the system, starting the Client before starting the Server will not initialise the login sequence. EDIT 2014 Data Validation Users Guide Page 9/176

10 If this happens, please reload the page after the server is started and the application client will be accessible for logging in. To stop the EDIT Client please check that the EDIT Server is firstly stopped. You then may simply close your browser, but it is always recommended to instead use the logout option of the UI for security reasons. Logging in Process Note: This step applies only for the EDIT Standalone version or for server deployments that do not use ECAS. As soon as you press the Login button for starting the Client, the default browser is started (Internet Explorer). The browser address should be To login, enter the provided User name and Password and click the Log In button. Note: The username and password are case sensitive. You may change from the default values, but please note that in the standalone mode you will not be able to retrieve your passwords if you lose them. EDIT 2014 Data Validation Users Guide Page 10/176

11 Chapter 2: Legal Disclaimer Note: This chapter applies only for the EDIT server version. Each user is presented with the legal disclaimer message at login. Reject the discalimer is closed and the user is not allowed to login. Accept the disclaimer is closed and the homepage si displayed. At next login the disclaimer will be displayed again. Accept, Don t ask again the disclaimer is closed and the homepage is displayed. This message will not appear again for this user. EDIT 2014 Data Validation Users Guide Page 11/176

12 Chapter 3: Homepage Note: This chapter applies to the EDIT server version, as well as the EDIT Standalone version. The homepage contains the menus for each user. The users have different permisions, check the next chapters for details. Online Help The application contains an online help component that gives you information about the current page. This information can be accessed on any page of the application. When clicking on the Help link in the upper right side of the screen the help page is displayed in a popup window. EDIT 2014 Data Validation Users Guide Page 12/176

13 Language Interface Selection The default language of the application is English. It can be changed from the drop-down list in the upper right corner of the interface. All languages added by the Administrator are available in the list. When switching to another language, the message is received: Clicking on Ok, the user is redirected to the Home page and the language is changed. Clicking on Cancel, the window is closed and the language is not changed. EDIT 2014 Data Validation Users Guide Page 13/176

14 Change Password To change your password you can select the Change password option in the homepage. The system will ask you to enter your old password and your new password. Note: The new password must be at least 8 characters in length. Click on Save to store the change of password. View News Note: News are available only for the EDIT server version. Clicking on button Details, the news item is displayed. EDIT 2014 Data Validation Users Guide Page 14/176

15 Run Validation Flow Validation flows can be launched from the Homepage by clicking on the execute icon. The first step of the validation flow allows you to specify the file to be validated and the name under which the data will be saved in the system. EDIT 2014 Data Validation Users Guide Page 15/176

16 Click on Start to start the validation flow. The progress is displayed. When the job is completed the user is redirected to the jobs page. When the job is completed the Job Details page is displayed. For more details about the job results please see chapter View Job Results EDIT 2014 Data Validation Users Guide Page 16/176

17 Chapter 4: User functionalities In this section you will gain knowledge for: Language Interface Selection EDIT User Menu System User Workflow Note: This chapter applies to the EDIT server version, as well as the EDIT Standalone version. User Menu Presentation EDIT is a web-based application that runs inside a web browser. This chapter describes the controls available for the User inside the web pages of the application. The navigation bar (at the top of the screen) is the same on every screen of user account. It contains links which give quick access to user functionalities. Short presentation of the Menu and submenus for User: Menu Short description Click the Home menu to access the Home Page. Contains a link to the Change Password functionality and displays all validation flows configured EDIT 2014 Data Validation Users Guide Page 17/176

18 Click the Datasets menu to Search for Datasets, Import Datasets, Import Datasets DB import datasets from external Database or monitor Import/Export Tasks. Click the Jobs menu to Search for Jobs, Create Job, Job Statistics System User Workflow User Workflow The typical EDIT workflow of the system is kept to a minimum complexity for the users.. In order to perform a validation, after logging into the system, import a dataset for validation, create the jobs using the imported datasets and check the job results. EDIT 2014 Data Validation Users Guide Page 18/176

19 The system provides the users with additional supplementary capabilities further mentioned in this guide. EDIT 2014 Data Validation Users Guide Page 19/176

20 Chapter 5: Data Management In this section you will gain knowledge for: EDIT Dataset Import/Export Process Searching for available dataset(s) Manage Owner Group Datasets Details Export dataset to external Database Import dataset from external Database Viewing available dataset(s) Deleting available dataset(s) Importing/Exporting Historical tasks Delete Imported Datasets from the EDIT System Import/Export Dataset History from the EDIT System Create graphics Archive/Restore Dataset Import Dataset with type PREEDITING Note: This chapter applies to the EDIT server version, as well as the EDIT Standalone version. Note: This Chapter applies for user role functionalities and programmer role functionalities. EDIT Dataset Import/Export Process Prior to running any Validation Jobs (or simply referred to as Jobs), you must import data into the EDIT System. To do so you must select the Simplified Import option from the DATASETS drop down option of the EDIT System Navigation Menu Options. The Import option can read from a local file according to set of specified file properties and selected file fields and will write to the EDIT System s local database according to the selected Format of the dataset. The supported input formats of the EDIT System currently are: CSV files FLR files GESMES files SDMX-XML files EDIT 2014 Data Validation Users Guide Page 20/176

21 Dataset Import Process To import a dataset please follow the following order of steps: 1. Click Browse to select you locally saved file from the Open File Dialog box. 2. Enter a name for the new dataset. 3. Select the Import template from the drop-down list. a. To add a new import template or to check the selected one expand the Advanced Configuration node. Optionally, enter the maximum number of errors for the dataset imported. The application will abort the import process if the specified maximum number of error is exceeded. The default value is 100 errors. If left blank, the application will stop the import process when at least one error occurs (the same behaviour as when entered the value 0 ). b. Select the File Type. The application will display the File Properties panel associated to the file type you have selected. The user can add templates for the File Properties field from the section below. c. Select from the list the Dataset Format to correspond the selected dataset. d. A file fields template is a predefined selection of fields you have used in a previous occasion and saved as a template. Use from File Fields the desired template to select the corresponding fields from the listed Available Fields with the format. Upon selection, the application will populate the File Fields panel with the selection defined in the selected template. Optionally, a custom list of selected fields can be used to describe the fields inside the file to be imported. e. Select the data action: Create new or Repopulate Existing Dataset. In case the Create New radio button is checked, provide the name of the new dataset mentioned at point 2 above. Otherwise, when the Repopulate Existing Dataset option is selected, the Dataset Name field provides a drop-down list with the existing datasets that are using the same format selected in the Dataset Format field. 4. Click on Import EDIT 2014 Data Validation Users Guide Page 21/176

22 5. The system will create the import task and will redirect you to the Imports and Exports page. The import task will start immediately. In the Imports and Exports page you can monitor the progress of the import task. Dataset Import Process Errors In some cases the importing task generates error messages. EDIT 2014 Data Validation Users Guide Page 22/176

23 In the Actions area, the User can download the log which contains the error messages during import. There are different error messages depending on the produced error type. -Invalid Field Length: Error on line <line>; Invalid length for field <field_name>, value <value>, expected <expected>, actual length <Actual Length>. -Invalid Field Type: Error on line <line>; Invalid type for field <Field>, value <value>, expected <expected>. -Mandatory Field is missing: Error on line <line>: Non value for mandatory field <field> -Invalid XML File (Specific for SDMX Import plugin): Error in parsing XML file, at line <Line>, column <Column> The dataset import mechanism has a tolerant behavior; when too many or not enough values are available in the file in accordance with the list of fields selected for import, the importer will discard extra values or fill missing fields with empty values (if they are not mandatory). When importing a CSV file, the header can be used to map values in the file to the fields selected for import. The mapping is done using the name of the fields, which must match the names in the header. Any extra fields are discarded and the ordering of the values is done based on the mapping. For details please see section CSV File Properties. Dataset Export Process In the Dataset List panel accessible from the Search Datasets option, you can select the dataset you wish to export click the Export button. The EDIT System will access the Dataset Export page. Please follow the process described below for determining the export options for the selected dataset. 1. Select the File Type The application will display the File Properties panel associated to the file type you have selected. For CSV files, the file properties are: skip header lines, field delimiter, text qualifier, decimal point character and thousand delimiters. For FLR files, the file properties are: skip header lines, decimal point character and thousand delimiter. EDIT 2014 Data Validation Users Guide Page 23/176

24 2. In the File Properties panel, select the file properties. A file properties template is a predefined set of file properties you have used in a previous occasion and saved as a template. Upon selection, the application will populate the File Properties panel with the values predefined by the selected template. 3. You have now the option to update your File Properties list (the same options from the Import Dataset) 4. In the Select Fields panel, select the fields you wish to export. Choose the predefined value to select the certain fields from the listed Available Fields. A file fields template is a predefined selection of fields to be exported that you have used in a previous occasion and saved as a template. Upon selection, the application will populate the Select Fields panel with the values predefined by the selected template. 5. Optionally, in the Sort Data panel, select a field (only the ones added in the Selected Fields section are available) for sorting the data to be exported. The sorting criteria is: Ascending and Descending. EDIT 2014 Data Validation Users Guide Page 24/176

25 6. Click Export. The system will create the export task and will redirect you to the Import/Export Tasks page. The export task will start immediately. In the Import/Export Tasks page you can monitor the progress of the export task and download the generated data file after the task is finished. the EDIT EDIT 2014 Data Validation Users Guide Page 25/176

26 Search Datasets To search for all available imported datasets select from the navigation bar Dataset Search Datasets and the Dataset Search page will be displayed. In the Dataset List panel, the following actions can be performed: Action buttons - View Details - Export Description View the records of the specific dataset Export the specific dataset - Graphics Generate Graphics - Db Export - Delete - Manage Owner Group / - Archive / Restore - Edit Metadata Export the dataset to an external database Delete the specific dataset Manage Owner Group for the specific dataset Archives or restores the corresponding dataset Opens the metadata editor page for the specific dataset - Trigger Auto-Process Selects the dataset for auto-processing, available only if an autodomain is configured. Note: The Edit Metadata and Trigger Auto-Process buttons are available only when an autodomain is defined. In other cases they are hidden from the interface. You may: EDIT 2014 Data Validation Users Guide Page 26/176

27 1. Enter the search criteria by using any combination of the fields. The identification of the dataset: Dataset Name. The format used for it: Dataset Format. The user which uploaded the dataset: User drop-down list containing all the users present in the domain. The period of creation: Start Date and End Date (click on the field to popup a calendar control for the selection of the dates). Note: The User field is automatically filled in with the current user logged in. 2. Click the button. The system will populate the Dataset List panel with the list of matching records (all the datasets will be listed if you do not provide search criteria). Manage Owner Group In the Dataset List panel, locate the dataset you wish to set the Owner Group and click the Owner Group icon. The application will open the Owner Group page. In the Owner Group page, the application allows you to: 1. Select the group from the Owner Group list that you wish the dataset to be associated to. If the administrator has not yet associated any Owner Group to your user role(s) then the list will appear empty. 2. Click the Save button to save the change. By associating the dataset to the Owner Group, this will make the dataset accessible to the user that belongs to the selected group. 3. Click the Back button to return to the previous page. EDIT 2014 Data Validation Users Guide Page 27/176

28 Datasets Details In the Dataset List panel, you can see specific Dataset Details. Browse to the dataset you wish to view and click the Dataset Details button. The application will open the Dataset Details page. In the Dataset Details page, you may: 1. Select the fields you want to visualize. By default all the fields are displayed. Use the arrows to move the fields between the Displayed Columns and Hidden Columns lists. The application will automatically update the Dataset Content table with the visible columns you have selected (the maximum number of visible columns is 100). 2. Filter the records you want to display. EDIT 2014 Data Validation Users Guide Page 28/176

29 Two filtering types are available: Basic and Advanced. Basic Filtering Click on Add Field button to add the fields according to which the filtering will be applied. The application will open the Add Field pop-up window. Only the fields that are not already added for the filtering are displayed in the list. Enter the first letters of the field to be added and the application will display below all matching field names. By clicking on OK, the field selected is added to the basic filtering criteria. Clicking on Cancel, you will close the pop-up window and the field will not be added to the filtering criteria. Select the operators for each field from the drop-down list (the available values are equals, >, >=, <, <=, like and between ) and the filter value and then click on Search. The operators equals and like are available only for fields of type STRING. When selecting the operator between, two text fields are made available. Clicking on Search button validates the values selected for the filter (only numeric values entered for numeric fields). In case the filter is not valid, an error message will appear. The records are displayed according to the filter. Additionally, the filter can be saved as template for future use. The filter template is associated with the current user and the current format. When selecting an available template from the list, the saved fields are displayed, each with its operator and value. EDIT 2014 Data Validation Users Guide Page 29/176

30 Note: When the basic filter panel is expanded, if you have defined any filtering options on the last browse of the same format they will be displayed and applied automatically by the System. Advanced Filtering Filter the records using the advanced filtering. Using the available functions, operators and column names write an expression according to which the records will be displayed. Clicking on Search button validates the expression. In case the expression is not valid, an error message will appear. An example of a valid expression: upper(country)= UK AND round(price) > 5000 AND MONTH = APR. This expression will return all records which contain country UK, month APR and the PRICE greater than The filter can be saved as template for future use. The filter template is associated with the current user and the current format. When selecting an available template from the list, the Filter Expression field is automatically populated with the previously saved expression. Note1: Whenever a filtering panel is expanded the other filter is discarded. Note2: When the advanced filter panel is expanded, if you have defined any filtering options on the last browse of the same format they will be displayed and applied automatically by the System. 1. You may navigate through the record set by using the Jump to record option. By default, the first record is displayed. When specifying another number, for example 450: EDIT 2014 Data Validation Users Guide Page 30/176

31 The records displayed will be from the number specified onwards. 2. Click the button next to the record you wish to see the details. The application will open the Record Details page with the data corresponding to the selected record. 3. Optionally, click the Dataset Format, Basic Filtering and Advanced Filtering panel header bars to open or close the respective panels. Only one panel of the three can be expanded at a time. As soon as one is expanded, the other two are automatically collapsed. Import SDMX file using SDMX import converter Click on Import Dataset and select from Advanced Configuration the SDMXCONVERTER file type. Various SDMX keyset families are available depending on the installation setup. Select the xml file to be imported and click on Import button. EDIT 2014 Data Validation Users Guide Page 31/176

32 The SDMX file is imported and the contents of the dataset can be viewed. EDIT 2014 Data Validation Users Guide Page 32/176

33 DB Export In the Dataset List panel, locate a dataset you wish to export to an External Database and click to the action button - Db Export. The application will open the Dataset Export DB page. EDIT 2014 Data Validation Users Guide Page 33/176

34 Precondition: The Programmer user has prepared the Database Connection Template and Database Select Template To export a dataset to an External Database you should follow the steps: 1. Select valid "External database connection template" that match with the external database table 2. Select valid "Database Select Template" (insert statement) that match with the external database table 3. Select valid "Fields to be exported" that match with the external database table 4. Click on "Export" button.. The system will create the export task and will redirect you to the Import/Export Tasks page. The export task will start immediately. In the Import/Export Tasks page you can monitor the progress of the export task and view the dataset file that was exported to the external database. the EDIT EDIT 2014 Data Validation Users Guide Page 34/176

35 DB Import In the Datasets Menu, choose option Import Datasets DB to import a dataset from External Database. The application will open the Dataset Import DB page. Precondition: Programmer user has prepared the Database Connection Template and Database Select Template To import a dataset to an External Database you should follow the steps: 1. Select valid "External database connection template" that match with the external database table 2. Select valid "Database Select Template" (select statement) that match with the external database table 3. Select valid "Fields to be exported" that match with the external database table 4. Click on "Import" button.. The system will create the import task and will redirect you to the Import/Export Tasks page. The import task will start immediately. In the EDIT 2014 Data Validation Users Guide Page 35/176

36 Import/Export Tasks page you can monitor the progress of the import task and view the dataset file that was exported to the external database. the EDIT 5. Click on View Details to verify the imported data: Import SDMX Codelist Note: This functionality is available for programmer role. This option will enable the import of a dataset from a codelist from the SDMX Registry. The format describing the data structure will have to be available in EDIT. In the Datasets Menu, choose option Import Datasets SDMX to import a dataset from SDMX Registry. The Import Dataset SDMX dialog is displayed: EDIT 2014 Data Validation Users Guide Page 36/176

37 The programmer has to provide the Codelist name as well as the version which is use in the SDMX Registry. The following information will have to be provided: Code List Version Max number of Errors Dataset Format File Fields From SDMX Registry From SDMX Registry Defined by the Programmer Selected from the existing EDIT formats Selected from the existing file fields EDIT 2014 Data Validation Users Guide Page 37/176

38 Dataset Information To be defined by the Programmer, it will contains the Name of the dataset which will be created Confirm the Import procedure with the button. EDIT will then display the Dataset Import/Export History dialog. The dataset imported from SDMX Registry will have the following Type: Delete Imported Datasets from the Edit System In the Dataset List panel, locate the dataset you wish to delete. By click the Delete Dataset icon you can delete the specific dataset. A confirmation dialog will always appear to confirm the deletion request. 1. Click Ok to delete the dataset. 2. Click Cancel to cancel the deletion of the dataset. EDIT 2014 Data Validation Users Guide Page 38/176

39 Exception: A dataset that is involved in a Job execution can t be deleted. If you want to delete a dataset that is used as an input of a Job, first you have to delete the Job. Import/Export Datasets In the navigation bar by selecting Dataset Imports/Exports you will access the Dataset Import/Export History page. In the Dataset Import/Export History Search panel you may: 1. Enter the search criteria by using any combination of the fields: Task information: Name The status of the task: Status drop-down list with the possible values Running, Cancelled, Failed, Completed and Pending. The user which created the task: User drop-down list containing all the users present in the domain. The format used for the task: Format - drop-down list containing all the formats present in the domain. Period of creation of the task: Start Time and End Time (click on the field and the application will display a calendar to help you select the dates). Task Type drop-down list with the possible values: Import and Export. Note: The User field is automatically filled in with the current user logged in. 2. Click Search. The application will populate the Dataset Import/ Export History panel with the matching records (all the tasks will be listed if the search criteria are empty). EDIT 2014 Data Validation Users Guide Page 39/176

40 In the Dataset Import/Export History panel, the application will allow you to: Button - View Details Description View the dataset associated with the task File File - Delete Task - Download Data - Download Log Delete the task (associated datasets are not deleted only the task record). Download Data File generated by the task. for IMPORT tasks this will contain the rows that generated errors during import; for EXPORT tasks this will contain the exported data Download the error log that was generated during the execution of the task Generate Graphics In the Dataset List panel, locate a dataset and click the Graphics button-. Note: Graphics can be generated for all the listed datasets (inputs and outputs). After clicking on the action button Graphics, will appear a Page with the selection of the fields to be included in the graphic. - the values for X axis - the values of partition splits the graphics in multiple pages are accepted) - the selected field will be displayed on the Y- AXIS (only numerical values - removes all selected fields - removes only a selected field EDIT 2014 Data Validation Users Guide Page 40/176

41 Example of the generated graphics: EDIT 2014 Data Validation Users Guide Page 41/176

42 - this button is used to save the image of each displayed graphics (png file), after clicking this button, appears: The graphics can be Saved or Open in.png format: EDIT 2014 Data Validation Users Guide Page 42/176

43 Archive/Restore Datasets Note: This functionality is available for programmer role. In the Dataset List panel, locate a dataset and click the Archive button-. After confirming the archive, the dataset is set in archiving mode: Once the action is successful the dataset is archived and the Restore button is made available: Archiving/Restoring multiple dataset is possible by selecting them and clicking on the Archive/Restore selected buttons. Note: Whenever an archived dataset is selected for an action (input for a job, view details, export..) it is automatically restored. EDIT 2014 Data Validation Users Guide Page 43/176

44 Import dataset using Pre-editing Program The import of the datasets with a Preediting program gives the possibility to record in the validation error report errors that occurred during the reading of the input file. All errors (import and validation) will be written in the error report generated by the PREEDITING Program. Note: The preediting programs present an EXTERNAL file input type See the Edit 2013 Scripting Language document for details. Click on Browse button to select the dataset located on the local hard-disk and execute the program by clicking on Execute. The user will be redirected to the Job Search page. The dataset will be generated as OUTPUT.. After the completion, the output dataset is available also in Imports/Exports page. The only option available for the task is View. Delete is not available like for the other types tasks. Only the rows validated according to the rules in the program are imported. Clicking on View Task Details, the user is redirected to the job details page (see details on chapter Job Management, section View Job Results for Preediting Program). The datasets imported with PREEDITING type can be used as inputs for different programs. EDIT 2014 Data Validation Users Guide Page 44/176

45 Chapter 6: Job Management In this section you will gain knowledge for: Searching a Job Creating a Job Saving a job Viewing Job Results View Job Statistics Browse the dataset error(s) View dataset error statistics Pre-Editing Programs Priority queues defined for jobs Note: This Chapter applies for user role functionalities and programmer role functionalities. Search Job The Edit system core functionality lies on the definition of jobs for executing specific programs containing rules for the identification erroneous and invalid lookups and data. To define and schedule a Job you can select from the navigation bar the option Jobs Create Job In the Program Search panel, the application will allow you to: 1. Enter the search criteria by using any combination of the fields: The program information: Name. The period of creation Start Date and End Date, this latter refers to the creation date of the program (click this field and the application will open a calendar to help you select the date). The user which created the program: Created By drop-down list containing all the users present in the domain. The Program Type : drop-down list containing PREEDITING and VALIDATION values 2. Click Search. The system will populate the Program List panel with the matching records (all the programs will be listed if the search criteria are empty). EDIT 2014 Data Validation Users Guide Page 45/176

46 Note: The Edit system on user account should be preloaded with a number of programs. You can at anytime search for existing Jobs in the system by selecting Jobs Search Job and Edit will retrieve for you the Search Job Panel. In the Search Job panel, you may: 1. Enter the search criteria by using any combination of the fields: The job information: Name. The user which created the job: User drop-down list containing all the users present in the domain. The status of the job: Status drop-down list with all possible values COMPLETED, FAILED, PENDING, COMPILING, RUNNING, SAVED and INTERRUPTED. Period of creation of the job: Start Time and End Time (click on the field and the application will display a calendar to help you select the dates). The name of the queue as selected by the System: Execution Queue EDIT 2014 Data Validation Users Guide Page 46/176

47 The possible statuses for a change queue request: High priority execution request, a drop-down containing statuses NO REQUEST, SUBMITTED, APPLIED, REJECTED, CANCELLED Note: The User field is automatically filled in with the current user logged in. 2. Click Search. The system will populate the Job List panel with the matching records. All the jobs will be listed if the search criteria are empty. The Job List panel is dynamic and you may: Action Button Description - View Results View results of specific job already executed. The application will display the Job Details page to display the details of the job along with some useful functions such as View and Export input/output/error datasets and error view statistics. - Delete Job Delete the job. The application will ask you whether you wish to delete the job together with the output and error datasets. - Copy Job Copy the job. The application will create a copy of the job definition together with the original parameters. The name of the new job will be the original name plus the next successive number of the copy. The description of the new job will be the original description plus the Copy of prefix. The name of the outputs will be automatically regenerated with different counters. - Cancel Job Cancel the execution of a RUNNING job. The application will stop the execution of the job and will set its status to INTERRUPTED. - Execute Job Executes a saved job. Available only for the saved jobs. Edit Program Job Execution EDIT 2014 Data Validation Users Guide Page 47/176

48 Considering you have navigated at the Job Creation panel (details on previous section) click on button Create Job. Note: Each list will contain a primary action that is executed by users most of the time when accessing the list. Primary actions will be highlighted in the interface; extra space is used to separate the primary action from the secondary actions. Icons of actions will be highlighted when the mouse is hovered over them, to make it more evident what is the action to be executed. The Create Job action will open the Job page that will allow you to enter the validation program parameters for the rules to be run to the dataset and to select the execution queue. Create Job EDIT 2014 Data Validation Users Guide Page 48/176

49 Fields Description Mandatory/ Optional Job Name Name of the program. Unique value in the application, default value can be changed. Mandatory Execution queue The name of the queue as selected by the System. When several queues are available, the <AUTO> queue automatically selects the appropriate queue. Mandatory Parameter Set Preconfigured parameter set defined in the program Mandatory Parameter Set Details Default value from the selected Parameter Set defined in the program. Mandatory Override Value inputdatasets errdsd Fields that can replace values from parameter set fields Name of the dataset to be used as input for the validation program. The datasets are set by default, and are defined in the program, and are not be changed by the users. (They can vary from program to program) Output dataset set by default. Unique value in the application, default value can be changed. This dataset will contain the error report. Optional Mandatory Mandatory In order to define and execute the Job you have to follow the specific steps: 1. In the General Information panel, enter the Name of the job and select the Execution queue. 2. In the Parameter Set panel, enter the parameters required by the program. 3. In the Dataset References panel, select the input datasets for each input dataset required by the program (the application will provides the list(s) with the available datasets accordingly to the format specified by the program). 4. In the Dataset References panel, enter the name of the output error datasets required by the program (the application generates a default name that you can overwrite). 5. Optionally, Click the Save job button. The application will save the job without executing it for a later execution. EDIT 2014 Data Validation Users Guide Page 49/176

50 6. Click the Execute button. The application will create and launch the job. The application will redirect you to the Search Job page where you can monitor the progress of the scheduled job. 7. The Job will be in PENDING then RUNNING status 8. The Job will pass in FAILED if there are errors during the execution or COMPLETED when the job finished successfully. View Job Results In the Job List page, locate the job you wish to visualize, click on the View Results button and you will access the specific Job Details page for checking all the errors discovered during the Job execution. EDIT 2014 Data Validation Users Guide Page 50/176

51 The toolbar buttons: - Export: to bulk export reports and datasets after selecting them - Clear Selection: to clear the selection of all datasets Action Button Description - View dataset View the dataset (error, input, output, valid data or invalid data). The system will open the Dataset Details page where the content of the dataset is displayed. - Export Export the dataset (error, input, output, valid data or invalid data) you wish to export. The system will create the export task and will EDIT 2014 Data Validation Users Guide Page 51/176

52 redirect you to the Imports/Exports page where you can monitor the progress of the task. -View Statistics Visualize the error statistics. The system will redirect you to the Job Statistics page where it displays the job statistics details. - View Detailed Generates a Statistics Report in.csv format. Statistics Report View Job Error Statistics The Error Statistics page will allow you to see which rules failed during the program execution and the summary of the number of failures for each failed rule. The error statistics are displayed in a table format and it consists of the following columns: RULE NAME: No of Failures Rule Message: The name of the program rule that failed. Individual rows that the error appeared through job execution. Rule s error message as defined in the program Example: In this case, on the Rule Name column the rules that failed at job execution. For lookupcountry rule there are rows from the input dataset where the error appeared. For R21 rule, there are two rows from the input dataset where the error appeared. Following, there are the rest of the rules from the program where errors appeared. On the Rule Message column, the error message is mentioned as it is defined in the program: Error: Invalid country code. The next one for R21 rule: Error: <= Following, there are the rest of the error messages that are defined in the program, each corresponding to a rule. EDIT 2014 Data Validation Users Guide Page 52/176

53 Action Button Description - Export Export the dataset error statistics. The statistic error dataset will be saved locally. From the Add ERROR_IMPACT aggregation column drop-down list an error impact can be added for export. View Job Error Reports Programs can also generate datasets by running validations or dataset operations. A program that contains a validation will produce a Standard Error Dataset(s), which contains the errors found during the execution of the program. To retrieve the specific errors of the job run you can click on View dataset at the Error Reports section. You will then be redirected to the dataset content with the detailed errors information. The error reports are displayed in a table format and it consists of the following columns: ROW_NUMBER: MESSAGE: SEVERITY: program BLOCK INDEX: EXP_INDEX: EXP_ID: EXP_NAME: PARTITION: AUXILIARY_DATA: DSI_ID: ACTIONS: The number of the row in the error log dataset Rule s error message as defined in the program Severity message (e.g. Error, Warning) as defined in the Index of the block in which the error occurs Index of the expression Index of the expression in the block Name of rule for which the error is received Values of the partition Values fields used in the expression that failed Unique ID of the dataset View details button. Example: The Row Numbers are assigned automatically when creating the error dataset. For the first row: The error message defined in the program is Invalid country code, the severity is Error, the name of the rule is convertnumericvalues. The partition on which it occurs is EDIT 2014 Data Validation Users Guide Page 53/176

54 ROW_NUMBER = 1, this is a line from the input dataset as this is a horizontal rule. The auxiliary data is TER_UNIT=xx defined in the program. Note: Technical columns are hidden by default when browsing an error report. View Detailed Statistics Report Click on - View Detailed Statistics Report on the Error Reports section. A dialog window will be provided with the error report to be downloaded or saved in *.csv format: EDIT 2014 Data Validation Users Guide Page 54/176

55 The report contains information from the previous chapter - View Error Reports with the following fields order: - EXP_NAME - MESSAGE - SEVERITY - AUXILIARY_DATA Example: EDIT 2014 Data Validation Users Guide Page 55/176

56 View Job Statisctics You can view the System Job Statistics by selecting Jobs Job Statistics. The system statistics consists of the following columns: Nbr. Of running jobs: Nbr. Of pending jobs: Nbr. Of completed jobs: Nbr. Of cancelled jobs: Nbr. Of failed jobs: Overall status: The number of running jobs The number of pending jobs The number of completed jobs The number of cancelled jobs The number of failed jobs The overall status of existing jobs The system statistics column values reflect the state of the existing jobs since the last server start-up. Whenever the server is restarted the values are set to 0 default. EDIT 2014 Data Validation Users Guide Page 56/176

57 Pre-Editing Programs The Create Job page for a pre-editing program presents a Browse button to select the local dataset to be used as input. After the selection of the file, the job can be executed or saved. The page Job Details when running a pre-editing program presents in the Output Datasets section the validated dataset which is available in the Imports/Exports page. The rows which failed to be validated according to the rules in the program are present in the Error Reports section. EDIT 2014 Data Validation Users Guide Page 57/176

58 Selecting the Execution Queue When launching a job, the priority can be defined in the Create Job page, where the user selects the desired priority from the list. The jobs are saved. The user can request the change of the priority by clicking on. Field Type Description Current Text read-only The current selected queue for job execution New Drop-down box The rest of the available queues Change Button Saves the change of the queue Cancel Button Cancels the modification and returns to the precious page. Note: The priority of a job can be changed only if the job has not started yet. EDIT 2014 Data Validation Users Guide Page 58/176

59 Chapter 7: Programmer functionalities In this section you will gain knowledge for: Edit Programmer main functionalities and capabilities Edit Programmer Menu Programmer Workflow Note: This Chapter applies for programmer role only functionalities. Main functionalities: Metadata Definition Programmer role Define the structure of the formats to be used by the Edit programs Program Definition Define the steps and rules of the Edit programs Data Import/Export Import Auxiliary Metadata needed by the Edit programs Job Execution Execute an Edit program Programmer Menus Programmer Menu Presentation EDIT is a web-based application that runs inside a web browser. This chapter describes the controls available for the User inside the web pages of the application. The navigation bar (at the top of the screen) is the same on every screen of user account. It contains links which give quick access to user functionalities. Short presentation of the Menu and submenus for Programmer: EDIT 2014 Data Validation Users Guide Page 59/176

60 Menu Short description Click on Home menu to access the Home Page. Contains a link to the Change Password functionality. Click on Administration menu. News Search opens the news items editor: add, edit, delete, publish. Click on Datasets menu that has options: Search Datasets list of imported datasets with options to delete or view details o the dataset. By default the list is empty. Import Datasets file import of a dataset with options to set File Properties, File Fields and Dataset name. Import Datasets SDMX import of a dataset from an SDMX codelist. Import Datasets DB file import of a dataset from an external database Import/Exports the list of imported or exported datasets (option to view the file imported or exported, delete the task, status of the task) EDIT 2014 Data Validation Users Guide Page 60/176

61 Click on Programs menu: Search Program page with the option to view the list of programs (imported or added using the interface). Options on the page : view program, delete program, edit program, create job, export). By default the list of programs is empty. Add Program a toolbox with multiple options to create a program using a tree structure Import Program option to import a.pdl file from the local machine. Import EVE Program option to import a program built according to the eve format. Import SDMX Dataset Structure option to import a Dataset (SDMX format). Import Gesmes Dataset Structure option to import a Dataset (Gesmes format). Import Program w/ dependencies option to import a program along with its dependencies: formats and constant defined inputs (zip file). Search Script Templates page with the list of all script templates available. Option on the page: View Script. Generate Vertical Breakdown option to generate a hierarchical breakdown dataset from a hierarchical structure dataset. Script Template Generation option to generate a program with vertical breakdowns, using a pre-defined template script. Click on Jobs menu: Search Jobs list of the executed jobs with status and options :delete job, view results, copy job. By default the list of jobs is empty Create a Job is the page of program list (page Search Program) Job Statistics the statistics over the existing jobs Job Naming Rule option to set the name of the Job. This option validates that only valid metadata tokens are used in the naming EDIT 2014 Data Validation Users Guide Page 61/176

62 definition Click on Formats menu: Search Formats list of the imported dataset format with options: delete format, view format, export format. By default the list of formats is empty. Add Formats this functionality allows the programmer to edit Formats or to Add Formats using the interface. Import Format option to import a.fdl or.csv file from the local machine. Import SDMX Dataset Structure option to import a Dataset (SDMX format). Click the Templates menu: Connection Templates Search list of the connection templates with option to delete or edit each Connection Template that was saved Add Connection Template this functionality allows the programmer to save a Connection Template Selection Templates Search list of the selection templates with option to delete or edit each Selection Template that was saved Add Selection Template this functionality allows the programmer to save a Selection Template Insertion Templates Search list of the insertion templates with option to delete or edit each Insertion Template that was saved Add Insertion Template this functionality allows the programmer to save an Insertion Template Contacts Search list of all contacts defined in the application Add Contact this functionality allows the programmer to save a contact Contact Keys Search list of all keys defined EDIT 2014 Data Validation Users Guide Page 62/176

63 in the application Add Contact Key this functionality allows the programmer to save a contact key Click the Automation menu: Edit Definition This option allows the Programmer to define an automation of the jobs execution. It will enable the definition of the folder(s) and the mask of the files to be processed. Test Definition Testing the existing automated definition Search Dynamic Program Searching the existing Dynamic Programs Add Dynamic Program Creating a Dynamic program. Dynamic program will contain the association of the files format which will be processed automatically, according to the automation definition. Test Dynamic Program Testing the dynamic Program Validation Flow Search Searching the existing Validation Flow Validation Flow Add This functionality allows the programmer to save a Validation Flow EDIT 2014 Data Validation Users Guide Page 63/176

64 Programmer Workflows A programmer has the following functionalities: - Format management (add, import, edit format) - Program management (add, import, edit program) - Templates management EDIT 2014 Data Validation Users Guide Page 64/176

65 Chapter 8: Formats Management In this section you will gain knowledge for: Search Format Import Format Add Format Edit Format Note: In this Chapter only functionalities that apply to the programmer role are described. Format Menu A format defines the structure of a dataset which in turn is similar to a table in a relational database. A dataset format is defined in a text file with extension.fdl. On Format Menu click on Search Formats page In order to import programs there is necessary to have two standard dataset formats: - errdsd.fdl - statistic.fdl By default the application imports the two formats when programmer imports the first program in the application. Import Format At the Format Menu click on Import Format page On the Metadata folder there are Format files that can be imported: - country.fdl EDIT 2014 Data Validation Users Guide Page 65/176

66 - countryiso.fdl Steps to Upload a file format: 1. Choose a file 2. Click on Upload Format Details Go to Formats -> Search Formats On the page with dataset formats click on -view details EDIT 2014 Data Validation Users Guide Page 66/176

67 The format profile and the fields will be presented. EDIT 2014 Data Validation Users Guide Page 67/176

68 The columns in the above screenshot represent the attributes of the fields that compose the format whereas the rows represent the actual field definitions. The attributes of the fields are: Order: the position of the field within the record. Name: the physical name of the field. Caption: the label to which the field relates. Default Value: the default value if the value is missing when importing the dataset. Mandatory: yes indicates the value is required otherwise the record can be imputed. Length: the size of the field. Description: the detailed description of the field. Data Type: STRING, NUMBER, DOUBLE, BOOLEAN. Note: BOOLEAN values admitted as True ( 1, true, t, y ) and as False ( 0, false, f, n ); both cases are insensitive case. At the dataset details page the value for True will be 1 and for False will be 0. Precision: the number of decimal after the decimal. Note: If we declare a DOUBLE type with precision value as 0 or without precision; it ll be like declare a NUMBER type. The ROW_NUMBER field definition is automatically generated by the system and it is reserved for identifying the physical position of the records within the dataset. EDIT 2014 Data Validation Users Guide Page 68/176

69 Search Format Go to Formats -> Search Formats The format information: Name The user which created the format: Created By drop-down list containing all the users present in the domain. Note: The Created By field is automatically filled in with the current user logged in. Click on Search button Add Format Go to Formats -> Add Format Example of a format added from the interface: EDIT 2014 Data Validation Users Guide Page 69/176

70 1. Set Name = Format1 2. Ser Description = descr format 1 3. Click on Add Field button 4. Set the following fields : - Name = Field1 - Caption = Field1 - Default Value = empty - Mandatory = true - Data Type = string - Length = 10 - Scale = 0 - Description = addedfield1 5. Click again on Add Field button and set new fields: EDIT 2014 Data Validation Users Guide Page 70/176

71 6. Click on Save button and save the new format Edit Format 1. To edit a format click again on icon Edit Note: Only the formats that don t have datasets associated can be edited. EDIT 2014 Data Validation Users Guide Page 71/176

72 2. Choose some fields to be edited : 3. Click on Save button EDIT 2014 Data Validation Users Guide Page 72/176

73 4. To save a new format click again on icon Edit and change the name of the format, then click on Save As button EDIT 2014 Data Validation Users Guide Page 73/176

74 Chapter 9: Program Management In this section you will gain knowledge for: Import Program Search Program Add Program Edit Program Note: In this Chapter functionalities that apply only to the programmer role are described. A program comprises a set of validation rules to be applied to a dataset. The syntax of a program is described in document Edit 2013 Scripting Language. Import Program Default configuration for user Programmer - there are no Programs in the Database. The pdl script saved on the local hard-disk can be imported following the steps: 1. Click on Import Program button and page will be displayed EDIT 2014 Data Validation Users Guide Page 74/176

75 From this page can be imported a.pdl file in a normal format program or simplified format program 2. Click on Browse button to select a.pdl file path 3. Click on Import button and the program will appear in the program page list. Search Programs Go to Programs Search Program After the import of a few programs the program list appears like this: EDIT 2014 Data Validation Users Guide Page 75/176

76 The following actions are available for the programs available: Action buttons - View Details - Delete - Edit Program - Create Job - Export - Export Program with Dependencies Description View the details of the Program Delete the dataset Program editor Start execution of a program Export the program Export the program with dependencies (dataset references and formats) Export Programs Go to Programs and click on Export. Click on OK. The Browse Location window appears to select the path on the local hard-drive to download the program. The program will be saved at the selected location with pdl format. Click on Cancel or to close the window and abort the export of the program. Simplified Export Program Go to Programs and click on Export. Check the Export in simplified form? box and click on OK. EDIT 2014 Data Validation Users Guide Page 76/176

77 The Browse Location window appears to select the path on the local hard-drive to download the program. The program will be saved at the selected lcoation with csv format. Click on Cancel or to close the window and abort the export of the program. Delete Program Go to Programs and click on Delete Program. The confirmation window appears. Click on OK to delete the program. Note: If the program has associated jobs, the program will not be deleted and an appropriate message will appear in the application. Click on Cancel or to close the window and abort the deletion of the program. Export Program with Dependencies In the page with the list of programs, click on Export Program with Dependencies. The file exported is a zip containing the formats, constant datasets (lookups), the property sets templates and field sets templates and an xml file with the information about the archive. EDIT 2014 Data Validation Users Guide Page 77/176

78 It can be imported in a different system using the Import Program with dependencies functionality. Program Editor Program Editor In the page with the list of programs, click on Edit Program. The Constants Editor With the Controls: Field: Properties Additional Information: Navigator Constants Link Activates the Constants Editor Panel Editor - Name Text / Text Field Field for entering the name of the constant Editor Type Text / Drop Down List Field for selecting the type of the constant; List of values: STRING, NUMBER, DOUBLE, BOOLEAN Editor Value Text / Text Field Field for entering the value of the constant Editor Delete Icon Delete the constant from the list Add Constant OK Cancel Icon Icon Icon Add a new constant in the constants table Save the current constants table Cancel current modifications EDIT 2014 Data Validation Users Guide Page 78/176

79 The Validation Step Editor With the controls: Field: Properties Additional Information: {Fields} Input Current existing input fields Condition Text Area Field for entering the condition for the execution of the step OK Cancel Icon Icon Save the step Cancel current modifications The Dataset Operation Step Editor EDIT 2014 Data Validation Users Guide Page 79/176

80 With the Controls: Field: Properties Additional Information: Name Text Field Field for entering the name of the step Expression Text Area Field for entering the expression of the step Condition Text Area Field for entering the condition for the execution of the step Editor Toolbox - Templates List of Links Dataset Operation Templates; List of values to be added to the existing list: HIERARCHICAL DATA, IMPORT FILE, IMPORT DATABASE, EXPORT FILE, EXPORT DATABASE EDIT 2014 Data Validation Users Guide Page 80/176

81 OK Cancel Icon Icon Save the step Cancel current modifications Hierarchical Rules With the Controls: Field: Properties Additional Information: Name Text Field Field for entering the name of the rule Keysets Name Keysets Name Keysets Name Keysets Delete Text / Text Field Text / Text Field Text / Text Field Icon Field for entering the name of the keyset Field for entering the name of the keyset Field for entering the name of the keyset Delete the keyset from the list EDIT 2014 Data Validation Users Guide Page 81/176

82 Add Keyset Icon Add a new keyset in the keysets table Save As Icon Save the current keysets definition for later re-use Expression Text Area Field for entering the expression of the rule Editor Toolbox Templates Editor Toolbox Keysets OK Cancel List of Links Hierarchical RuleTemplates; List of values: CONDITIONAL, CONDITIONAL AUX FIELDS, COMPUTATION THEN, COMPUTATION THEN ELSE List of Links Icon Icon Hierarchical keysets definitions saved by the user. On click the user is asked for a confirmation and the keysets list is populated. Save the rule Cancel current modifications EDIT 2014 Data Validation Users Guide Page 82/176

83 Import EVE Program Note: Before importing an EVE program, a format compatible with the data structure on which the EVE rules are applied must be defined in Edit Go to Programs Import EVE Program After the selection of the option, the Import dialog appears like this: 1. Click on Browse to select the file (EVE Program) to be imported 2. Select the format (as mentioned previously, the format should have been defined previously in Edit or Imported from an SDMX Codelist) 3. Define a name for the new program Confirm the Import using the button. Edit will open the Search Program dialog, listing the existing programs. The program generated from the EVE Import option will have: in the Description. EDIT 2014 Data Validation Users Guide Page 83/176

84 Import SDMX Data Structure Go to Programs Import SDMX Data Structure The Import dialog appears: This option will enable the import of a dataset, to create the format and the associated program directly from the SDMX Registry. To launch the import procedure, programmer will have to provide the following information s from the SDMX Registry (Dataset ID for the SDMX Dataset Name and SDMX version): The mapping of the information is hereunder: SDMX Registry Edit EDIT 2014 Data Validation Users Guide Page 84/176

85 Programmer will also have to specify the format of the dataset which will be imported. The format will be generated automatically by Edit. The name for the program which will be created is also to specify. Once defined, click on to launch the procedure. Edit will then switch to the Search Program dialog and the list of the existing programs will be displayed: The description of the programs created from the SDMX Registry import will contain the following: Import GESMES Data Structure Go to Programs Import Gesmes Data Structure The Import dialog appears: This option will enable the import of a dataset, to create the format and the associated lookups. By default, the Generate Validation Program is checked. Enter a name for the program. Note1: If the Generate Validation Program is checked the EBB Program Name is mandatory. Note2: If the Generate Validation Program is not checked the program will not be created. EDIT 2014 Data Validation Users Guide Page 85/176

86 Once defined, click on to launch the procedure. Edit will then switch to the Search Program dialog and the list of the existing programs will be displayed: The description of the programs created from the SDMX Registry import will contain the following: Import Program with Dependencies Import program with dependencies Go to Programs Import Program w/ dependencies The Import dialog appears: This option will enable the import of a zip file which contains the program, the formats, the constant input datasets, if available and the file type templates and field set templates, if available. Also an xml file with the information about the archive. Note: If the file is not valid, the system will generate an error message and the program along with its dependencies will not be imported. Once defined, click on to launch the procedure. In case the program, constant, datasets and templates don t exist in the application the following window will appear: EDIT 2014 Data Validation Users Guide Page 86/176

87 Formats list is expanded by default and clicking on collapses the current selected one. icon, expands the selected section and Edit will then switch to the Search Program dialog and the list of the existing programs will be displayed with the new program imported. In case the program, formats, constant datasets or templates already exist the following window will appear. Check the box Do not delete and rename to in order to keep the existing program. The name for the old program must be filled. In case there are different fields or different types for the fields for one format, it will be displayed in the incompatible area and will be erased when clicking on Import for the program with dependencies. EDIT 2014 Data Validation Users Guide Page 87/176

88 The Purge (formats) option erases the format and all data linked to it (programs, datasets) and uploads the new one; Overwrite (lookups and templates) option overwrites the existing lookups/templates with the new ones. Chapter 10: Script Vertical Templates Breakdown In this section you will gain knowledge for: Search Script Templates Generate Vertical Breakdown Script Template Generation Note: This Chapter describes functionalities that apply only to the programmer role. Search Script Templates Go to Programs Search Script Templates. The page is displayed: EDIT 2014 Data Validation Users Guide Page 88/176

89 The following action is available for the templates available: Action buttons - View Details Description View the details of the script template View Template Details - COMP: Computation Template which assigns the sum of children to the father. << VERTICAL TRKEY_TOTAL_COMP { } >> EXPRESSION { KEYS KEYS; TRKEYS TRKEY; VAL['TOTAL'] := sumtr('val', 'A,B,C', ','); } - COMP_NOT_MISSING: Computation Template which assigns the sum of children to the father if there is no missing child << VERTICAL TRKEY_TOTAL_COMP_NOT_MISSING { EXPRESSION { EDIT 2014 Data Validation Users Guide Page 89/176

90 KEYS KEYS; TRKEYS TRKEY; VAR BOOLEAN NOT_MISS_CHILDREN := countmissingtr('val', 'A,B,C', ',') = 0; CONDITION NOT_MISS_CHILDREN; THEN { VAL['TOTAL'] := sumtr('val', 'A,B,C', ','); } } } >> - VAL_ABS_TOL: Validation Template which has no missing elements and the value of father has a tolerance compared to the sum of children << VERTICAL TRKEY_TOTAL_init { EXPRESSION { KEYS KEYS; TRKEYS TRKEY; VAR ERR_MSG := 'TOTAL' + 'RELATION' + replacestring('a,b,c', ',', ' + '); VAR AUX_DATA_MSG := dumpvectormap('val', 'TOTAL' + ',' + 'A,B,C', ','); VAR DOUBLE VAL_CHILDREN_SUM := nvl(sumtr('val', 'A,B,C', ','), 0.0); VAR DOUBLE VAL_FATHER := nvl(val['total'], 0.0); VAR BOOLEAN NO_MISSING_ELEMENTS := countmissingtr('val', 'TOTAL' + ',' + 'A,B,C', ',') = 0; } VAR STRING RELATION := 'RELATION'; VAR DOUBLE TOL_A1 := TOL_A1; } VERTICAL TRKEY_TOTAL { EXPRESSION { KEYS KEYS; TRKEYS TRKEY; VAR DOUBLE PERC; VAR DIFF := VAL_FATHER - VAL_CHILDREN_SUM; EDIT 2014 Data Validation Users Guide Page 90/176

91 CONDITION (NOT isnull(val_father)) AND VAL_FATHER <> 0; THEN { PERC := (100 * DIFF) / VAL_FATHER; AUX_DATA_MSG := AUX_DATA_MSG + ', DIFF=' + DIFF + ', PERC=' + PERC; } CONDITION NO_MISSING_ELEMENTS AND compare(val_father, VAL_CHILDREN_SUM, RELATION, TOL_A1); } >> ERRMSG ERR_MSG SEVERITY 'ERROR' AUX_DATA_MSG; } - VAL_ABS_TOL_YEAR: Validation Template which has valid year, no missing elements and the value of father has a tolerance compared to the sum of children << VERTICAL TRKEY_TOTAL_init { EXPRESSION { KEYS KEYS; TRKEYS TRKEY; VAR ERR_MSG := 'TOTAL' + 'RELATION' + replacestring('a,b,c', ',', ' + '); VAR AUX_DATA_MSG := dumpvectormap('val', 'TOTAL' + ',' + 'A,B,C', ','); VAR NUMBER YEAR_START := YEAR_START; VAR NUMBER YEAR_END := YEAR_END; VAR DOUBLE VAL_CHILDREN_SUM := nvl(sumtr('val', 'A,B,C', ','), 0.0); VAR DOUBLE VAL_FATHER := nvl(val['total'], 0.0); VAR BOOLEAN VALID_YEAR := TRUE; VAR BOOLEAN NO_MISSING_ELEMENTS := countmissingtr('val', 'TOTAL' + ',' + 'A,B,C', ',') = 0; VAR STRING RELATION := 'RELATION'; VAR DOUBLE TOL_A1 := TOL_A1; } } VERTICAL TRKEY_TOTAL { EXPRESSION { KEYS KEYS; TRKEYS TRKEY; EDIT 2014 Data Validation Users Guide Page 91/176

92 CONDITION NOT isnull(year_start); THEN { VALID_YEAR := YEAR_START <= YEAR; } CONDITION NOT isnull(year_end); THEN { VALID_YEAR := VALID_YEAR AND YEAR <= YEAR_END; } VAR DOUBLE PERC; VAR DIFF := VAL_FATHER - VAL_CHILDREN_SUM; CONDITION VALID_YEAR AND (NOT isnull(val_father)) AND VAL_FATHER <> 0; THEN { PERC := (100 * DIFF) / VAL_FATHER; AUX_DATA_MSG := AUX_DATA_MSG + ', DIFF=' + DIFF + ', PERC=' + PERC; } CONDITION VALID_YEAR -> NO_MISSING_ELEMENTS AND compare(val_father, VAL_CHILDREN_SUM, RELATION, TOL_A1); ERRMSG ERR_MSG SEVERITY 'ERROR' AUX_DATA_MSG; } } >> - VAL_ALL_TOL: Validation Template which has no missing elements and the value of father has an absolute and relative tolerance compared to the sum of children << VERTICAL TRKEY_TOTAL_init { EXPRESSION { KEYS KEYS; TRKEYS TRKEY; VAR ERR_MSG := 'TOTAL' + 'RELATION' + replacestring('a,b,c', ',', ' + '); VAR AUX_DATA_MSG := dumpvectormap('val', 'TOTAL' + ',' + 'A,B,C', ','); VAR DOUBLE VAL_CHILDREN_SUM := nvl(sumtr('val', 'A,B,C', ','), 0.0); VAR DOUBLE VAL_FATHER := nvl(val['total'], 0.0); VAR BOOLEAN MISS_CHILDREN := countmissingtr('val', 'A,B,C', ',') > 0; VAR BOOLEAN MISS_FATHER := isnull(val['total']) ; VAR FATHER_ROW := nvl(row_number['total'], 0.0); VAR TOL_A1 := TOL_A1; VAR TOL_A2 := TOL_A2; VAR TOL_R1 := TOL_R1; VAR TOL_R2 := TOL_R2; EDIT 2014 Data Validation Users Guide Page 92/176

93 } } VERTICAL TRKEY_TOTAL_MissingTotal { EXPRESSION { KEYS KEYS; TRKEYS TRKEY; } } CONDITION NOT ( MISS_FATHER AND ( NOT MISS_CHILDREN) ); ERRMSG 'TOTAL missing ' SEVERITY 'ERROR' (FATHER_ROW); VERTICAL TRKEY_TOTAL_MissingValues { EXPRESSION { KEYS KEYS; TRKEYS TRKEY; CONDITION NOT ( MISS_FATHER AND MISS_CHILDREN ); ERRMSG 'TOTAL and some values are missing ' SEVERITY 'WARNING' AUX_DATA_MSG; } } VERTICAL TRKEY_TOTAL_MissingInfo { EXPRESSION { KEYS KEYS; TRKEYS TRKEY; VAR BOOLEAN MISS_VAL_ABS_VAR := TRUE ; VAR BOOLEAN MISS_VAL_REL_VAR := TRUE ; CONDITION ( NOT MISS_FATHER ) AND MISS_CHILDREN -> VAL_FATHER > VAL_CHILDREN_SUM + TOL_A1; ELSE{ MISS_VAL_ABS_VAR := FALSE ; } CONDITION ( NOT MISS_FATHER ) AND MISS_CHILDREN AND ( VAL_FATHER <> 0 ) -> (VAL_FATHER - VAL_CHILDREN_SUM)/VAL_FATHER > TOL_R1; ELSE{ MISS_VAL_REL_VAR := FALSE ; } CONDITION (MISS_VAL_ABS_VAR OR MISS_VAL_REL_VAR); ERRMSG ERR_MSG SEVERITY 'ERROR' AUX_DATA_MSG; } } VERTICAL TRKEY_TOTAL_AllInfo { EXPRESSION { KEYS KEYS; TRKEYS TRKEY; EDIT 2014 Data Validation Users Guide Page 93/176

94 VAR BOOLEAN ALL_INFO_ABS_VAR := TRUE ; VAR BOOLEAN ALL_INFO_REL_VAR := TRUE ; CONDITION (( NOT MISS_FATHER ) AND ( NOT MISS_CHILDREN ) ) -> compare(val_father, VAL_CHILDREN_SUM, '=', TOL_A1, TOL_A2); ELSE{ ALL_INFO_ABS_VAR := FALSE ; } CONDITION (( NOT MISS_FATHER ) AND ( NOT MISS_CHILDREN ) AND VAL_FATHER <> 0 ) -> compare(1, VAL_CHILDREN_SUM/VAL_FATHER, '=', TOL_R1, TOL_R2); ELSE{ ALL_INFO_REL_VAR := FALSE ; } CONDITION ALL_INFO_ABS_VAR OR ALL_INFO_REL_VAR; ERRMSG ERR_MSG SEVERITY 'ERROR' AUX_DATA_MSG; } } >> - VAL_ALL_TOL_YEAR: Validation Template which has valid year, no missing elements and the value of father has an absolute and relative tolerance compared to the sum of children << VERTICAL TRKEY_TOTAL_init { EXPRESSION { KEYS KEYS; TRKEYS TRKEY; VAR ERR_MSG := 'TOTAL' + 'RELATION' + replacestring('a,b,c', ',', ' + '); VAR AUX_DATA_MSG := dumpvectormap('val', 'TOTAL' + ',' + 'A,B,C', ','); VAR NUMBER YEAR_START := nvl(year_start, 0); VAR NUMBER YEAR_END := nvl(year_end, 10000); VAR DOUBLE VAL_CHILDREN_SUM := nvl(sumtr('val', 'A,B,C', ','), 0.0); VAR DOUBLE VAL_FATHER := nvl(val['total'], 0.0); VAR BOOLEAN VALID_YEAR := TRUE; VAR BOOLEAN MISS_CHILDREN := countmissingtr('val', 'A,B,C', ',') > 0; VAR BOOLEAN MISS_FATHER := isnull(val['total']) ; VAR FATHER_ROW := nvl(row_number['total'], 0.0); VAR TOL_A1 := TOL_A1; VAR TOL_A2 := TOL_A2; VAR TOL_R1 := TOL_R1; VAR TOL_R2 := TOL_R2; EDIT 2014 Data Validation Users Guide Page 94/176

95 VAR STRING RELATION := 'RELATION'; CONDITION NOT isnull(year_start); THEN { VALID_YEAR := YEAR_START <= YEAR; } CONDITION NOT isnull(year_end); THEN { VALID_YEAR := VALID_YEAR AND YEAR <= YEAR_END; } } } VERTICAL TRKEY_TOTAL_MissingTotal { EXPRESSION { KEYS KEYS; TRKEYS TRKEY; CONDITION VALID_YEAR -> NOT ( MISS_FATHER AND ( NOT MISS_CHILDREN) ); ERRMSG 'TOTAL missing ' SEVERITY 'ERROR' (FATHER_ROW); } } VERTICAL TRKEY_TOTAL_MissingValues { EXPRESSION { KEYS KEYS; TRKEYS TRKEY; CONDITION VALID_YEAR -> NOT ( MISS_FATHER AND MISS_CHILDREN ); ERRMSG 'TOTAL and some values are missing ' SEVERITY 'WARNING' AUX_DATA_MSG; } } VERTICAL TRKEY_TOTAL_MissingInfo { EXPRESSION { KEYS KEYS; TRKEYS TRKEY; VAR BOOLEAN MISS_VAL_ABS_VAR := TRUE ; VAR BOOLEAN MISS_VAL_REL_VAR := TRUE ; EDIT 2014 Data Validation Users Guide Page 95/176

96 CONDITION VALID_YEAR AND ( NOT MISS_FATHER ) AND MISS_CHILDREN -> VAL_FATHER > VAL_CHILDREN_SUM + TOL_A1; ELSE{ MISS_VAL_ABS_VAR := FALSE ; } CONDITION ( NOT MISS_FATHER ) AND MISS_CHILDREN AND ( VAL_FATHER <> 0 ) -> (VAL_FATHER - VAL_CHILDREN_SUM)/VAL_FATHER > TOL_R1; } ELSE{ MISS_VAL_REL_VAR := FALSE ; } CONDITION VALID_YEAR -> (MISS_VAL_ABS_VAR OR MISS_VAL_REL_VAR); ERRMSG ERR_MSG SEVERITY 'ERROR' AUX_DATA_MSG; } VERTICAL TRKEY_TOTAL_AllInfo { EXPRESSION { KEYS KEYS; TRKEYS TRKEY; VAR BOOLEAN ALL_INFO_ABS_VAR := TRUE ; VAR BOOLEAN ALL_INFO_REL_VAR := TRUE ; CONDITION VALID_YEAR AND (( NOT MISS_FATHER ) AND ( NOT MISS_CHILDREN ) ) -> compare(val_father, VAL_CHILDREN_SUM, RELATION, TOL_A1, TOL_A2); ELSE{ ALL_INFO_ABS_VAR := FALSE ; } CONDITION VALID_YEAR AND (( NOT MISS_FATHER ) AND ( NOT MISS_CHILDREN ) AND VAL_FATHER <> 0 ) -> compare(1, VAL_CHILDREN_SUM/VAL_FATHER, RELATION, TOL_R1, TOL_R2); } >> ELSE{ ALL_INFO_REL_VAR := FALSE ; } CONDITION VALID_YEAR -> ALL_INFO_ABS_VAR OR ALL_INFO_REL_VAR; ERRMSG ERR_MSG SEVERITY 'ERROR' AUX_DATA_MSG; } EDIT 2014 Data Validation Users Guide Page 96/176

97 - VAL_NO_TOL: Validation Template which has no missing elements and the value of the father equals the sum of children << VERTICAL TRKEY_TOTAL_init { EXPRESSION { KEYS KEYS; TRKEYS TRKEY; VAR ERR_MSG := 'TOTAL' + ' = ' + replacestring('a,b,c', ',', ' + '); VAR AUX_DATA_MSG := dumpvectormap('val', 'TOTAL' + ',' + 'A,B,C', ','); VAR DOUBLE VAL_CHILDREN_SUM := nvl(sumtr('val', 'A,B,C', ','), 0.0); VAR DOUBLE VAL_FATHER := nvl(val['total'], 0.0); VAR BOOLEAN NO_MISSING_ELEMENTS := countmissingtr('val', 'TOTAL' + ',' + 'A,B,C', ',') = 0; } } VERTICAL TRKEY_TOTAL { EXPRESSION { KEYS KEYS; TRKEYS TRKEY; CONDITION NO_MISSING_ELEMENTS -> compare(val_father, VAL_CHILDREN_SUM, '='); } ERRMSG ERR_MSG SEVERITY "ERROR" AUX_DATA_MSG; } >> - VAL_NO_TOL_YEAR: Validation Template which has valid year, no missing elements and the value of father equals the sum of children << VERTICAL TRKEY_TOTAL_init { EXPRESSION { KEYS KEYS; TRKEYS TRKEY; VAR ERR_MSG := 'TOTAL' + ' = ' + replacestring('a,b,c', ',', ' + '); VAR AUX_DATA_MSG := dumpvectormap('val', 'TOTAL' + ',' + 'A,B,C', ','); VAR NUMBER YEAR_START := YEAR_START; EDIT 2014 Data Validation Users Guide Page 97/176

98 VAR NUMBER YEAR_END := YEAR_END; VAR DOUBLE VAL_CHILDREN_SUM := nvl(sumtr('val', 'A,B,C', ','), 0.0); VAR DOUBLE VAL_FATHER := nvl(val['total'], 0.0); VAR BOOLEAN NO_MISSING_ELEMENTS := countmissingtr('val', 'TOTAL' + ',' + 'A,B,C', ',') = 0; VAR BOOLEAN VALID_YEAR := TRUE; } } VERTICAL TRKEY_TOTAL { EXPRESSION { KEYS KEYS; TRKEYS TRKEY; CONDITION NOT isnull(year_start); THEN { } VALID_YEAR := YEAR_START <= YEAR; CONDITION NOT isnull(year_end); THEN { VALID_YEAR := VALID_YEAR AND YEAR <= YEAR_END; } CONDITION VALID_YEAR AND NO_MISSING_ELEMENTS -> compare(val_father, VAL_CHILDREN_SUM, '='); ERRMSG ERR_MSG SEVERITY "ERROR" AUX_DATA_MSG; } } >> Generate Vertical Breakdown To generate the vertical breakdown, a hierarchical structure must exist in the system. The hierarchical structure must contain the following mandatory fields: - ID doesn t allow empty values - FATHER_ID allows empty values - LVL doesn t allow empty values. Note1: If the name of the columns is not the same as above, the dataset won t be recognised as hierarchical. EDIT 2014 Data Validation Users Guide Page 98/176

99 Note2: The system will automatically detect during the dataset import if a dataset can be used as a hierarchical structure or as a vertical breakdown. To generate the vertical breakdown follow the steps: Go to Programs Generate Vertical Breakdown. The page is displayed: The fields available are: Hierarchical Structure: combo-box displaying all hierarchical structure dataset available. Dataset Name: the name of the dataset to be generated. Create New: radio button to select the option to create a new dataset. Repopulate Existing Dataset: radio button to select the option to create a new dataset. When selected, the dataset to be updated can be selected from the combo-box Dataset Name field. Back: button to cancel the generation of the vertical breakdown and retunring to the previous page. Generate Vertical Breakdown: button to generate the bertical breakdown. From the combo-box Hierarchical Struture select the hierarchical dataset and fill in the name for the new dataset to be generated. Click on Generate Vertical Breakdown button. After a few seconds the message is displayed. EDIT 2014 Data Validation Users Guide Page 99/176

100 Cancel: closes the message and the pre-filled Generate Vertical Breakdown page is displayed without generating the hierarchical breakdown. Ok: redeirects the user to the dataset details page of the newely generated hierarchical breakdown. It contains the same mandatory fields except the ID and FATHER_ID fields become CHILDREN and respectively FATHER. Also the children are displayed all on the same row with the corresponding father: - FATHER doesn t allow empty values - CHILDREN allows empty values - LVL doesn t allow empty values. EDIT 2014 Data Validation Users Guide Page 100/176

101 In addition, a format is created for the dataset: <structure dataset format name>_breakdown. Script Template Generation Go to Programs Script Template Generation. The page is displayed: Field Type Action Template Combo-box The script template to be used Structure hierarchical structure / hierarchical breakdown Program New Program / Refresh Existing Program Combo-box / Combobox Text / Combo-box The hierarchical dataset to be used The program name to be generated / the program name to be updated Input Dataset Definition Combo-box The input dataset format to be defined in the porgram Keyset Text The keyset to be used for the vertical rules Transposition keys Text The transposition keys to be used for the vertical rules. Select the VAL_ALL_TOL_YEAR script template. The needed fields template are displayed. EDIT 2014 Data Validation Users Guide Page 101/176

102 Fill in the fields. The parameters defined are used as parameters in the program. Their value can be changed at each launch of the job. Click on Generate. EDIT 2014 Data Validation Users Guide Page 102/176

103 The program is generated and saved automatically. It contains the vertical rules for each breakdown and the parameters added when configuring the script. Launch the job. EDIT 2014 Data Validation Users Guide Page 103/176

104 The parameters can be overwritten with the desired values. The Job results page displays the errors stored in the ErrorLog. EDIT 2014 Data Validation Users Guide Page 104/176

105 Chapter 11: Templates Management In this section you will gain knowledge for: Add Connection Templates Search Connection Templates Edit Connection Template Delete Connection Template Add Selection Templates Search Selection Template Add Insertion Templates Search Insertion Templates Add a Contact Key Search Contact Keys Add a Contact Search Contacts Verify Contact Note: This Chapter describes functionalities that apply only to the programmer role. EDIT 2014 Data Validation Users Guide Page 105/176

106 Add Connection Template Go to Menu Templates Add Connection Template 1. To add a Connection Template it is necessary to complete all the fields with valid values Example: 2. Click on Save button 3. List of available Connection Template is displayed EDIT 2014 Data Validation Users Guide Page 106/176

107 Connection template it is used on the export/import to an external database operations. Example: Search Connection Template On Menu Templates Connection Templates Search are listed all the saved connections templates Field: Properties Additional Information: Search - Name Text Field Field to be used for filtering based on the name of the template Search - User Drop-down Field Field to be used for filtering based on the user, to be selected from the drop-down list Search Button Trigger the search operation Add Template Icon Add a new template List Name Read Only Name of the template EDIT 2014 Data Validation Users Guide Page 107/176

108 Text List Description Read Text Only List User Read Only Text List Created Read Only Text Actions - Edit Icon Description of the template User that created the template Date when the template was created Edit the template Actions - Delete Icon Delete the template; a confirmation is requested from the user Note: The User field is automatically filled in with the current user logged in. Example of searching a connection template: The connection template information: Template Name = db The user which created the task: User drop-down list containing all the users present in the domain. Click on Search button. Edit Connection Template On the list of Connection Templates page choose option - edit EDIT 2014 Data Validation Users Guide Page 108/176

109 Example of editing a Connection Template: Click on Save button Delete Connection Template On Menu Templates Connection Templates Search there is option to delete a lost connection. Example of deleting Connection Templates: Select a registration and click on Delete button: Confirm the deletion and the registration is deleted with a confirmation message. EDIT 2014 Data Validation Users Guide Page 109/176

110 Add Selection Template Go to Menu Templates Add Selection Template Example of Selection Template registration: This template it is use on import to an external Database: Search Selection Template Go to Menu Templates Selection Template Search EDIT 2014 Data Validation Users Guide Page 110/176

111 Description of controls from Data Selection Template Page Field: Properties Additional Information: Search - Name Text Field Field to be used for filtering based on the name of the template Search - User Drop-down Field Field to be used for filtering based on the user, to be selected from the drop-down list Search Button Trigger the search operation Add Template Icon Add a new template List Name Read Only Text List Description Read Text Only List User Read Only Text List Created Read Only Text Actions - Edit Icon Name of the template Description of the template User that created the template Date when the template was created Edit the template Actions - Delete Icon Delete the template; a confirmation is requested from the user Note: The User field is automatically filled in with the current user logged in. EDIT 2014 Data Validation Users Guide Page 111/176

112 Add Insertion Template Go to Menu Templates Add Insertion Template Example of Insertion Template registration: This template it is use on export to an external Database: Search Insertion Template Go to Menu Templates Insertion Template Search List of Insertion Templates is displayed: The options on this page are: Field: Properties Additional Information: Search - Name Text Field Field to be used for filtering based on the name of the template EDIT 2014 Data Validation Users Guide Page 112/176

113 Search - User Drop-down Field Filed to be used for filtering based on the user, to be selected from the drop-down list Search Button Trigger the search operation Add Template Icon Add a new template List Name Read Only Text List Description Read Text Only List User Read Only Text List Created Read Only Text Actions - Edit Icon Name of the template Description of the template User that created the template Date when the template was created Edit the template Actions - Delete Icon Delete the template; a confirmation is requested from the user Note: The User field is automatically filled in with the current user logged in. Add Contact Key Go to Menu Reports Add Contact Key 1. To add a Contact Key it is necessary to complete all the fields with valid values Example: EDIT 2014 Data Validation Users Guide Page 113/176

114 2. Click on Save button 3. The list of available contact keys is displayed. Contact Keys Search On Menu Reports Contact Keys Search are listed all the saved keys EDIT 2014 Data Validation Users Guide Page 114/176

115 Field: Properties Additional Information: Search - Name Text Field Field to be used for filtering based on the name of the key Search - User Drop-down Field Field to be used for filtering based on the user, to be selected from the drop-down list Search Button Trigger the search operation Add Key Icon List Name Read Only Text List Description Read Text Only Add a new key Name of the key Description of the key Actions - Delete Icon Delete the key; a confirmation is requested from the user Example of searching a key: The connection template information: Key = key2 Click on Search button. Add Contact Go to Menu Reports Add Contact EDIT 2014 Data Validation Users Guide Page 115/176

116 4. To add a Contact it is necessary to complete all the fields with valid values Example: 5. Click on Save button 6. The list of available contacts is displayed. EDIT 2014 Data Validation Users Guide Page 116/176

117 Contacts Search On Menu Reports Contact Search are listed all the saved contacts Field: Properties Additional Information: Search - Name Text Field Field to be used for filtering based on the name of the contact Search - GSM Text Field Field to be used for filtering based on the phone of the contact Search - Key Drop-down field Field to be used for filtering based on the key, to be selected from the drop-down list Search - Text Field Field to be used for filtering based on the of the contact Search Button Trigger the search operation Add Contact Icon Add a new contact List Name Read Only Text List Keys Read Only Text List GSM Read Only Text List Verified Read Only Text List Read Only Text Name of the contact The keys assigned The phone number Phone number verified or not The List Verified Read Only verified or not EDIT 2014 Data Validation Users Guide Page 117/176

118 Text List Active Read Only Text Actions - Edit Icon Contact active or not Edit the contact Actions Verify Contact Icon Verify the contact; the Contact Info Verification page is displayed Actions - Delete Icon Delete the contact; a confirmation is requested from the user Verify Contact Once a contact is added the Verify Contact icon will allow the verification of the GSM and entered. The verification codes are sent automatically to the phone number / address specified in the contact. Example of code transmission Dear User, This is your verification code : <XXX-XXX-XXX> EUROSTAT-EDIT THIS IS AN AUTOMATICALLY GENERATED . REPLIES TO THIS MESSAGE ARE NOT MONITORED OR ANSWERED. Enter the code received in the respective field. Click on button to check and verify the GSM/ address. EDIT 2014 Data Validation Users Guide Page 118/176

119 If the code is valid the following message appears: The Verification Code field is hidden and near the GSM/ fields, the confirmation message is displayed. If the code is not valid an error message is displayed:. Click Resend GSM/ Code button to receive the validation code: EDIT 2014 Data Validation Users Guide Page 119/176

120 Chapter 12: Automation Management In this section you will gain knowledge for: Edit Definition Test Definition Search Dynamic Program Add Dynamic Program Test Dynamic Program Note: This Chapter describes functionalities that apply only to the programmer role. Edit Definition Go to Menu Automation Edit Definition The Autodomain Definition dialog is displayed: Note: The first time the page is opened no values are displayed in any of the fields. EDIT 2014 Data Validation Users Guide Page 120/176

121 Folders definition: The Edit Definition dialog will allow to provide the following information: In this text box, Programmer can define the folders that will be use for the incoming files to be processed by Edit. Note: Several folders can be defined by using new line as a separator: File Masks: The File Mask section will allow to define an input mask for the files to be processed. For example, Test_* mean that all the files stored in the folder(s) selected previously and having a name starting by Test_ will be processed. Custom File Masks: The Custom File Masks section allowsthe definition of a mask to be used for the extraction of metadata from the file name. In order to create a custom file mask the user needs to select, the add option from Autodomain Definition Screen. EDIT 2014 Data Validation Users Guide Page 121/176

122 Once the user selects Add option, the Sytem will present empty fields to type the desire values for the custom file mask. Type values could be CUSTOM or EDAMIS values. RegEx value it referes to regular expresion values. Tokens values it referes to the regular expresion groups used on RegEx field. For example, the RegEx for Census Domain to meet all table names are: 9]+_[a-zA-Z0-9]+)_([a-zA-Z0-9][a-zA-Z0-9])_\d+(_\d+)*\.([a-zA-Z]+) (cens_[a-za-z0- Enabled /Disabled option: The automation process can be Enabled or Disabled The mail address will be used to inform the receiver in case of problem. For example, several files having the same name. Confirm the definition using the button. The system will check if the defined folders exist, and if all the information have been provided. A confirmation will be displayed: EDIT 2014 Data Validation Users Guide Page 122/176

123 If a parameter is missing (above example for the adress missing), Edit will inform the programmer: Test Definition Go to Menu Automation Test Definition The Test Definition dialog is displayed: This option will allow programmer to test if a file, entering it s file name, will be taken into account by the Autodomain execution. The dialog contains two text area: Name: Enter the name of the file to be tested Status: Results of the test. The test is launched using the option The status of the test can be: - File name matches the following automated domain(s): (the concern domains will be listed) - File name does not match any automated domain EDIT 2014 Data Validation Users Guide Page 123/176

124 Note: If a file matches multiple domains, the automatic processing will not take place and the configuration must be fixed. Search Dynamic Program Go to Menu Automation Search Dynamic Program The Search Dynamic Program dialog is displayed: This dialog will list the existing Dynamic propgrams. The programmer can enter the name of a Dynamic Program and use the search option : The list of the Dynamic Programs matching the name will be displayed in the Dynamic Domain Program List. EDIT 2014 Data Validation Users Guide Page 124/176

125 Add Dynamic Program Go to Menu Automation Add Dynamic Program This option enable the definition of the dynamic programs which will be attached to the Autodomain defined previously. The Add Dynamic Program dialog is displayed: To define a Dynamic Program, the programmer will have to fill the following information: Name of the program: Format of the file which wil be processed: EDIT 2014 Data Validation Users Guide Page 125/176

126 File properties: The file properties will be defined and saved in the Dataset Import dialog. It contains all the information regarding the file format to be imported (Field delimiter, Header option, Decimal delimiters,etc.) Program: The validation program which will be used by the Dynamic program File Fields : The file fields will be defined and saved in the Dataset Import dialog. It contains all the field that will be selected from the defined format for the Importation of the file which will be validated. Parameter Set: If the program uses Parameters, the Parameters will be displayed and available for overriding. Enabled / Disabled: Option to switch On/ Off the program Priority: Enable the definition of the priority of the program. A value between 1 and 1000 can be assign to the program. Note: two program cannot have the same priority. EDIT 2014 Data Validation Users Guide Page 126/176

127 Conditions: This option will enable the definition of condition for the processing of the file. Using Keywords listed in the Dataset Metadata section, programmer will be able to restrict the validation according to some criteria. For ex: Year = 2011 will mean that an incoming file, respecting the criteria defined in the Autodomain definition (file name and defined folder) will be validated only if the year is The program will be saved, using the button. If some parameters are incorect or missing, Edit will inform that the program cannot be saved and provide the reason (for ex, Priority must be between 0 and 1000, etc.) Test Dynamic Program Go to Menu Automation Test Dynamic Program This option enables the testing of the Dynamic program which will be used for a specific file. EDIT 2014 Data Validation Users Guide Page 127/176

128 Providing a file name in the Name text box: and using the option will provide the following results: The system will check the file and will display the associated information: The file properties (Fields separators, Decimal separator, etc..) File Fields: (List of field associated to the import..) Program: (The program which will be use to validate the file) Parameters: If the program uses parameters, the list of parameters and the values that will be used for the job execution will be displayed. Validation Flow Add Go to Menu Automation Validation Flow Add This option enables the addition of a validation flow available for all users. EDIT 2014 Data Validation Users Guide Page 128/176

129 Field Type Description Flow Name Text The name of the flow Description Text The description of the flow File Type Combo-Box The type of the file to be imported: CSV, FLR, SDMX, GESMES File Properties Combo-Box Saved templates for file type Dataset Definition Combo-Box The format used for the import of the file File Fields Combo-Box Saved templates for file fields Program Combo-Box The available programs Parameter Set Combo-Box The parameter sets defined in the programs Run Parameters Text read only The parameters with their default value and the option to overwrite Back Button Returns to the previous page without saving Save Button Saves the validation flow After clicking on Save, the flow is saved and available in the list. Also, the validation flows are also available in the homepage for each user. EDIT 2014 Data Validation Users Guide Page 129/176

130 Validation Flow Search Go to Menu Automation Validation Flow Search The search criteria presents the options: Field Type Description Flow name Text The name of the flow File type Combo-box The available file types Dataset definition Combo-box The available formats Description Text The description of the flow Program Combo-box The available programs Search Button Searches according to the criteria The validation flows are displayed in the list. The options for each are: Run validation flow: - executes the validation flow directly View Details: entered. - opens the Validation Flow View page to visualize the values EDIT 2014 Data Validation Users Guide Page 130/176

131 Edit: Delete: - opens the Validation Flow Edit page to edit the values entered. - deletes the validation flow. A confirmation message is displayed. Validation Flow Edit Go to Menu Automation Validation Flow Search and click on Edit button next to the validation flow to be edited. Modify the data and click on Save button to save the changes. Validation Flow View Go to Menu Automation Validation Flow Search and click on View button next to the validation flow. EDIT 2014 Data Validation Users Guide Page 131/176

132 The flow information is displayed. Clicking on Back displays the search flows page. EDIT 2014 Data Validation Users Guide Page 132/176

133 EDIT 2014 Data Validation Users Guide Page 133/176

134 Chapter 13: News Management In this section you will gain knowledge for: Add News Search News Delete News Add News Administrator Menus Click on Administration -> News Search. Click on Add button. EDIT 2014 Data Validation Users Guide Page 134/176

135 Field Headline Expiration Date Image Type of News Domain Summary Cancel Save Description The News Headline. It is deleted from the list of news in the homepage. Image to ne attached to the news. The type of news: Maintenance, New_Function, Domain_Specific_Updates. The list of domains. The news message. Button to cancel the addition of the news. Button to save the news. Once the news item is saved it is displayed in the list of news. It is also displayed in the list from the homepage as soon as it is published. EDIT 2014 Data Validation Users Guide Page 135/176

136 Publish News Note: The Publish option is available only for news which are not published. In other cases, the button is not displayed in the Action are associated to the news. Click on button to publish a news item. A confirmation message is displayed: EDIT 2014 Data Validation Users Guide Page 136/176

137 Cancel cancels the publishing of the news item and returns to the search news page. Ok publishes the news item. The icon Unpublish is displayed in the Action area associated to the news item and the Published Date is filled with the current date and time. Several news selected can be published all at once by clicking on the button Publish from the top bar. In case one or more news selected are already published, when clicking on Publish button the message is received: Note: Only the user who created the news item can publish it. In other case the message is displayed and the news is not published. Unpublish News Note: The Unpublish option is available only for news which are already published. In other cases, the button is not displayed in the Action are associated to the news. Click on button to unpublish a news item. A confirmation message is received: Cancel cancels unpublishing the news item Ok unpublishes the news item and the Published Date is erased. The option to publish the news is available. EDIT 2014 Data Validation Users Guide Page 137/176

138 The icon Publish is displayed in the Action area associated to the news item and the Published Date is cleared. Several news selected can be unpublished all at once by clicking on the button Unpublish from the top bar. In case one or more news selected are already published, when clicking on Publish button the message is received: Note: Only the user who created the news item can unpublish it. In other case the message unpublished. is displayed and the news is not Edit News Click on button to edit a news item. There are two cases to be considered: The news item is not published - returns to the previous page without saving the changes. - saves the changes and displays the search news page. EDIT 2014 Data Validation Users Guide Page 138/176

139 The news item is published. The message when clicking on button Edit is displayed. Cancel closes the message and displays all news. Ok closes the message and displays the Edit News Item page. - returns to the previous page without saving the changes. - saves the changes and another message is displayed: Check-box Save without unpublish default option selected, saves the edited news and updates the already published one available to the users, keeping the previous published date. Check-box Save and unpublish saves the edited news and unpublishes it. Cancel cancels the action and returns to the Edit News Item page. EDIT 2014 Data Validation Users Guide Page 139/176

140 Ok saves the changes and performes the action selected above. Search News Click on Administration -> News Search. Field Type Description Headline Text The headline of the news Creation Date from - To Published date from - To Text Text The creation date of the news, available to. The published date of the news, available to. Username Combo-box The user who created the news. Type of news Combo-box The type of the news: Maintenance, New Function, Domain Specific Updates. Domain List The domain on which the news is available Search Button Searches the news Reset Button Resets to default values. Delete News To delete a news item, click on Delete button. A confirmation message is presented. In addition, several news can be deleted all at once by clicking on the button Delete from the top bar. View News The users visualize the news in the homepage. Only the publish ones are displayed. EDIT 2014 Data Validation Users Guide Page 140/176

141 Clicking on button Details, the news item is displayed. EDIT 2014 Data Validation Users Guide Page 141/176

142 Chapter 15: Administrator Functionalities In this section you will gain knowledge for: Administrator Menu Add Domain Search Domain Edit Domain Delete Domain Add User Search User Edit User Delete User Add User Group Search User Group Edit User Group Delete User Group Interface Languages Add Queues Search Queues Set Default Queue Administrator Menus The navigation bar (at the top of the screen) is the same on every screen in EDIT. It contains links which give quick access to many of EDIT s most useful functions. The users with the role of an Administrator the options Home and Administration are available. EDIT 2014 Data Validation Users Guide Page 142/176

143 Menu Short description Click the Home menu to access the Home Page. Contains a link to the Change Password functionality. Click the Administration menu to : User Search search available users by: Login name, Last name, Domain. From this menu are available actions: add user, delete user, edit user Add User create a user with all available properties (First name, Last name, Login name, Password, Domain, Role, Group ) User Group Search search available groups by: name or domain. From this menu are available actions: add group, delete group, edit group Add User Group create a group with all available properties (Name, Domain, Description ) Domain Search search available domains by Name. From this menu are available actions: add domain, delete domain, edit domain Add Domain create a domain with all available properties (Name, Description ) Application settings sets the default queue for the domain. Interface Languages manages the languages of the interface Search Queue opens the Search Queues page Add Queue opens the Add Queue page EDIT 2014 Data Validation Users Guide Page 143/176

144 Disclaimer Audit Search opens the disclaimer audit to visualize each user s disclaimer audit acceptance (see Chapter 2) News Search manages the news items (see Chapter 13). Add Domain Click on Administration -> Add Domain, the page with all the fields to add a Domain will appear. Example to add Domain: Complete the fields: - Name = DOMAIN1 - Description = ADD DOMAIN Click on button, the domain will appear on the list Search Domain Click on Administration -> Domain Search, option to search a domain by Name is displayed Example: - Search criteria: Name=Domain1 - Click on Search button If you leave the field blank and click on Search button all available domains will be listed. EDIT 2014 Data Validation Users Guide Page 144/176

145 Edit Domain Example: - select a domain DOMAIN1 click on - edit button - edit displayed data: Name = DOMAIN1_edit Description = edit Domain - click on Save button The group is listed with the new details Delete Domain On the Domain List click on from the list. - delete button. The registered domain will disappear Add User Click on Administration -> User Search, the page with all the fields to add a user will appear Example to add a user with multiple roles: Complete the fields: - First Name = user1 - Last Name = usertest EDIT 2014 Data Validation Users Guide Page 145/176

146 - Login name = user1 - Password = Confirm password = Add properties of the user, domain, role, group - Domains = DEMO click on button Add User to Domain -> will appear other fields to fill (domain, role, group) - Role = admin - click on button Add role -> will appear a list with the selected role - Role = user - click on button Add role -> will appear a list with the selected role - Role = programmer - click on button Add role -> will appear a list with the selected role - select a group = group1 (previous added) and click on Add group Save the new user by clicking on The user will appear in the list EDIT 2014 Data Validation Users Guide Page 146/176

147 Logout and try to Login with the new user. The user has multiple roles and will have access to all the functions of the application. Search User Click on Administration -> User Search and the following page will appear: There are default users in the application that are displayed in User List. Search criteria are available in the User Search part of the page. Examples - Search criteria: Login name=admin, Domain= DEMO EDIT 2014 Data Validation Users Guide Page 147/176

148 Click on button from the User List to edit a user. Example: Edit User - select user admin and click on button - change password, write a New Password and Confirm Password - delete the domain by clicking on - role of the user disappears - Select another available domain, for example Test and click on EDIT 2014 Data Validation Users Guide Page 148/176

149 - select role admin and click - the result is: - click button - login using the new password and domain Delete User On user list click on button to delete a user. The current user can t be deleted. Add user Group Click on Administration -> Add User Group, the page with all the fields to add a group will appear Example to add user group: Complete the fields: - Name = group1 - Domain = test - Description = Add user group test Click on button, the group will appear on the list EDIT 2014 Data Validation Users Guide Page 149/176

150 Search User Group Click on Administration -> User group Search, option to search a group by Name and Domain is displayed Example: - Search criteria: Name=group1, Domain=test: - Click on Search button If you leave the field blank and click on Search button all available groups will be listed. Edit User Group Example: - select a group group1 click on - edit button - edit displayed data: Name = group1_edit Domain = DEMO (change from test) Description = edit user group test - click on Save button The group is listed with the new details EDIT 2014 Data Validation Users Guide Page 150/176

151 Delete User Group On the Group List click on the list. - delete button. The registered Group will disappear from Interface Languages The Interface Languages dialog presents the available languages with the options to Delete, Add and Clear Selection for one or more languages by selecting them. To add a new Language click on Add. Code: the code of the language (fr, de..) Name: the name to be used for EDIT (Francais, Deutsch ) State: check-box to enable the language to the users. Action Button Description - Edit Opens the dialog to edit the selected language - Import Opens the dialog to import a translation for the languages selected. - Export Opens the dialog to export the translation for the languages selected. - Delete Deletes the language selected. EDIT 2014 Data Validation Users Guide Page 151/176

152 To change the language of the interface select the language from the drop-box in the right upper corner of the UI. Add Queue Click on Administration - > Add Queue. Fill in the needed information and click on Save. If the Enabled is checked, the new queue is available in the list of Execution queues for the users. Search Queues Click on Administration - > Search Queue. EDIT 2014 Data Validation Users Guide Page 152/176

153 In the list, all queues are displayed. The Actions button contains Edit, where the queue can be modified. The Edit Queue dialog presents the same fields as the Add Queue. Set default queue Click on Administration - > Application Settings. In the list, all queues are displayed. Select one queue and click on Save. The new queue is set to default. Disclaimer Audit Search Click on Administration -> Disclaimer Audit Search. The search criteria shows the state of each user s acceptance to the disclaimer. Field Type Description Username Combo-box The list of users Domain Combo-box The list of domains Time from Text Time from when the user has selected the disclaimer Time to Text Time to when the user has selected the disclaimer Event Combo-box The type of the disclaimer: Accepted, Rejected, Always EDIT 2014 Data Validation Users Guide Page 153/176

154 Accept. Search Button Searches according to the criteria Reset button Resets the values. EDIT 2014 Data Validation Users Guide Page 154/176

155 Chapter 16: Validation Flow Example In this section you will gain knowledge for: Defining, executing and obtaining the validation results for a full scenario in EDIT 1. Go to the Homepage and click to execute the validation flow. 2. Select the file to be processed and click on Start. The progress is displayed. When the job is completed the user is redirected to the jobs page. EDIT 2014 Data Validation Users Guide Page 155/176

156 3. When the job is completed the Job Details page is displayed. EDIT 2014 Data Validation Users Guide Page 156/176

157 4. From the Error Reports section, manage the error dataset. Click on View Dataset button: All the errors are displayed. 5. Click on View Statistics button. EDIT 2014 Data Validation Users Guide Page 157/176

158 The statistic dataset is displayed. It contains all the rules that failed, the number of times they failed and the error message received. 6. Click on Export button. The export dataset page is displayed. Select values for the fields needed and click on Export. By default, the error Dataset format is errdsd. The Imports / Exports page will appear and you will be able to see the status of the export dataset. 7. Download the dataset by clicking and select a location on the local hard-drive. EDIT 2014 Data Validation Users Guide Page 158/176

159 From the Generated Report section, manage the generated report from the program. Note: The Generated reports section appears at Job details page only if in the program associated was defined a specific dataset operation for generating reports. The rule must contain the information: the reporting template to use (see definition in Chapter Reporting Templates Management) and the file format. The window displays the information: - Report Name: the name of the generated report as specified in the program - The Download Report action: to download the report on local hard-disk in the format specified in the program - The Republish Report action: to re-publish the report to EDAMIS. The action is displayed only for reports with a publish option defined in the program From the Input Datasets section, manage the input dataset(s). All the datasets that are used as inputs are mentioned. Click on View Details button. The selected input dataset is displayed. EDIT 2014 Data Validation Users Guide Page 159/176

160 Click on Export button. The input dataset is moved to the Imports / Exports Datasets section where it can be downloaded on the local hard-drive. From the Output Datasets section, manage the output dataset(s). All the output datasets are enumerated in this section. EDIT 2014 Data Validation Users Guide Page 160/176

161 Chapter 17: Advanced information on the EDIT system In this section you will gain knowledge for: View Predefined Program Scripts Create Program Scripts through the Program Editor View CSV file properties View FLR file properties Import Dataset file properties templates Import Dataset file fields Job Naming Rule Execution Queues Edit Languages Interface View Predefined Program Scripts In the Create Job page. the View Details action will open the Program Detail page and will display the source script of the program. The viewing of the program will allow you to see detailed information on the validation rules applied to the dataset. EDIT 2014 Data Validation Users Guide Page 161/176

162 The - Export action will ask you to confirm the download of the program file. An advanced Programmer s Guide is available for the EDIT system for the Users that want to write their own validation programs, which is intended to be used with the Scripting Language Documentation. Note: The program is stored in a PDL format. EDIT 2014 Data Validation Users Guide Page 162/176

163 Import Dataset CSV File Properties details For CSV files, the file properties are: Skip header lines, Field delimiter, Text qualifier, Decimal character, Thousand delimiters, Header Definition and Automatic Header Recognition. CSV file properties Description The first records in a CSV file might represent the names of the following columns of data or other comments like the author etc., and are generally referred to as lines headers. If the values of Skip Header Lines is specified then the specified number of columns in the beginning of the file will be ignore on import. This field accepts only numbers and it needs to be used in combination with header definition. Example of input file: Records are separated with the character that is introduced on this field. The Field Delimiter must be the same as the delimitation character from the imported file, otherwise the file will not be imported and error will be triggered. This field accepts only one character. Example of input file : EDIT 2014 Data Validation Users Guide Page 163/176

164 This filed is to determine which Decimal Point Character will be used for Double values from the file. This field accepts only one character. Example of data: 11,22 or Fields (cells) that contain a Filed Delimiter character in their value must be escaped with a Text Qualifier character. Some input values are escaped by embedding the field inside a set of double quotes, referred to Text Qualifier, a single cell with the text, for example: Agricultural, fishery and related laborers becomes Agricultural, fishery and related ". This field accepts only one character. Example for Text Qualifier = and Field Delimiter =, This field is to determine which character will be used to separate thousand decimals from the rest of the digits in a number inside the file. This field accepts only one character. Example of data: 1000 or 1,000 The Header Definition property will be used by the System to identify if the header is meaningful or meaningless. This property will be used in combination with Skip Header Lines property. If the Header Definition property has Meaningless as value, the System will discard the number of lines indicated by the Skip Header Lines property. If the Header Definition property has Meaningful as value, the System will try to match the fields selected to be imported to the ones in the header info from the line number specified by the Skip EDIT 2014 Data Validation Users Guide Page 164/176

165 Header Lines property value. Ex: Initial Parameters for CSV file type: Skip Header Lines Parameter=2 Header definition=meaningful Field Delimiter=; The dataset file to be imported: The System will discard the first line, it ll analyse the line number two and it ll try to get the header info. So the Header fields will be COLUMN1, COLUMN2, COLUMN3 and COLUMN4. These fields will be matched to the ones selected by the user for import during the import process. Automatic Header Recognition needs to be a regular expression pattern and will be used by the importer plugin to recognize the header. If the User uses this parameter, Skip Header Lines will get zero as value, Header Definition parameter will get Meaningless as value and both fields will be disabled until the user removes the regex from the Automatic header Recognition field. The System will use this parameter to detect if a header is present on the first line of the dataset file. In the process of header recognition the System will discard the empty lines. EX: Initial Parameters for CSV file type: Automatic Header Recognition=(COLUMN[0-9]+) Field Delimiter=; The dataset file to be imported: EDIT 2014 Data Validation Users Guide Page 165/176

166 The System will analyze the first line searching the header fields and it ll use the (COLUMN[0-9]+) as the regular expression pattern for it; so the first line will be identified as a header line and will be ignored. Import Dataset FLR File Properties details For FLR files, the file properties are: skip header lines, decimal character, and thousand delimiters. FLR file properties Description The first records in an FLR file might represent the names of the following columns of data or other comments like the author etc., and are generally referred to as column headers. If the values of Skip Header Lines is specified then the specified number of columns in the beginning of the file will be ignore on import. This field accepts only numbers. Example of input file: This filed is to determine which Decimal Point Character will be used for Double values from the file. This field accepts only one character. EDIT 2014 Data Validation Users Guide Page 166/176

167 Example of data: 11,22 or This field is to determine which character will be used to separate thousand decimals from the rest of the digits in a number inside the file. This field accepts only one character. Example of data: 1000 or 1,000 Import Dataset File Properties templates A file properties template is a predefined set of file properties you have used in a previous occasion and saved as a template. Optionally, you can select a file properties template from the File Properties list. Upon selection, the application will populate the File Properties panel with the values defined in the selected template. You can manage the File Properties list as follow: Button - delete - save - save as Description Delete the selected template from the list. Update the selected template with the current values. Save new file properties template Example of saving a new file properties template: EDIT 2014 Data Validation Users Guide Page 167/176

168 After clicking Ok button the properties of the file introduced on the page will be saved in the list. Import Dataset File Properties templates During the import process you can select the fields to be imported and arrange them at the specific order you require. Fields and buttons Description After the Dataset Format is selected, the list of Available Fields is populated with the columns from the selected dataset definition. Selects all the columns and puts them in the list of Selected Fields Returns all the selected columns from the list Selected Fields back in the list of Available Fields Puts only the selected column from the Available Fields to EDIT 2014 Data Validation Users Guide Page 168/176

169 Selected Fields. Puts only the selected column from the Selected Fields back to Available Fields. Represents the columns that will be imported from the input file (dataset). These fields must be found inside the file to be imported. A file fields template is a predefined selection of fields you have used in a previous occasion and saved as a template. Optionally, you can select a file fields template from the File Fields list. Upon selection, the application will populate the File Fields panel with the selection defined in the selected template. You can manage the File Fields list as follows: Button - delete - save - save as Description Remove the selected template from the list. Update the selected template with the current fields selection Add a new template to the list using the current fields selection Job Naming Rule Click on Jobs -> Job Naming Rule. EDIT 2014 Data Validation Users Guide Page 169/176

170 Only valid metadata tokens can be used in the naming definition. The defined name will be used for the future jobs. Execution Queues The definition of the execution queues is not available in standalone mode, only on server mode. Execution queues can be defined by the administrator to prioritise the execution of jobs. By default, each domain has a DEFAULT INSTALLED QUEUE which is automatically selected as the default queue for the current domain. Add a priority queue From the menu Administration, click on sub-menu EDIT 2014 Data Validation Users Guide Page 170/176

171 Field Type Description Name Text The name of the queue Description Text The description of the queue Enabled Check-box Minimum input size Text The minimum size of the input dataset Maximum input size Text The maximum size of the input dataset Accept vertical rules Check-box A flag to indicate whether it can be used for programs with vertical rules (or only with horizontal rules) Selection order Text The priority of the queue Domains List The domains for which the queue is available Available to all domain users Check-box A flag to indicate whether it is available to regular users Save Button Saves the queue The queue is added and available in the list. Note: An EDIT Executor must be running and listening for the specified queue name in order for jobs submitted to this queue to be processed. EDIT 2014 Data Validation Users Guide Page 171/176

172 Edit a priority queue In the list of queues, click on associated to the queue The window presents the same fields as the Add Queue window. It has also the button, which returns to the previous page, without saving the changes. Delete a prority queue Click on the button. A confirmation message will be displayed to confirm the deletion of the queue Set the default queue for the current domain. In the Administration menu, click on. Field Type Description EDIT 2014 Data Validation Users Guide Page 172/176

173 Default execution queue List The available queues for the domain. Save Button Saves the changes. In the sub-menu, all queues added are displayed. The Search criteria presents the field to perform the search. EDIT Languages Interface This sub-chapter applies to the administrator role functionalities. By default, the application is displayed in English language. More languages can be added by the administrator from the menu. Add Language: Clicking on the button to add a new language., the administrator opens the window Field Type Description Code Text The code used for the language Name Text The name of the language EDIT 2014 Data Validation Users Guide Page 173/176

174 State Check-box Whether the language is available to regular users or not Save Button Saves the language Back Button Returns to the previous page without saving After clicking on Save, the new language will be available in the list. Edit Language: To edit a language, the button is available. It presents the prefilled fields Code, Name and State (the same as the Add Language window) to be edited. Clicking on Save applied the modifications and Back returns to the previous page without saving the changes. Import Language: The translation must be added by clicking on button. EDIT 2014 Data Validation Users Guide Page 174/176

175 After selecting the translation, the Upload button starts the import process. The language is now available in the drop-down list upper right corner to be selected by the users. Export Language: The language can be exported using. The Browse window will be displayed to select the local path where the translation will be saved. The translation is saved in xls format. EDIT 2014 Data Validation Users Guide Page 175/176