Tips and Tricks to loading data using the Data Import Specification

Transcription

1 Tip or Technique Tips and Tricks to loading data using the Data Import Specification Product(s): IBM Cognos Controller 8.3 or higher Area of Interest: Financial Management

2 Tips and Tricks to loading data using the Data Import Specification 2 Copyright Copyright 2008 Cognos ULC (formerly Cognos Incorporated). Cognos ULC is an IBM Company. While every attempt has been made to ensure that the information in this document is accurate and complete, some typographical errors or technical inaccuracies may exist. Cognos does not accept responsibility for any kind of loss resulting from the use of information contained in this document. This document shows the publication date. The information contained in this document is subject to change without notice. Any improvements or changes to the information contained in this document will be documented in subsequent editions. This document contains proprietary information of Cognos. All rights are reserved. No part of this document may be copied, photocopied, reproduced, stored in a retrieval system, transmitted in any form or by any means, or translated into another language without the prior written consent of Cognos. Cognos and the Cognos logo are trademarks of Cognos ULC (formerly Cognos Incorporated) in the United States and/or other countries. IBM and the IBM logo are trademarks of International Business Machines Corporation in the United States, or other countries, or both. All other names are trademarks or registered trademarks of their respective companies. Information about Cognos products can be found at This document is maintained by the Best Practices, Product and Technology team. You can send comments, suggestions, and additions to cscogpp@ca.ibm.com.

3 Tips and Tricks to loading data using the Data Import Specification 3 Contents 1 INTRODUCTION PURPOSE...4 APPLICABILITY EXCLUSIONS AND EXCEPTIONS INTRODUCTION IMPORT SPECIFICATION INTRODUCTION IMPORT SPECIFICATION WIZARD DEFINE IMPORT WIZARD DEFINE IMPORT SPECIFICATION DEFINE IMPORT SPECIFICATION IMPORT SPECIFICATION OVERVIEW...7 GENERAL TAB FILTERS OPERATION TEMPLATES VARIABLES FIELDS (1) FIELDS(2) DISTRIBUTE ACCOUNTS DELETE CHECK OTHER DOCUMENTS LIST OF OTHER SOURCES... 29

4 Tips and Tricks to loading data using the Data Import Specification 4 1 Introduction 1.1 Purpose This document will provide tips/tricks to using the data import feature in Controller. 1.2 Applicability Controller 8.1 or higher 1.3 Exclusions and Exceptions The files must be ASCII OEM or ASCII ANSI, in other words ordinary text files. Files can have a fixed record length or be common separated. The process will also cover if staging tables are used as the method of loading data into Controller. The Staging tables approach was introduced in Controller 8.2 and higher. 2 Introduction Import Specification 2.1 Introduction The import module in IBM Cognos Controller is rather sophisticated; you can create an import specification in many ways and import rather complex files. The GL wizard is the more user-friendly way to define an import specification. If you have to define a specification for a more complex file, you sometimes have to use the other module Define Import Specification. This document will show how you could use the module Define Import specifications to define complex import specifications. You should not see this document as the one and only solution, there are many ways to define an Import specification and it s possible that this document doesn t contain the very best solution.

5 Tips and Tricks to loading data using the Data Import Specification 5 3 Import Specification Wizard 3.1 Define import Wizard The import wizard is the simplest way to load the data into controller. The file formats does not need to be change and the file can be loaded into system under its current format see below example Data File: Import Wizard: Import Wizard define the fields

6 Tips and Tricks to loading data using the Data Import Specification 6 The data in this files is on every row, so we can easily mapped the file the following way a) Repeated for each row Actuality = 1 st Column b) Repeated for each row Company = 2 nd Column c) Repeated for each row date = 3 rd column d) Repeated for each row account = 4 th Column e) Repeated for each row amount = 5 th Column The only item missing is currency code and we can achieve this by selected currency and ticking Get Local Currency Code from Company code. -NB- The import wizard can also be modify to ask question see below info What company? What actuality? What period?

7 Tips and Tricks to loading data using the Data Import Specification 7 4 Define Import Specification 4.1 Define import specification The define import specification allows you to strip the file in order for the file to be imported in controller. 4.2 Import Specification Overview 4.3 General Tab On this tab, you register information about the file you wish to import a) Source: ASCII OEM (text file format for DOS) or ASCII ANSI (Text File format for windows) or Controller Staging Table or Framework Manager) b) Line separator if the records in the file end with CR+LF (carriage return and line fee), select this option. This is the normal way to create records. If the file has no CR+LF, it must have a fixed length, which must be specified.

8 Tips and Tricks to loading data using the Data Import Specification 8 c) If you select Framework Manager as your data source then you will required to define the model and report. d) Symbols: I) Blank Substitution This parameter is used for define fields variables and so forth. Here you can indicate the character to use instead of the blanks. The character need not occur in the input file which means this information is optional. II) Row Continuation Character Normally one line is one unit of information. In some files, a single block can continue onto the next line. In such cases Cognos Controller should not import the second line as new information, so you must specify the character signalling that a line will continue in the import file at the end of each line that continues. III) Digit Group Symbol Specify the digit grouping symbol, for example dot or comma. If you use spaces in the file, leave the field blank. Do the same if you do not wish to use any digit-grouping symbol. IV) Decimal Symbol Specify the decimal symbol in the file, for example dot or comma.

9 Tips and Tricks to loading data using the Data Import Specification 9 e) Reconciliation I) Perform reconciliations between accounts Indicate if reconciliation should be run between accounts. The reconciliation is then performed based on codes entered in the account structure. Also, indicate if the reconciliation should be run at the total dimension level or individually for each dimension code. If you choose to perform reconciliation, you can specify the accounts where IBM Cognos Controller should store any differences that may arise. You can specify different accounts for a positive and negative difference. If difference should be calculated for the dimension totals, you should also specify to which members of the dimension any differences should be registered.

10 Tips and Tricks to loading data using the Data Import Specification 10 f) Options: I) Expand Tabs if the fields in the file are tabseparated, select this control that they expand to become the proper number of blank positions. Note that if the file contains fields with internal tab separation, this control must not be selected. II) III) IV) Test Run If you mark this field, the data are imported from the file into a test database but not IBM Cognos Controller s database. This selection is useful during testing, because you can check the import in a special log report under (Transfer/Log Reports) Automatic transfer Net Income between IS and BS Indicate if the net income in the income statement should be automatically transferred to balance sheet when the file is imported. Track Data This feature allows you track data as it is being loaded into Controller. Highly recommended to have this field check. V) Allow Separate Counter Companies Select this option when values for intercompany balances are in different files, in which the values apply to the same account, company, period, but different counter companies. VI) VII) Allow Separate Dimensions Select this option to allow values for separate dimensions in different files to be imported to the same account, company, period, actuality with out deleting previously imported dimension values. Copy accounts according to Control between Accounts If values in the imported accounts should be copied automatically to another account according to coded reconciliation rules in the account structure.

11 Tips and Tricks to loading data using the Data Import Specification Filters A Filter defines which rows to include or skip based on a given criterion. If you have defined more than one filter, -NB- all filters must be satisfied for the row to be imported from the file To create a new filter, Click New Filter. Each filter should have a defined name containing a maximum of 12 characters. Depending on the selections you make, all of the fields described below may not appear on your screen. a) Filter Type I) Start Read Skip all lines in the file until the filter criteria are true. II) III) IV) End Read Skip all lines in the file for which the filter criteria are true Skip Lines Skip all lines that fulfil the filter criteria Include Lines Skip all lines that do not fulfil the filter criteria.

12 Tips and Tricks to loading data using the Data Import Specification 12 b) Filter Subtype: A filter subtype is used to tell which type of criterion to use for a specific filter. Line No Test a special line number in the file Line Contains Test if a line contains a specific text/value Item Contains Test if an item in the file contains a specific text/value. Depending on the filter type and subtype chosen, one or more of the following options will appear. c) Other requirements When dealing with filters subtype the following may also need to be considered: I) Line specific line number in the file II) III) IV) From-To From one position to another position in a record Field a specified field in a line Use of value of the variable instead of specifying a line number, the FROM and TO positions or the Field number, you can use the value in a specific VARIABLE. V) Separator Character to separate fields in the file. A special character, for example a Tri-Graph character, can be used

13 Tips and Tricks to loading data using the Data Import Specification 13 d) Filter Value I) Value The value you supply for the test can either be a normal text, a value or a regular expression. You need not enter a value, in which case you can choose Blank, Zero or Blank or Zero II) Regular Expression This option selected indicates that the value in the value field is an expression. The search string in a regular expression can contain the characters * or?, the socalled wildcard. E.G. If the search string *F?ZYZ* is used with an include filter, all lines that contain the string FxZYZ (where x could be any character) somewhere will be included.

14 Tips and Tricks to loading data using the Data Import Specification 14 Example: e) Skip Lines See below print screen The filter option using a regular expression is a way to skip particular items in the data extract. The assumption is that the account number starts with N then you can skip this line by using the regular expression ^N. Similarly you can skip the line using the regular expression A$ where you want to skip all the items (i.e. account code) ending with A.

15 Tips and Tricks to loading data using the Data Import Specification Operation Templates An Operation template consists of one or more operations to performed one after the other. This section describes how to create operation templates and which operations can be executed. The initial value in the chain can be a value from a text file stored in a field, temporary field or variable. The output resulting from the execution of the operation template is stored in the same field, temporary field or variable to which the template is connected. Operation template names must begin with an (_), i.e. _account or _Actuality or _Period, etc. Operations sometimes need arguments to be executed. If an operation needs an argument, you can either type it in directly or pick from a pop-up list that will appear. The pop-up list contains variables and temporary fields that have been set up. You can type constants or calculations directly in the field. Arguments for operations the input to an operation is assumed to be the output from the preceding operation (or, for the first operation, the origin of the field or variable). You can use constraints, variables, temporary fields and #INPUT) as arguments in an operation. # INPUT contains input for the current operation and need only be used in exception cases. Existing Operations The operations are presented in the below table with there description Template ADD nval ALPA DIVIDE BY nval DROP ch GETITEM n sep Description Adds nval ( a value to the current input Converts a numerical value into an alphanumeric value; IBM Cognos Controller Converts automatically when necessary, if the value is not valid, it will be printed in the error log. Divides the current input nval (a value) Skips all characters up to the character ch (example Operation: DROP; INPUT: XXX; YYY Output: YYY Gets an item n from the current input with the separation character sep ; can be Tri-graph symbol

16 Tips and Tricks to loading data using the Data Import Specification 16 GETITEMEXPR Loads the expression in a field GETITEMLOOP GETLINE IIF expr val1 and val2 INSERTBYMASK chs mask LEFT n LEFTALIGN LOCALCURRENCY LOOKUP lookup table on no match exclude/default LOOKUPFACTOR lookup table on no match exclude/default Loads values for each loop. The operation is displayed but can only be created using the import specification wizard. Loads the contents of n th row from the import file If the expression expr is true, return val1, otherwise return val2 Inserts the character in chs based on mask in current input. Example: INSBYMASK**XX_XX_XX Input: Output: Returns the first n characters from current input Aligns current input left Loads the local currency for the company code in the current input according to the company database from the company table Loads a value from the lookup table using current input as search key; the search is performed in the lookup tables from columns; if current input is found in the from columns, the value returned is either the value in the TO column or a default value, if a default is define. Load a value from the factor column in the lookup table, the search is performed in the lookup table s from columns; if current input is found in the from columns, the value return is either the value in the factor column or a default value, if a default value is defined. LOOKUPSPLIT Determines the breakpoint for when a field on the fields(2) tab should be converted to different codes depending on the total values. Example: The account code A should be positive, but if the aggregated value is negative, the account code should be converted to B.

17 Tips and Tricks to loading data using the Data Import Specification 17 LOWERCASE MASK mask Operation: LOOKUPSPLIT Argument 1: X (name of lookup table) Argument 2: 0 (value) Argument 3: ; (separator) From TO A B;A Converts current input from uppercase to lowercase input Performs the opposite function of INSBYMASK Example: Operation: MASK xx_xx_xx Input: Output: MULTIPLY BY nval Multiplies current input by nval (a value) NDROPLEFT n NDROPTRAILING n NUM PACK ch Numerical, deletes n characters from the beginning of the current input Numerical, deletes n characters from the end of the current input Performs the opposite of ALPHA Deletes all repeating characters ch from the current input string Example: Operation: PACK C Input: ACCDCECC Output: ACDCEC PADDLEFT ch n Extends a character string to the left with the characters; if n is less than the number of characters in the current input, input will be shortened to n characters Example: Operation: PADDLEFT X 10 Input: YYYY Output:XXXXXXYYYY PADDRIGHT ch n Extends a character string to the right with the character(s) ch to n numbers of characters; if n is less than the number

18 Tips and Tricks to loading data using the Data Import Specification 18 characters in the current input, input will be shortened to n characters. Example: Operation: PADDRIGT x 10 INPUT: YYYY OUTPUT: YYYYXXXXXXX PREFIX str Puts together str before the current input RELPERIOD REPLACE str1 str2 REVERSE Operation defines the relative period for non-cumulative data. Determine the relative period based on the period in the current input, a positive integer for the offset period and the offset actuality. Example: The operation uses #INPUT as argument1 which is origin used for the field REL_PERIOD on the fields(1) tab. Argument2 is the offset from the period in #INPUT. For instance, an argument 2 set 1 denotes monthly, argument 2 set to 3 would denote quarterly data. Argument3 is the offset actuality. The offset actuality is particularly important when dealing with incremental weekly data. Operation: RELPERIOD Argument1: #INPUT Argument2: 1 Argument3: AC Replaces all occurrences of str1 with str2 in the current input Converts the string to the opposite order Example: ABCD would be turned DCBA RIGHT n Returns the n last characters from the current input RIGHTALIGN ROUND n SELECTIF criteria Aligns current input right Rounds the current input to n decimal places. Includes the current input if criteria are true; input not included is shown on an error report.

19 Tips and Tricks to loading data using the Data Import Specification 19 SKIPIF criteria SQUEEZE ch Performs the opposite of SELECTIF Deletes all occurrences of the character(s) ch in current input Example: Operation: SQUEEZE Input: a text where Everything Output a text where everything STRIP ch STRIPLEADING ch STRIPTRAILING Deletes all characters ch at the beginning and end of current input period Deletes all characters ch at the beginning of current input Deletes all characters ch at the end of current input. SUBSTRING SUBSTRINGEXPR SUBSTRACT nval SUFFIX STR Loads n characters from the pos position Loads values for each loop. The operation is displayed but can only be created by the import specification wizard. Subtract nval (a value) from the current input Puts together str after the current input TAKE ch TRANSLATECHARACTERS chr1 chr2 Performs the opposite function of DROP Translate all characters chr1 into corresponding characters in chr2 in current input UPPERCASE Changes input to uppercase Example a) SKIPIF command syntax is as follows Operation: SKIPIF Argument 1: ( Static value optional marked) #input= 05805KM where KM is an example input to be skipped, that can come from a temporary field which can be defined on the fields (1) tab. Account Define the account on the fields(1) tab to have the origin to be set to be temporary and then set the operation to be the operation template (_Template Name)

20 Tips and Tricks to loading data using the Data Import Specification 20 By setting the Account field this way the Temporary is used the #Input or Origin to the operation template _Template Name 4.6 Variables

21 Tips and Tricks to loading data using the Data Import Specification 21 Variables contain information separate from the information in a file that is the values do not change in every line. The information can be loaded from the file name. Variables can be used in field definitions and operation templates and must start with an ampersands (&). Depending on the selections you make, all of the fields described below may not appear on your screen. To define a new variable, Click New Variable. Specify the name of the variable, using a maximum of 12 characters, and enter any explanatory text you wish in the neighbouring field. Repeat Factor allows you to process the file several times. For example, if the file contains actual and budget amounts on every line, you have to import the file twice to get both actual and budget values. This is done automatically if this field is selected. Only one variable can be defined with a repeat factor. Enter the number of times that the file should be processed. Format define the formats for variables, Alpha, Alpha Upper, numeric then the width and, if numeric, the number of decimal places. A blank fields means no decimal places. Origin Origin is the source of the variable s initial value, that is where the value has been retrieved from: Name File Description Information exists in the file. This function works even if the line specified is skipped by a filter. Prompt Filename File ext Static Variable Asks for the value on the screen. If you are applying a repeat factor, you are prompted only the first time. Obtains the filename extension as a value Obtains the filename extension as a value Makes the variable static Refers to another defined variable Depending on the origin selected, one or more of the following data should be specified: line number, what should be entered in the on-screen prompt, the static value or the name of the variable. Operation

22 Tips and Tricks to loading data using the Data Import Specification 22 Operation indicates if an operation must be performed before the value can be created. None Item Substr Op.template Nothing is done Loads a specific field from a file, such as from a comma-separated file. Trigraph characters can be used as field separators. Loads a sub string of a line or filename Executes a defined operation template to obtain a value Example: 4.7 Fields (1) There are two fields tabs that allow two different types of fields: IBM Cognos Controller fields and temporary fields. The identity of a temporary field must begin with character. IBM Cognos Controller fields are fixed and correspond to IBM Cognos Controller data-base identities. Temporary fields are use to execute various calculations in multiple steps. Temporary fields can only be used on the fields(1) tab.

23 Tips and Tricks to loading data using the Data Import Specification 23 Name Account Actuality Amount Company Currency C_Company C_Dim All IBM Cognos Controller fields are predefined in a list. This is the final definition in the file before it is imported into Controller. Variables, temporary fields, operation templates and lookup tables are only tools for obtaining a final value for a field. Description Account Actuality Amount Company Currency Counterpart Company Extended Counter Dimension EXTDIM1 Extended Dimension 1 EXTDIM2 Extended Dimension 2 EXTDIM3 Extended Dimension 3 EXTDIM4 Extended Dimension 4 Journal O_Company Period Rel_ACT Rel_Period TRANAMOUNT TRANCURR Journal Type Original Company Period Relative actuality to describe preceding period if the file does not contain cumulative values; you must never use this field when you import cumulative values. Relative period to describe preceding period (YYMM) if the file does not contain cumulative values; you must never use this field when you import cumulative values. Transaction Amount Transaction Currency Creating Temporary Field To create a Temporary Field, click New temporary field. Specify the identity using a maximum of 12 characters. Enter a description of the temporary field if you wish. Fields cannot be created or changed but are predefined in a list.

24 Tips and Tricks to loading data using the Data Import Specification 24 Format - define the format for the temporary field, alpha or numeric. Specify the length and if numeric, the number of decimal places Origin The origin is the source of the field s temporary field s initial value Name None File Variable Temp. Field Description Loads in formation from the file, a specified field or a specific position Use the value in a variable Uses the value in a temporary field Relative Line indicate the relative line for the field. A 0 (Zero) indicates every line in the file. The highest value +1 for this check (for all fields) indicates how many physical lines make up a logical line. Operation indicates if an operation must be performed before the value can be created. None Item Substring Op.template Loads a specific item from a commaseparated file Loads a substring of a variable or temporary field Applies the defined operations to create the value Use last non empty value Select this field if IBM Cognos Controller should use the last value filled in according to specified criteria. For example, if you enter a date in the first field in the first line of the file, you can easily repeat that data by leaving the same field on all other lines blank. The system will retrieve the value in the first field on line 1 and use it on all lines for which the same field is blank. See below example:

25 Tips and Tricks to loading data using the Data Import Specification 25

26 Tips and Tricks to loading data using the Data Import Specification Fields(2) Fields(2) lets you performed additional operations on the values already stored in fields on the fields(1) tab.

27 Tips and Tricks to loading data using the Data Import Specification Distribute Accounts Use the function Distribute Accounts when you wish to use the same value from a file in several accounts in IBM Cognos Controller. Specify the Source Account from which the value should be copied and the factor, in other words whether the value should be multiplied by 1 or -1. Specify the target account to which the value should be copied. NB- that you can copy accounts on the general tab according to defined reconciliation rules in IBM Cognos Controller. In that case, you need not specify each account to copy, because IBM Cognos Controller automatically copies the values based on coded reconciliation rules in the account structure.

28 Tips and Tricks to loading data using the Data Import Specification Delete If you import a new file for a period you previously imported, the only values read the second time are those in accounts in the new file. To ensure that old values are erased, you should define which forms and codes to zero out on the Delete tab. Select the dimension codes and forms to zero out for the current period before reading the new values Check When the specification is ready, you can run Check. The check function verifies the logical structure of the specification. To run a check, click in the bottom row.

29 Tips and Tricks to loading data using the Data Import Specification 29 5 Additional Tips 5.1 Import Wizard Sometimes the user can start with wizard and map the data that is apparent in the file. The other operations in the file can be mapped using operation templates and/or fields, etc. 6 Other Documents 6.1 List of other sources Guidelines to setting up and using lookup tables Controller Guidelines to Import Data into Controller using Microsoft SQL_Controller83 Guidelines to importing data using staging table Controller 8 Guidelines to importing data into controller using data manager and staging tables

30 Tips and Tricks to loading data using the Data Import Specification 30