TenUp User's Guide. (212)

Transcription

1 TenUp User's Guide (212)

2 TenUp User's Guide Contents 2 Contents Introduction... 3 Prerequisites... 4 Installing TenUp...6 Invoking TenUp from a Command Line... 8 Basic Usage... 9 Connection Management Server-Side Management...18 Memory Management Segmentation Settings...21 Providing Metadata...23 Advanced Topics: Automating TenUp Setting Environment Variables Session Management...27 Example Scripts...28 Support Requests...31 Return codes...32 Spec tree tags...34 Spec tree example data, Inc. All rights reserved.

3 TenUp User's Guide Introduction 3 Introduction The 1010data TenUp User's Guide provides information about installing and using TenUp to load data into the 1010data Insights Platform. TenUp is an alternate way for users to interact with the 1010data Insights Platform. TenUp is a command line tool that loads data from flat files or data that is extracted from ODBC-compliant databases into the Insights Platform. TenUp provides a robust set of options for loading the data precisely as intended. This includes, but is not limited to, creating new tables and appending to existing ones. Document conventions This guide uses brackets ([]) with text in all capital letters and underscores instead of spaces to represent variables. For example, [PATH_TO_DESTINATION] indicates that this is a variable that should be replaced with the actual path to the destination folder. TenUp prompts users to input their 1010data Insights Platform password if they do not include it in their TenUp command, this guide omits the password switch from examples unless it is pertinent to the topic. TenUp provides switches in a short form and a long form. The short form is a dash and a character. -[S] The long form uses two dashes and the switch name. --[SWITCH] This guide uses the short form, where available, in examples, but they are interchangeable. The command to invoke TenUp depends on your operating system. Examples in this manual use tenup64.

4 TenUp User's Guide Prerequisites 4 Prerequisites These are the steps that should be taken before attempting to install and work with TenUp to ensure success. Loading flat file data only All of the following information and preparation are necessary before trying to upload data from a flat file using TenUp. All users must have: A table tree containing the information about the target table, including column names, data types, etc. The table tree can be entered into the command line, but we recommend you use a table tree file. For more information about table trees, see Table Tree in the 1010data API Reference Manual. A flat file containing data that you want to load. A valid 1010data login information. A 1010data destination folder that the user account has permission to write to. A Linux or MacOS user who is only loading flat file data must still install and configure unixodbc. For instructions, see. Loading ODBC data All of the following information and preparation are necessary before trying to upload data from an ODBCcompliant database using TenUp. All users must have: connection string information. At a minimum, you'll need of the following pieces of information to connect to a data source and extract data for loading: DSN (Data Source Name) Data source user name Data source password Connection strings are often sent in the following format, but this may vary from one data source to another: "DSN=[DATA_SOURCE_NAME];UID=[USER_NAME];PWD=[PASSWORD]" A valid query that will extract data from the data source An ODBC client application for testing your DSN configuration Valid 1010data Insights Platform login information The path to an Insights Platform destination folder that the user account has permission to write to A Linux or MacOS user must install and configure unixodbc. Most of the items above are self-explanatory, but 1010data strongly recommends that you confirm your data source configuration with a simple ODBC client application before attempting to use TenUp. On Linux and Mac, you can use isql, a free, open-source client application included with unixodbc. Install unixodbc (Linux and MacOS To install unixodbc on Linux and MacOS, first run the following commands from your command line: $ curl -O ftp://ftp.unixodbc.org/pub/unixodbc/unixodbc tar.gz $ tar -xzf unixodbc tar.gz $ cd unixodbc-2.3.2/ $./configure $ make $ sudo make install

5 TenUp User's Guide Prerequisites 5 This series of commands will download the source files for UnixODBC 2.3.2, compile the source into an executable, and install it on your system. It is important that you install version or later, and earlier versions will cause a runtime error. On many Linux distributions /usr/local/lib (where UnixODBC will install by default) is not searched when loading shared objects. This can cause the following error to occur: error while loading shared libraries: libodbc.so.2: cannot open shared object file: No such file or directory. To avoid this error, export the following environment variable: export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/local/lib" To test your ability to connect to the DSN, you can run the following command: $ isql -v [DATA_SOURCE_NAME] where DATA_SOURCE_NAME is the DSN for the database you're connecting to. isql is a remote console for interacting with databases via ODBC. It is useful to test your connection with isql because it requires that the data source has an ODBC driver correctly installed and functioning. This, in turn, is important, because TenUp is only able to extract data from ODBC data sources that have an ODBC driver correctly installed and functioning. The average user should not attempt to install and configure an ODBC driver for a database. This is a job for your friendly neighborhood Database Administrator.

6 TenUp User's Guide Installing TenUp 6 Installing TenUp You can install TenUp on Linux, MacOS, and Windows. To install TenUp on your system first download it from the following link: downloads/tools/tenup/tenup.zip. Clicking this link will automatically download the zip file containing the TenUp installers. To unzip the file from your terminal on MacOS or Linux you can use the unzip command from the directory where the TenUp zip file is saved, as follows: $ unzip tenup.zip This command will unpack the contents of the tenup.zip file. Next, identify if you are connecting a 32- bit or 64-bit ODBC driver. The version of TenUp you need to use corresponds to the combination of your operating system and the ODBC driver of the datasource, as follows: 64-bit OS and 64-bit driver - use tenup64 64-bit OS and 32-bit driver - use tenup32 32-bit OS and 32-bit driver - use tenup32 If you aren't sure which architecture your ODBC driver supports, contact the database administrator for information about the datasource you're connecting to. From there, follow the instructions for installation that are specific to your operating system, as specified below. Installing TenUp on Windows To install on Windows, simply double-click the TenUp installer and follow the prompts on the screen. Installing TenUp on Linux or Mac To install TenUp on a Linux, you must identify whether you have a 32-bit or 64-bit version. Use the command uname -m to determine if you have 32-bit (i686 or i386) or 64-bit (x86_64). After you've determined which version of Linux you have, move the TenUp executable (e.g., either tenup32 or tenup64) into the bin directory. This directory has the file path /usr/local/bin. To install TenUp on a Mac, copy the mac executable file (tenup) from the osx folder of the unzipped TenUp folder. into the bin folder. This directory has the file path /usr/local/bin. Note: In order to copy the file to /usr/local/bin, you may need administrative access on your computer. Alternatively, you can add the directory where TenUp is stored to your $PATH variable. This method doesn't require you to have administrative access. To run TenUp as a command from any directory, simply add the directory where TenUp is saved by following these steps. 1. Go to your home directory with the following command: $ cd ~ 2. Open your bash profile with the following command: $ vim.bash_profile Note: vim is just one text editor that is available in a standard bash installation. If you're not familiar with Vim you can use your favorite text editor instead. 3. Add the following command to the next empty line of the file: export PATH="$PATH:$HOME/[CURRENT_TENUP_DIRECTORY]" 4. Save the changes to your file After you've added the current TenUp directory to your $PATH, you can invoke TenUp from any directory on your computer.

7 TenUp User's Guide Installing TenUp 7 Note: You may have to close your terminal and re-open it before the changes to your $PATH take effect. After you've copied the file to its correct location, you can able to run TenUp as a shell command. However, the exact command is dependent on the version of the operating system and ODBC driver combination you're using. The next section covers all the basic information you need to perform a basic extract and load with TenUp.

8 TenUp User's Guide Invoking TenUp from a Command Line 8 Invoking TenUp from a Command Line On Windows and Linux, invoke 32-bit TenUp with the tenup32 command. To use the 64-bit version of TenUp, the command is tenup64. On a Mac, you do not need to use different commands for 32-bit and 64-bit versions. The Mac command for TenUp is simply tenup. Examples in this manual use tenup64.

9 TenUp User's Guide Basic Usage 9 Basic Usage After you've successfully installed TenUp on your system, you can start uploading data to 1010data. There are just a few simple components to a basic TenUp command. Here's the anatomy of a basic query in TenUp: $ tenup64 -u [USERNAME] -p [PASSWORD] [PATH_TO_NEW_TABLE] Keep in mind that this is a basic TenUp command. There are options and alternative methods available, and we'll get to them a little later in this manual. But these are the pieces of information that are required by TenUp. Note: Words in all caps enclosed in brackets ([ ]) are variables. When you see [USERNAME] in a command, you substitute your 1010data user name. This notation is used throughout this document. TenUp provides two kinds of switches for performing the same task. The order of the switches in the command can be in any order. The two switches work as follows (where [S] represents a command option, or "switch"): -[S] Or... --[SWITCH] This manual uses the short version for its examples. If you are curious about the longer form for each option, you can find them all as well as other useful information about each option by running the following command: $ tenup64 --help Required Information for a TenUp Command All TenUp commands contain the following: tenup32/tenup64/tenup How the TenUp program is invoked. All TenUp commands begin with this. -u [USERNAME] -u tells TenUp you are providing a username to send to 1010data. After the -u, type a space and then your 1010data username. -p [PASSWORD] -p tells TenUp you are providing a password to send to 1010data. After the -p, type a space and then your 1010data password. While this basic command structure is all that is required of basic interactions using TenUp, there is important information of which you should be aware: If you don't provide a password in your command, TenUp will automatically prompt you for it. Examples for the rest of the manual won't include passwords. If your company uses a custom URL to access 1010data, or you want to use a version of 1010data that's different from the version your company has been provisioned to use, you will need to provide a gateway in addition to the information detailed above. Loading with a Flat File If you are loading data from a flat file, there are a additional components for the basic TenUp query. Here's the anatomy of a basic TenUp query for use with uploading a flat file: $ tenup64 -u [FLAT_FILE_NAME]

10 TenUp User's Guide Basic Usage [FLAT_FILE NAME] The at symbol signals TenUp that you are providing a table tree file name. After type the name of the table tree file. The table tree file is an xml file. The table tree contains the destination path as well as information about the columns of the uploaded table in 1010data. See Table Tree for more information. Note: Table trees are necessary. Table tree files are not, but they are recommended as a best practice. The name of the flat file. Flat file delimiters The default delimiter for flat files is set for commas (,). You can use the --columnseparator=[delimiter] option to change the delimiter. Delimiters must be single-byte characters. If you're using a delimiter that is an option in the shell you're working with, use quotation marks (" ") to set off the delimiter. For example, if you were using the pipe delimiter ( ), you would enter --column-separator=" " In Linux, you can also use backslash (\), e.g. --columnseparator=\ $ tenup64 -u [USERNAME] -p [FLAT_FILE_NAME] --column-separator=[delimiter] Loading from ODBC-compliant databases If you are extracting data from an ODBC-compliant database and then loading that data into 1010data, there are a few additional components of a basic TenUp query. Here's the anatomy of a basic TenUp query for use with an ODBC-compliant database: $ tenup64 -u [USERNAME] -C [CONNSTR] [PATH_TO_NEW_TABLE] [QUERY] -C [CONNSTR] The Connection String for the ODBC-compliant database you're extracting data from. Connection strings often contain a user name and password that are specific to your account on the ODBCcompliant database. The most basic form of a connection string simply contains the name of the data source you're extracting from, i.e.,"dsn=[data_source_name]" However, most connections strings contain authentication information in addition to the DSN: "DSN=[DATA_SOURCE_NAME]; UID=[DATA_SOURCE_USER_NAME]; PWD=[DATA_SOURCE_PASSWORD]" Note: Above is a simple connection string. Many connection strings also pass configuration information to the ODBC driver/database. If you're not sure what information must be in the connection string, contact your database administrator.

11 TenUp User's Guide Basic Usage 11 [PATH_TO_NEW_TABLE] [QUERY] The name of the new 1010data table you're creating. 1010data tables are expressed as follows: training.retail.item Where training is a top-level parent folder, retail is a subfolder of training, and item is the name of the table. Note: You must have permission to write to the folder where you're attempting to save your new 1010data table. A query that will retrieve data from your ODBCcompliant database. The most common form of query is SQL. The simplest SQL query looks like this: "SELECT * FROM [TABLE_NAME];" You can either write a query yourself or acquire one from your IT department. Keep in mind, SQL databases are not the only ODBC-compliant databases. The above is used strictly as an example. 64-bit integers TenUp can load 64-bit integers (e.g SQL_BIGINT) as a 1010data bigint column. In order to ensure that SQL_BIGINTs are loaded properly, use the --use-long-ints switch to indicate the data contains 64-bit data. The maximum value for a 64-bit integer is 9,223,372,036,854,775,806, and the minimum value for a 64-bit integer is -9,223,372,036,854,775,806. tenup64 -u [USER_NAME] -p [PASSWORD] -C [CONNSTR] [PATH_TO_NEW_TABLE] [QUERY] --use-long-ints Currently, you can only load 64-bit integers from ODBC databases. Note: The 64-bit integer support is available for use in versions prime and beta or later. Using a query file (ODBC only) Placing your query in the command itself is perfectly workable for small queries. However, if you want to provide a more complex query to the data source for an extract and load operation, writing the query in a file is much better. You can tell TenUp to query the data source with a query located inside a file with syntax it provides: tenup64 -u [USERNAME] -C [CONNSTR] Where to save your new table Whenever you extract data from a data source outside of 1010data, or load it from a flat file, you need know where to save that data in the 1010data system. You need to provide a full path to the new table being created. If you are appending to an existing table, you will need to provide the path to that table. If you're familiar with the file browser in the 1010data web interface it's easy to find the location where you want to save your table. Each folder has a name, and subfolders and tables are denoted by a. character. Here's an example of a folder name at the highest level:

12 TenUp User's Guide Basic Usage 12 Notice that the name of the folder, training is a single word with no capital letters or spaces. Next, we'll look at the name of the retail folder:

13 TenUp User's Guide Basic Usage 13 In this example, the Retail folder is highlighted in the browser. The name of the folder is actually, training.retail because it is a subfolder of the Training Examples folder. When you load data with TenUp, you will provide the command with a folder location and the name of the new table you are creating. If you're loading data from a flat file, the path to the folder or the table needs to be provided as part of the table tree. If you were to use the location from our example above in the table tree's name attribute, it would appear as follows: name="training.retail.newtablename" If you're extracting and loading data from an ODBC-compliant database, and you were to use this location with our example command above and create a new table it would appear in the TenUp command as follows: $ tenup64 -u [USERNAME] -C [CONNSTR] training.retail.newtablename [QUERY] To illustrate the process, let's look at a basic extract and load example. To start, we need a data source. To make sure we can connect to the data source through an ODBC driver, we'll use the isql tool to take a quick peek at the data we're going to extract and load. Note: Some information in the commands are obfuscated to protect the anonymity of 1010data's employees. We will assume the isql connection information has been configured correctly and then issue the following command: $ isql -v mysql_odbc

14 TenUp User's Guide Basic Usage 14 where mysql_odbc is the DSN (data source name) used for this command. For the purposes of this guide we're using the same sample SQL dataset for most examples. To start, we'll run a very basic SQL query using isql: SELECT * FROM testtable; which will return all the data from a table named testtable. The results of the query in isql appear below: What we have here is a collection of some data from a popular computer game, taken from the public API of the developer and loaded into a SQL database. While it's nice to have it in SQL, it would be much better to have it 1010data where it can be more thoroughly analyzed. We'll use the basic TenUp command to extract the data and load it into 1010data. Again, keep in mind certain values in this command are replaced with generic values to protect the information: $ tenup64 -u myusername -C "DSN=datasourcename;UID=username;PWD=myPa$$w0rd!" training.retail.mygamedata -t "WoW Guild Roster" "select * from testtable;" After running the TenUp command, assuming no errors were returned, you can login to 1010data and see the newly created table, as shown below: Note: To use the command for this extract and load you can't be logged in to 1010data already. The options for killing or possessing an existing section are detailed in the Managing Connectivity section of this guide.

15 TenUp User's Guide Basic Usage 15 Specifying a gateway (-g) When you log in to 1010data, whether it's via TenUp or the web GUI, by default you will be automatically logged in to the version of 1010data that your organization has been provisioned to use. This is the default behavior and no action on your part is required. However, there are some situations in which you may want to use a different version of 1010data than your default version. For instance, our beta environment is often used by 1010data users who want to see what features and improvements are on the horizon. Furthermore, some of our clients use custom URLs for 1010data access. This means that instead of going to the standard URL for access, they have some custom corporate URL such as: In either instance (e.g., you want to use a specific version of 1010data you're not provisioned for, or your company uses a custom corporate gateway), you will need to specify your gateway to TenUp. The following two examples show the syntax for each circumstance you may encounter when you need to specify a gateway. In both cases you will use the -g option. To specify a specific version of 1010data, you only need to specify the version, as follows: $ tenup64 -u [USERNAME] -g beta C [CONNSTR] [PATH_TO_NEW_TABLE] [QUERY] The example above tells TenUp not to use the standard gateway logic but to instead create a session in 1010data version The current beta versions of 1010data begin at version To specify a custom corporate gateway, you need to provide the fully qualified URL, as follows: $ tenup64 -u [USERNAME] -g -C [CONNSTR] [PATH_TO_NEW_TABLE] [QUERY] If you need to use a corporate gateway and also want to use a specific version of 1010data you can combine the previous examples, as follows: $ tenup64 -u [USERNAME] -g -C [CONNSTR] [PATH_TO_NEW_TABLE] [QUERY] Testing your setup One step you may want to take before trying your first extract and load is to perform a "dry run" using the - D option, as shown in the command below: $ tenup64 -D -u [1010data_USERNAME] -C [CONNSTR] [PATH_TO_NEW_TABLE] "SELECT * FROM [TABLE_NAME];" The command above validates that the connection information you provided works for both the data source and 1010data, but the command does not actually extract and load any data.

16 TenUp User's Guide Connection Management 16 Connection Management This section contains the various configuration options for TenUp connectivity. TenUp provides functionality for terminating sessions, keeping sessions alive and connecting to 1010data from behind corporate proxy servers. Basic connectivity options There are many options available in TenUp for managing connectivity. The two most basic are the -l and -L options. The -l option tells TenUp to logout of its 1010data session before the program exits: $ tenup64 -u [USERNAME] -C [CONNSTR] -l Note: The above is the default behavior for TenUp. In the above example, TenUp will complete its extract and load process and then logout of 1010data. Conversely, you can also tell TenUp to stay connected to its current session and then exit after the extract and load process is complete: $ tenup64 -u [USERNAME] -C [CONNSTR] -L In the above example, TenUp will exit after it completes the extract and load process but your User ID will remain logged in to 1010data. TenUp also provides an option for outputting the current session ID and password to the console: $ tenup64 -u [USERNAME]-S Using the -S option will output your session ID and your encrypted password. This information can be used to initiate a TenUp transaction using an existing session ID and encrypted password: $ tenup64 -u [USERNAME] -C [CONNSTR] -s [TENTENSID] Note: The above two options will not logout of a 1010data session by default. Terminating and possessing an existing session You may encounter a situation in which the account you used to connect to 1010data has already established a connection outside of your TenUp session. This could happen if you recently logged into 1010data via another interface and didn't log out, or if you use a shared account within your work group. If you try to load data with TenUp while your account is already logged in, TenUp will return an error: Error Already logged in In this event, you will need to end or possess the existing session before TenUp can use the account to connect to 1010data. You do this by killing the existing session with the -k option, or by possessing the existing seesion with the -K option. To terminate (or 'kill') the existing session, apply the -k option as follows: $ tenup -k -u [USERNAME] -C [CONNSTR] [PATH_TO_NEW_TABLE] [QUERY] If you prefer to possess the existing session, us the -K command, as follows: $ tenup -K -u [USERNAME] -C [CONNSTR] [PATH_TO_NEW_TABLE] [QUERY] The command above tells TenUp to use the existing session instead of terminating it and starting a new one. Connecting from a corporate proxy If you connect to the internet from behind a corporate proxy server, you can tell TenUp to connect through the proxy. TenUp provides this functionality with the -P option, followed by a space and the proxy information, fomatted as follows:

17 TenUp User's Guide Connection Management 17 $ tenup -u [USERNAME] -P [PROXYUSERPASSWORD]@ -C [CONNSTR] [PATH_TO_NEW_TABLE] [QUERY] This command routes the TenUp connection through the specified proxy server, then connects to 1010data and runs your query as normal. If you do not know the correct proxy server information to provide, ask your network administrator. Note: Not all proxy servers require a username and password. Check with your network administrator to find out exactly what information you need to connect via your company's proxy server. SAM pools If you access 1010data as part of a SAM pool, you need to use different login credentials than if you had your own, unique 1010data ID. SAM stands for: Shared Access Management. SAM pools are generally used to share a single set of login credentials with a larger pool of actual users. Specifying your SAM pool is done with the --pool option in TenUp. It takes the Pool Name as it's only parameter. However, you will still need to pass TenUp a username and password, as follows: $ tenup --pool=[sampoolname] -u [POOLOWNERNAME] -p [POOLOWNERPASSWORD] -C [CONNSTR] [PATH_TO_NEW_TABLE] [QUERY] This tells TenUp to login to 1010data with the first free account in the SAM pool. If no accounts are presently available, TenUp will return an error stating that the system is busy.

18 TenUp User's Guide Server-Side Management 18 Server-Side Management This section contains information about specifying how data is saved to the 1010data system. When you load data with TenUp, you can overwrite a table or create a new one. You can control metadata about the table, such as the name, title and subtitle. You can also control who has permission to view the new table. This section will cover TenUp operations that control server-side data. Overwriting an Existing Table In addition to creating a brand new table, you can overwrite an existing table on 1010data with TenUp. To overwrite a table, use the -y option. Here's an example: Note: You must own a table to overwrite it. $ tenup64 -u [USERNAME] -C [CONNSTR] -y This command will overwrite an existing 1010data table with the results of the query provided. If you do not use the -y option, and the name of the table you're trying to write already exists, TenUp will return an error. Appending to an existing table TenUp also has an option for appending new data to an existing table (as opposed to creating a new table or overwriting and existing table). An append operation is performed with the -a option: Note: You must own a table to append to it. $ tenup64 -u [USERNAME] -p [PASSWORD] -C [CONNSTR] [PATH_TO_EXISTING_TABLE] [QUERY] -a Note that instead of specifying the location of a new table to be created, an existing table is specified. Assuming the table exists, TenUp will append to it automatically. Note: To append data to a table, the data being appended must have the same number of columns, with the same column names, that are of the same type as the existing table. Controlling new table permissions You can also control the access permissions to the new tables you create with TenUp interactions. By default, all new tables you save are private and can only be seen by you. You can change the permissions of your new table with the -Y option. To set the permission of a table to those of its parent folder, use the -Y command followed by the dot character (.), as follows: $ tenup64 -u -[USERNAME] -C [CONNSTR] [PATH_TO_NEW_TABLENAME] [QUERY] -Y. This command applies the permissions of the parent folder to the specified table. Finally, to grant access to specific users to a new table you use the -Y option and provide usernames and group names as the arguments, as follows: $ tenup64 -u [USERNAME] -C [CONNSTR] [PATH_TO_NEW_TABLENAME] [QUERY] -Y [USER_1,USER_2,USER_3...USER_n] This command grants permission to view the table only to the users specified in the command. The usernames should be separated by commas (, ) with no spaces between each comma and the next username. Notice that as the owner of the table, you do not need to specify yourself.

19 TenUp User's Guide Memory Management 19 Memory Management This section contains important memory considerations to keep in mind when using TenUp to load data from flat files or ODBC compliant databases. Whatever computer TenUp is run on acts as an intermediary between the original data source and 1010data. This means that your computer's memory (RAM) is used as a staging area as data is pulled from the origin source of data and loaded into 1010data. TenUp provides several options to make sure your computer doesn't have issues during the extract and load process. This section details the options and best practices for managing local memory while using TenUp. Memory management and loading from flat files Flat files are read from disk into memory and then uploaded into 1010data. These files do not need to be on your local machine, but they must be available to your local file browser. To increase or decrease the impact of loading from a flat file, you need to change the setting on the upload "window size". The "window size" in TenUp is the amount of data held in TenUp's buffer between each load task. The -W option defaults at 16 MB for flat file uploads. $ tenup64 -u [USERNAME] -p [FLAT_FILE_NAME] -W [WINDOW_SIZE] Note: A small subset of users may have to account for the effect of reading data from a file over the network. Memory management and loading from ODBC-compliant databases TenUp uses the local memory of the computer it's running on to store the extracted data and prepare it to be loaded in 1010data. The following options enable you to change the amount of data being read from the database into memory before being sent to 1010data. Setting the Number of Rows The -R option gives you control over how much data is requested from the data source of origin. By default, the number of rows stored in local memory is set to This number is very small compared to the usual size of data tables handled by 1010data, but it is a safe starting point. Before raising the limit, keep in mind that the number of rows aren't the only important factor in how much memory is taken up. The number of columns, and the width of those columns, are also factors in the amount of memory that will be used by the data. Testing may be necessary to know the optimum size of the window for a given table size and origin database, but you can usually safely raise the window size to rows without experiencing issues. This is done as follows: $ tenup64 -u [USERNAME] -p [PASSWORD] -C [CONNSTR] [PATH_TO_NEW_TABLE] [QUERY] -R [NUMBER_OF_ROWS] Here is an example with a value provided for the [NUMBER_OF_ROWS] variable: $ tenup64 -u [USERNAME] -p [PASSWORD] -C [CONNSTR] [PATH_TO_NEW_TABLE] [QUERY] -R Note: The value provided must be a whole number (integer) and must not contained any punctuation such as commas or periods. Setting the upload window size The "window size" in TenUp is the amount of data held in TenUp's buffer between each load task. This setting is controlled with the -W option, which can also be used for flat file uploads.-w is intended to be used when -R is changed to be either very small or very large. If R is set to be very small, then

20 TenUp User's Guide Memory Management 20 W will prevent excessive round trip communications with 1010data. If R is set to be very large, it will prevent tenup from sending too much data to 1010data and prevent errors from occurring. The default value for -W is $ tenup64 -u [USERNAME] -p [PASSWORD] -C [CONNSTR] [PATH_TO_NEW_TABLE] [QUERY] -W [MAX_UPLOAD_ROWS] $ tenup64 -u [USERNAME] -p [PASSWORD] -C [CONNSTR] [PATH_TO_NEW_TABLE] [QUERY] -R It should be noted that it is possible to have -R and -W set to different values. However, 1010data strongly recommends that these values be the same. By default, both settings are set to Setting the maximum column size allowed In some databases there is no way to get the maximum display size of a column. The -m option sets the maximum number of display characters permissible in a single variable-width column when loading data. The default value for loading from an ODBC database is The default for data loaded from flat files is This is the syntax: $ tenup64 -u [USERNAME] -p [PASSWORD] -C [CONNSTR] [PATH_TO_NEW_TABLE] [QUERY] -m [MAX_CHARACTERS_IN_COLUMN] And here is the same example with a value provided for -m $ tenup64 -u [USERNAME] -p [PASSWORD] -C [CONNSTR] [PATH_TO_NEW_TABLE] [QUERY] -m 2048

21 TenUp User's Guide Segmentation Settings 21 Segmentation Settings Segmentation is a vital aspect of how the 1010data Insights Platform operates. This section covers specifying segments using TenUp. Segments are how data is broken up into files on the 1010data servers, and it is done specifically to ensure that all like-values for a given segmentation arrangement are stored together. For instance, if a table is segmented first by date, then all records for each date in the table are guaranteed to be stored in the same logical space. Note that many dates may be contained in a single logical storage space. However, segmentation makes sure that a single date cannot be stored separately. Segmentation is very important for using 1010data's most powerful features, such as g_functions and time-series analysis. TenUp provides several options for specifying the segmentation of your new table. To begin, TenUp provides an option for specifying the target of the segments of a given table in terms of the number of records in each segment. 1010data servers can handle segments of more than 8 million rows. However, 3-5 million row segments are the most common. To specify the segment size of a table, use the -b option: $ tenup64 -u [USERNAME] -p [PASSWORD] -C [CONNSTR] [PATH_TO_NEW_TABLE] [QUERY] -b [NUMBER_OF_ROWS_IN_SEGMENT] The default segment size for extract and load jobs that do not specify segment size is Note that some segmentation strategies may cause the final segment size to be different than that specified with the -b option. Segmentation columns When loading data with TenUp you may choose to segment the data by specifying a column or column using either segby or sortseg. Segby is a specific segmentation where the values in the columns listed in the segby statement govern the way the rows are split. If a table is segby a given column, no unique value of the column can be found in more than one segment. This allows for quick computation of aggregate/grouping functions (ex: sums, averages, etc.) on that column since we only need to look in one file for each unique value of that column. To achieve this, the rows of the table frequently need to be reordered. So, if a table is segmented by date, then all values for a given date must be co-located in the same segment. The -j or --segby option is used for specifying segmentation columns: Note: If you use the -j/--segby option you must specify at least one column. $ tenup64 -u [USERNAME] -p [PASSWORD] -C [CONNSTR] [PATH_TO_NEW_TABLE] [QUERY] -b j [SEGBY_COL1],[SEGBY_COL2],...[SEGBY_COLn] It is also possible to specify how the table is to be sorted within a segment after the data is loaded in 1010data. Sort order is specified with the -$ option: $ tenup64 -u [USERNAME] -p [PASSWORD] -C [CONNSTR] [PATH_TO_NEW_TABLE] [QUERY] -b j date,transid -$ [SORT_COL1],[SORT_COL2],...[SORT_COLn] Note that the table is sorted in reverse order of the column names provided. In the instance above, the table will first be sorted by [SORTCOLn], then by [SORT_COL2], and finally by [SORT_COL1]. A sortseg is a special kind of segmentation that stipulates that each group of like values in a segment starts with the lowest or most recent value and ends with the highest or latest value. sortseg improves performance for some kinds of analysis by enabling the system to assume the start and end points and skip past entire groups of like values or even entire segments. sortseg is controlled with the -J option: $ tenup64 -u [USERNAME] -p [PASSWORD] -C [CONNSTR] [PATH_TO_NEW_TABLE] [QUERY] -b J [SORTSEG_COL1], [SORTSEG_COL2],...[SORTSEG_COLn]

22 TenUp User's Guide Segmentation Settings 22 Indexing columns TenUp provides an option for indexing specific columns after they are loaded in 1010data. Indexing improves the performance of some kinds of analysis by mapping the location of specific values in a column based on where they are located in a segment. Indexing is performed with the -X option: $ tenup64 -u [USERNAME] -p [PASSWORD] -C [CONNSTR] [PATH_TO_NEW_TABLE] [QUERY] -X [INDEX_COL1],[INDEX_COL2],...[INDEX_COLn] Time-series If you are moving from the Powerloader to TenUp to load data, you can mimic the functionality of the timeseries tag (<ts>) using the --time-series or -+ options. This option sorts the segments by TIME_COL and generates time-series metadata. This option can only be used in conjunction with segby (-j/--segby) and sortseg. It cannot be used with sort (-$/--sort). $ tenup64 -u [USERNAME] -p [PASSWORD] -C [CONNSTR] [PATH_TO_NEW_TABLE] [QUERY] -j [SEGBY_COL1],...[SEGBY_COLn] --time-series=[time_col] Linking to an existing table in 1010data TenUp provides an option for linking to an existing table during a load process. Linking with TenUp actually performs a prelink between the table that is being loaded and the foreign table. A prelink simply creates a map between the two tables, based on the columns specified in the list of local columns and the list of foreign columns. The map is then saved with the new table. In TenUp, the -, option specifies a link: $ tenup64 -u [USERNAME] -C [CONNSTR] [PATH_TO_NEW_TABLE] [QUERY] -, [PATH_TO_FOREIGN_TABLE]:[LOCAL_COL1],[LOCAL_COL2],...[LOCAL_COLn]: [FOREIGN_COL1], [FOREIGN_COL2]...[FOREIGN_COLn] TenUp also provides an optional argument for the -, switch where a list of columns in the foreign table to denormalize into the loaded table may be specified, as follows: $ tenup64 -u [USERNAME] -C [CONNSTR] [PATH_TO_NEW_TABLE] [QUERY] -, [PATH_TO_FOREIGN_TABLE]:[LIST_OF_LOCAL_COLUMNS]:[LIST_OF_FOREIGN_COLUMNS]: [LIST_OF_COLS_TO_DENORM] Note: When a list of columns to denormalize is included, the foreign table is not pre-linked to the base table.

23 TenUp User's Guide Providing Metadata 23 Providing Metadata TenUp provides numerous options for specifying the metadata of a table after it is uploaded to 1010data. This section details how to specify metadata for columns and tables. TenUp uses only one source of metadata for each option, such as --column-names. Different options may be taken from different sources of metadata. If multiple sources of metadata are provided, TenUp uses the one with the highest priority. Metadata in the command line takes priority over any metadata provided in either a table tree or a spec tree (flat file only), and table tree metadata takes priority over spec tree metadata. If no additional sources of metadata are provided, TenUp uses the metadata from the source database if it is available. If you're loading data from a flat file, the metadata can be provided in a table tree or spec tree. ODBC-compliant data sources cannot use the spec tree. Additionally, ODBC-compliant data sources do not have some of the kinds of table metadata that the 1010data Insights Platform provides. For example, while ODBC-compliant data sources are familiar with the idea of a column name, they do not have an equivalent to Insights Platform column labels. Use the options in this section to more precisely control the resultant table's metadata when it appears in the Insights Platform. Table information By default, if no additional information is provided to TenUp, the table name of the resultant table is the path to the table that was provided in the TenUp command. However, you can also provide a table title with the -t option: $ tenup64 -u [USERNAME] -p [PASSWORD] -C [CONNSTR] [PATH_TO_NEW_TABLE] [QUERY] -t "My Table" You can also provide both a short description and a long description for the table with the -n and -N options respectively: $ tenup64 -u [USERNAME] -p [PASSWORD] -C [CONNSTR] [PATH_TO_NEW_TABLE] [QUERY] -n [SHORT_DESCRIPTION] -N [LONG_DESCRIPTION] In the 1010data web-based GUI, you can see the table information for a table by clicking Info > About this table: Column information Much like the table metadata, you can also provide column metadata for each column of a table. By default, column names in the new 1010data table that is created will be the same as the column names in the original data source (only TenUp will force them into an all lower-case format). If you want to change the column names from those of the original data source, use the -h option: $ tenup64 -u [USERNAME] -p [PASSWORD] -C [CONNSTR] [PATH_TO_NEW_TABLE] [QUERY] -h [COL1_NAME],[COL2_NAME],...[COLn_NAME] Additionally, TenUp provides an option for creating column headings. By default, the column heading is the same as the column name. Specify new column headings with the -H option:

24 TenUp User's Guide Providing Metadata 24 $ tenup64 -u [USERNAME] -p [PASSWORD] -C [CONNSTR] [PATH_TO_NEW_TABLE] [QUERY] -H [COL1_HEADING],[COL2_HEADING],...[COLn_HEADING] You can also provide a description for each column with the -E option: $ tenup64 -u [USERNAME] -p [PASSWORD] -C [CONNSTR] [PATH_TO_NEW_TABLE] [QUERY] -E [COL1_DESC],[COL2_DESC],...[COLn_DESC] Column types and formats Column types are simplified in 1010data compared to the ODBC specification. During a normal extract and load process TenUp makes its best effort to translate both the column types and formats from the data source to 1010data equivalents. If you find that this translation isn't meeting the needs of the job, or would simply like to alter the way the system interprets the types and formats, TenUp provides options. Change the type of the data in a given column using the -i option: $ tenup64 -u [USERNAME] -p [PASSWORD] -C [CONNSTR] [PATH_TO_NEW_TABLE] [QUERY] -i [COL1_TYPE],[COL2_TYPE],...[COLn_TYPE] Valid values to pass to the -i option are: i - integer values f - floating point numbers a - alphanumeric text j - 64-bit integers (bigint) After -i, provide a comma separated list of types, e.g. -i i,i,a,f,f,j Users should specify j to load as a big integer. When loading a SQL_BIGINT column, any provided type will take priority over --use-long-ints. For example, if a user specifies -i i (integer), the column as an integer column, even if --use-long-ints is specified. The --use-doubles loads SQL_BIGINT columns as double precision floating point columns. This is the current default behavior if --column-types is not used. Column formats in 1010data are controlled with the -f option. You may want to pay special attention to this setting if you have unusually wide or variable-width columns in the original data source: $ tenup64 -u [USERNAME] -p [PASSWORD] -C [CONNSTR] [PATH_TO_NEW_TABLE] [QUERY] -f [COL1_FORMAT],[COL2_FORMAT],...[COLn_FORMAT] To see the available options for column formats, see our User's Guide sections on data types and display formats. You can override the default column display width with --default-column-width. The width is the number of characters to display in the column (1-100). The default width is 12. If the contents of a cell exceeds the specified width, the value is truncated. In this case, the number of visible characters is reduced by two, and " >" is appended to the value. For example, for width:4, the string Example would be displayed as Ex >. Clicking on the > displays the full contents of the cell in a separate window. To set a new column width, use the --default-column-width switch followed by the new[column_width]. $ tenup64 -u [USERNAME] -p [PASSWORD] -C [CONNSTR] [PATH_TO_NEW_TABLE] [QUERY] --default-column-width [COLUMN_WIDTH] When uploading from a flat file, the --max-column-size option can be used to override the default mask width. The default is 5000 characters. The --skip-rows option omits some number of leading rows from the flat file.

25 TenUp User's Guide Providing Metadata 25 Specifying information in a file If you're making a larger adjustment to your uploaded table, providing the specifications in the command the command line could be cumbersome. You can use the table tree or spec to provide specifications in a file. These external files contain information about the table and columns. The table tree specifies information about the destination table, and it can be used with an ODBC-compliant database or flat file upload. The spec tree can specify information about the source and the destination table, and it can only be used with flat file uploading. To specify a table tree with an ODBC-compliant database as the source: $ tenup64 -u [USERNAME] -p [PASSWORD] -C [QUERY] To specify a table tree with a flat file as the source: $ tenup64 -u [USERNAME] -p [FLAT_FILE_NAME] For more information on the 1010data table tree, see Table Tree in the 1010data API Reference Manual. To specify a spec file, which can only be used with flat file uploads: $ tenup64 -u [USERNAME] -p To specify both a table tree and a spec tree, which can only be used with flat file uploads: $ tenup64 -u [USERNAME] If you're using a spec tree, the [FLAT_FILE_NAME] is specified in the spec tree. For more information see, Spec tree tags on page 34.

26 TenUp User's Guide Advanced Topics: Automating TenUp 26 Advanced Topics: Automating TenUp Since the most powerful uses of TenUp all have to do with automation, in a way all the previous sections of this manual have led us to this moment. Here, we will learn how to customize your TenUp settings so that it knows certain information about your account. This will allow you to include less information in each command. Finally, we will look at how to use TenUp to build and run shell scripts. In Windows, a shell script is called a Batch File. Batch files always end in a.bat file extension. In MacOS and Linux, shell scripts go by a few different names, but both Shell Script or Bash Script are appropriate. Shell script files in MacOS and Linux end in the.sh file extension. Shell scripts are extremely useful. They allow you to write many commands in a single file that will be executed in the order in which they appear. Often, these commands work together, each command using the results of the one that preceded it to accomplish a task that would be difficult or impossible with a single command. However, you can create a shell script that runs a series of completely unrelated commands as well. This section will detail the concepts and best practices needed for writing shell scripts that contain multiple TenUp commands. Setting Environment Variables While reading this manual you might have noticed that most, if not all, TenUp commands contain the same information. For instance, every example TenUp command in this guide contains -u [USERNAME] as part of the command. This is fine if you're only sending a single command. But what if you want to use TenUp to send 10 queries to 1010data? Or 100? In such a case, the easiest thing to do is set your username once so you don't have to include it in every single command. To help you do this, TenUp uses numerous Environment Variables that you can set to valid 1010data values as shown in the table below. This means that once you've set these values, you no longer have to include them in subsequent TenUp commands. The following table describes the available TenUp environment variables: Table 1: Environment Variables Environment Variable Equivelent TenUp Option/ Switch Description TENTENGW -g Sets your 1010data gateway. You need to specify this value if you use a custom corporate gateway to access 1010data, or if you are interested in using specific version of 1010data. The default value is: www2.1010data.com/cgi-bin/gw.k TENTENUID -u Your 1010data username. There is no default value for this variable. TENTENPW -p Your 1010data password. No default for this variable. TENTENPROXY -P Sets the proxy server for your TenUp session. No default for this variable. TENTENSID -s Your 1010data Session ID. A unique identifier assigned to you when you begin your 1010data

27 TenUp User's Guide Advanced Topics: Automating TenUp 27 Environment Variable Equivelent TenUp Option/ Switch Description session. This can be used as an alternate method to authenticate with 1010data, so long as you have the same session open. We will go into greater detail about Session IDs in the next session. Setting environment variables in windows In the Windows Command Prompt, you set a variable with the set keyword. The following command sets the default gateway for your 1010data session in Command Prompt: C:\> set TENTENGW= Once you send this command to Windows, this environment variable will be available to TenUp as long as you keep your command prompt open. If you include this line as part of a batch script, then the environment variable will be set every time you run the script. You can also permanently add any evironment variable to your Command Prompt by going to the Environment Variables dialog. This is found on the Advanced tab of the System Properties dialog. Setting environment variables in MacOS and in Linux In a Linux or Mac shell, you set the environment variables using the export keyword, as follows: $ export TENTENGW= You can also add environment variables to your shell environment permanently by adding an environment variable, like the example above, to your.profile or.bashrc file. No matter which operating system you're working with, you should get in the habit of setting your environment variables at the very beginning of your shell script. Session Management This section contains information about session management while using TenUp. Possessing and terminating an existing session When using TenUp, and especially when writing shell scripts that contain TenUp commands, it is important to effectively manage your session status. In some cases you will want to start a new session, while in others you will want to reuse a session. TenUp provides options for all these situations. This section focuses on two options that will help you manage your TenUp sessions. But first, let's clarify what a session is exactly. When you login to 1010data you are assigned a unique session ID. This ID has many purposes, one of which is to make sure that a single username can't login to 1010data more than once at the same time. If you try to login to 1010data while you already have a session ID associated with your username, you have two options: terminate the existing session and start a new one, or possess the existing session. Possessing an existing session Resuming a session means you don't need to login to 1010data again. Logging in takes both time and server resources, so ideally you will only login to 1010data once during a working session. Note: If you use the -K option and no session exists for your account then a new session will be created. TenUp allows you to resume a session by "posessing" it. To possess a session you will include the -K option in your TenUp command, as follows: