DataLoader/MX Reference Manual

Transcription

1 DataLoader/MX Reference Manual Abstract This manual describes the features and functionality of the DataLoader/MX product, a tool to load HP NonStop SQL/MX, SQL/MP, and Enscribe databases. Product Version DataLoader/MX H01 Supported Release Version Updates (RVUs) This publication supports H06.03 and all subsequent H-series RVUs until otherwise indicated by its replacement publication. Part Number Published November 2006

2 Document History Part Number Product Version Published DataLoader/MX G10 April DataLoader/MX G10 September DataLoader/MX H01 November 2006

3 DataLoader/MX Reference Manual Glossary Index Figures Tables What s New in This Manual vii Manual Information vii New and Changed Information vii About This Manual ix Audience ix Organization of This Manual Related Documentation x Notation Conventions xiii ix 1. Introduction to DataLoader/MX DataLoader/MX Features 1-1 DataLoader/MX Applications 1-1 Loading Scenario 1-2 Relationship to Other Loading Tools 1-3 Using DataLoader/MX With SQL/MP and Enscribe DataLoader/MX Components DataLoader/MX File Package 2-1 DataLoader/MX Structure 2-2 Main Logic 2-2 User Exits 2-2 Library 2-2 DataLoader/MX File System Running DataLoader/MX Syntax of DataLoader/MX 3-1 Considerations DataLoader/MX Process 3-9 Default Exit Examples 3-12 Parallelism 3-12 Creating Parallelism 3-13 Taking Advantage of Parallelism 3-13 Building Your Loading Application 3-14 Hewlett-Packard Company i

4 Contents 3. Running DataLoader/MX (continued) 3. Running DataLoader/MX (continued) Analyzing Your Configuration Specifying File-Related Options for DataLoader/MX Interpretations 4-2 BROADCAST 4-2 CSV 4-2 DUMP 4-6 EBCDIC 4-6 INDIRECT 4-7 KEYRANGE 4-8 TEXT 4-9 Modifiers 4-9 BUFSIZE 4-9 MAX 4-9 NOREWIND 4-10 NOUNLOAD 4-10 num% 4-10 NUMBUFS 4-11 PERSIST 4-11 RECFORM 4-11 SUBTYPE 4-12 TYPE 4-12 WAIT Creating a Customized Version of DataLoader/MX User Exits 5-1 User Exit Descriptions 5-4 BUILDKEY 5-4 CHECKARG 5-5 CONVERTIT 5-5 DELETEIT 5-6 DONEWITHTRANSACTION 5-6 EXITSDESCRIPTION 5-7 GETNEXTRECORD 5-7 INITIALIZE1 5-7 INITIALIZE2 5-8 INSERTIT 5-8 INSTRUCTIONS 5-9 DataLoader/MX Reference Manual ii

5 Contents 5. Creating a Customized Version of DataLoader/MX (continued) 5. Creating a Customized Version of DataLoader/MX (continued) MISC1, MISC2, MISC3, and MISC4 5-9 NEWTRANSACTION 5-10 NEXTINDIRECTFILE 5-10 SKIPPING 5-10 STATISTICSTIME 5-11 TERMINATING 5-11 TERMINATING T0351H01_DEFAULTEXITS_C and T0351H01-DEFAULTEXITS-COBOL 5-11 UPDATEIT 5-12 Default User Exits 5-13 DataLoader/MX Library 5-14 Data Conversion 5-14 Sequential I/O 5-19 User Messages C 5-21 User Messages COBOL 5-24 SQL Error Checking and Statistics Maintenance 5-26 Generating Test Data 5-28 Sorting With DTLSort 5-35 Miscellaneous Utility Procedures 5-39 Building a Customized Version of DataLoader/MX 5-41 Example DataLoader/MX Examples Single Source Parallel Loading 6-1 Single Source Parallel Maintenance 6-4 Multiple Source Parallel Loading 6-6 Multiple Source Parallel Maintenance 6-9 Combined Usage With FTP Recovery Strategies Why Is Recovery Important? 7-1 Defining Load 7-1 Design Time Considerations 7-2 Tools 7-2 Some Basic Rules 7-2 Simple Recovery Approaches 7-3 Potential Problems 7-3 The Two Main Approaches 7-3 DataLoader/MX Reference Manual iii

6 Contents 7. Recovery Strategies (continued) 7. Recovery Strategies (continued) Restarting From An Unknown State 7-4 Restarting From a Known State 7-6 Batch Totals 7-7 Multiprocess Considerations 7-8 Parallel Considerations 7-8 A. Error and Warning Messages Conversion Routine Messages A-1 User Exit Messages A-1 File System Messages A-2 Partition Boundary Estimation Messages A-9 $RECEIVE Messages A-10 Test Data Generation Messages A-11 TMF Messages A-11 Miscellaneous Messages A-13 Warnings A-13 B. Processing Flowcharts C. C-Only Error Functions Index Figures Figure 1-1. Loading a Single Input With DataLoader/MX 1-2 Figure 2-1. Structure of DataLoader/MX 2-2 Figure 6-1. Single Source Parallel Loading 6-2 Figure 6-2. Single Source Parallel Maintenance 6-5 Figure 6-3. Multiple Source Parallel Loading 6-7 Figure 6-4. Multiple Source Parallel Maintenance 6-10 Figure B-1. Overall Processing B-1 Figure B-2. Initialization B-2 Figure B-3. Argument Processing B-3 Figure B-4. Getting Next Record B-4 Figure B-5. Processing Records B-5 Figure B-6. Processing User Exits B-6 Figure B-7. Post Record Processing B-7 Figure B-8. Normal Termination B-8 Figure B-9. Abnormal Termination B-9 DataLoader/MX Reference Manual iv

7 Contents Tables Tables Table 2-1. DataLoader/MX I/O Blocking Behavior 2-5 Table 5-1. Exits and Their Parameters 5-2 Table 5-2. Parameter -X Exit Codes 5-2 Table 5-3. Default User Exits 5-13 DataLoader/MX Reference Manual v

8 Contents DataLoader/MX Reference Manual vi

9 What s New in This Manual Manual Information Abstract DataLoader/MX Reference Manual This manual describes the features and functionality of the DataLoader/MX product, a tool to load HP NonStop SQL/MX, SQL/MP, and Enscribe databases. Product Version DataLoader/MX H01 Supported Release Version Updates (RVUs) This publication supports H06.03 and all subsequent H-series RVUs until otherwise indicated by its replacement publication. Part Number Published November 2006 Document History Part Number Product Version Published DataLoader/MX G10 April DataLoader/MX G10 September DataLoader/MX H01 November 2006 New and Changed Information Change to the H06.07 RVU: Changed the cobol compiler as ecobol on page Changes to the G06.24 RVU Deleted the mxcmp command, under Building a Customized Version of DataLoader/MX on page 5-40 because the use of the mxcmp command for module compilation is no longer supported. Updated the information under MX MP on page 5-41 that it is no longer necessary to compile a module if you specify the object file name directly. DataLoader/MX Reference Manual vii

10 What s New in This Manual Changes to the G06.24 RVU DataLoader/MX Reference Manual viii

11 About This Manual This manual describes DataLoader/MX software, which is used to load and maintain NonStop SQL/MX, SQL/MP, and Enscribe databases. It contains reference information for the product and describes its features and syntax. Audience This manual is written for database administrators and programmers. End users can also use this manual. Readers of this manual should know the fundamentals of: SQL/MX NonStop operating system HP NonStop Open System Services (OSS) The SQL/MX conversational interface (MXCI) Organization of This Manual Section Title This section... 1 Introduction to DataLoader/MX Describes the features and uses of DataLoader/MX, a typical loading scenario, and DataLoader/MX s relationship to other loading tools. 2 DataLoader/MX Components Describes the file package delivered with DataLoader/MX as well as its structure and file system. 3 Running DataLoader/MX Explains how to run DataLoader/MX. It describes parameters defined by HP and gives examples of the use of these parameters. It also discusses parallelism and how to build a load application to run DataLoader/MX most effectively. 4 Specifying File-Related Options for DataLoader/MX 5 Creating a Customized Version of DataLoader/MX Explains the two types of file-related options for DataLoader/MX: interpretations and modifiers. It further describes how to control the way DataLoader/MX reads, converts, and processes a file s contents. Explains how to create a customized version of DataLoader/MX by combining user exits and utility routines. DataLoader/MX Reference Manual ix

12 About This Manual Related Documentation Section Title This section... 6 DataLoader/MX Examples Provides examples to demonstrate DataLoader/MX. These examples represent the four basic loading scenarios from which nearly all real loading scenarios are derived, as well as their use with FTP. 7 Recovery Strategies Discusses strategies for planning DataLoader/MX run scenarios so that you will be able to rerun DataLoader/MX and successfully load your database if you experience a failure during the load operation. A Error and Warning Messages Lists messages that relate to DataLoader/MX along with their cause and any recovery steps that should be taken. Messages are grouped by the type of operation that produces them. B Processing Flowcharts Provides diagrams that describe the processing flow for DataLoader/MX. C C-Only Error Functions Describes C-only SQL error-checking functions. These functions work for C programs, but do not work for COBOL programs. They are retained for compatibility with earlier versions of DataLoader, but you should not use them for new customizations. Related Documentation This manual is part of the SQL/MX library of manuals, which includes: Introductory Guides SQL/MX Comparison Guide for SQL/MP Users SQL/MX Quick Start Reference Manuals SQL/MX Reference Manual SQL/MX Connectivity Service Administrative Command Reference DataLoader/MX Reference Manual Describes SQL differences between SQL/MP and SQL/MX. Describes basic techniques for using SQL in the SQL/MX conversational interface (MXCI). Includes information about installing the sample database. Describes the syntax of SQL/MX statements, MXCI commands, functions, and other SQL/MX language elements. Describes the SQL/MX administrative command library (MACL) available with the SQL/MX conversational interface (MXCI). Describes the features and functions of the DataLoader/MX product, a tool to load SQL/MX databases. DataLoader/MX Reference Manual x

13 About This Manual Related Documentation SQL/MX Messages Manual SQL/MX Glossary Programming Manuals SQL/MX Programming Manual for C and COBOL SQL/MX Programming Manual for Java SQL/MX Guide to Stored Procedures in Java Specialized Guides SQL/MX Installation and Management Guide SQL/MX Query Guide SQL/MX Data Mining Guide SQL/MX Queuing and Publish/Subscribe Services SQL/MX Report Writer Guide SQL/MX Connectivity Service Manual Online Help The SQL/MX Online Help consists of: SQL/MX Reference Help SQL/MX Messages Help SQL/MX Glossary Help Describes SQL/MX messages. Defines SQL/MX terminology. Describes how to embed SQL/MX statements in ANSI C and COBOL programs. Describes how to embed SQL/MX statements in Java programs according to the SQLJ standard. Describes how to use stored procedures that are written in Java within SQL/MX. Describes how to plan, install, create, and manage an SQL/MX database. Explains how to use installation and management commands and utilities. Describes how to understand query execution plans and write optimal queries for an SQL/MX database. Describes the SQL/MX data structures and operations to carry out the knowledge-discovery process. Describes how SQL/MX integrates transactional queuing and publish/subscribe services into its database infrastructure. Describes how to produce formatted reports using data from a NonStop SQL/MX database. Describes how to install and manage the SQL/MX Connectivity Service (MXCS), which enables applications developed for the Microsoft Open Database Connectivity (ODBC) application programming interface (API) and other connectivity APIs to use SQL/MX. Overview and reference entries from the SQL/MX Reference Manual. Individual messages grouped by source from the SQL/MX Messages Manual Terms and definitions from the SQL/MX Glossary. DataLoader/MX Reference Manual xi

14 About This Manual Related Documentation The following manuals are part of the SQL/MP library of manuals and are essential references for information about SQL/MP Data Definition Language (DDL) and SQL/MP installation and management: Related SQL/MP Manuals SQL/MP Reference Manual SQL/MP Installation and Management Guide Describes the SQL/MP language elements, expressions, predicates, functions, and statements. The following documentation is helpful for understanding the concepts and terminology of this manual: Other Related Documentation Dataloader/MP Reference Manual FastSort Manual File Utility Program (FUP) Reference Manual Guardian Procedure Calls Reference Manual Open System Services User s Guide Describes how to plan, install, create, and manage an SQL/MP database. Describes installation and management commands and SQL/MP catalogs and files. Describes the features and functionality of the DataLoader/MP and native mode (NM) DataLoader/MP products, tools to load SQL/MP databases. Describes FastSort, the sort-merge utility to use when you initiate NonStop SQL/MP queries that sort entries or load data. Provides an overview of the File Utility Program (FUP) and the detailed syntax for its commands. Describes how to manage disk files, nondisk devices (printers, terminals, and tape drives), and processes (programs) running on an HP NonStop Server. Describes the syntax of most of the Guardian procedure calls. Contains reference material describing the OSS environment and the OSS shell (OSH). DataLoader/MX Reference Manual xii

15 About This Manual Notation Conventions Notation Conventions General Syntax Notation The following list summarizes the notation conventions for syntax presentation in this manual. UPPERCASE LETTERS. Uppercase letters indicate keywords and reserved words; enter these items exactly as shown. Items not enclosed in brackets are required. For example: MAXATTACH lowercase italic letters. Lowercase italic letters indicate variable items that you supply. Items not enclosed in brackets are required. For example: file-name [ ] Brackets. Brackets enclose optional syntax items. For example: TERM [\system-name.]$terminal-name INT[ERRUPTS] A group of items enclosed in brackets is a list from which you can choose one item or none. The items in the list may be arranged either vertically, with aligned brackets on each side of the list, or horizontally, enclosed in a pair of brackets and separated by vertical lines. For example: LIGHTS [ ON ] [ OFF ] [ SMOOTH [ num ] ] K [ X D ] address-1 { } Braces. A group of items enclosed in braces is a list from which you are required to choose one item. The items in the list may be arranged either vertically, with aligned braces on each side of the list, or horizontally, enclosed in a pair of braces and separated by vertical lines. For example: LISTOPENS PROCESS { $appl-mgr-name } { $process-name } ALLOWSU { ON OFF } Vertical Line. A vertical line separates alternatives in a horizontal list that is enclosed in brackets or braces. For example: INSPECT { OFF ON SAVEABEND } DataLoader/MX Reference Manual xiii

16 About This Manual Notation for Messages Ellipsis. An ellipsis immediately following a pair of brackets or braces indicates that you can repeat the enclosed sequence of syntax items any number of times. For example: M address-1 [, new-value ]... [ - ] { }... An ellipsis immediately following a single syntax item indicates that you can repeat that syntax item any number of times. For example: "s-char..." Punctuation. Parentheses, commas, semicolons, and other symbols not previously described must be entered as shown. For example: error := NEXTFILENAME ( file-name ) ; LISTOPENS SU $process-name.#su-name Quotation marks around a symbol such as a bracket or brace indicate the symbol is a required character that you must enter as shown. For example: "[" repetition-constant-list "]" Item Spacing. Spaces shown between items are required unless one of the items is a punctuation symbol such as a parenthesis or a comma. For example: CALL STEPMOM ( process-id ) ; If there is no space between two items, spaces are not permitted. In the following example, there are no spaces permitted between the period and any other items: $process-name.#su-name Line Spacing. If the syntax of a command is too long to fit on a single line, each continuation line is indented three spaces and is separated from the preceding line by a blank line. This spacing distinguishes items in a continuation line from items in a vertical list of selections. For example: ALTER [ / OUT file-spec / ] CONTROLLER [, attribute-spec ]... Notation for Messages The following list summarizes the notation conventions for the presentation of displayed messages in this manual. Nonitalic text. Nonitalic letters, numbers, and punctuation indicate text that is displayed or returned exactly as shown. For example: Backup Up. DataLoader/MX Reference Manual xiv

17 About This Manual Change Bar Notation lowercase italic letters. Lowercase italic letters indicate variable items whose values are displayed or returned. For example: p-register process-name [ ] Brackets. Brackets enclose items that are sometimes, but not always, displayed. For example: Event number = number [ Subject = first-subject-value ] A group of items enclosed in brackets is a list of all possible items that can be displayed, of which one or none might actually be displayed. The items in the list might be arranged either vertically, with aligned brackets on each side of the list, or horizontally, enclosed in a pair of brackets and separated by vertical lines. For example: LDEV ldev [ CU %ccu CU %... ] UP [ (cpu,chan,%ctlr,%unit) ] { } Braces. A group of items enclosed in braces is a list of all possible items that can be displayed, of which one is actually displayed. The items in the list might be arranged either vertically, with aligned braces on each side of the list, or horizontally, enclosed in a pair of braces and separated by vertical lines. For example: LBU { X Y } POWER FAIL process-name State changed from old-objstate to objstate { Operator Request. } { Unknown. } Vertical Line. A vertical line separates alternatives in a horizontal list that is enclosed in brackets or braces. For example: Transfer status: { OK Failed } % Percent Sign. A percent sign precedes a number that is not in decimal notation. The %þnotation precedes an octal number. The %Bþnotation precedes a binary number. The %Hþnotation precedes a hexadecimal number. For example: % P=%p-register E=%e-register Change Bar Notation Change bars are used to indicate substantive differences between this manual and its preceding version. Change bars are vertical rules placed in the right margin of changed portions of text, figures, tables, examples, and so on. Change bars highlight new or revised information. For example: The message types specified in the REPORT clause are different in the COBOL85 environment and the Common Run-Time Environment (CRE). DataLoader/MX Reference Manual xv

18 About This Manual Change Bar Notation The CRE has many new message types and some new message type codes for old message types. In the CRE, the message type SYSTEM includes all messages except LOGICAL-CLOSE and LOGICAL-OPEN. DataLoader/MX Reference Manual xvi

19 1 Introduction to DataLoader/MX DataLoader/MX is a software product that is used to load and maintain NonStop SQL/MP, SQL/MX, and Enscribe databases. DataLoader/MX Features DataLoader/MX takes full advantage of the parallelism of the NonStop SQL/MX database product. DataLoader/MX is designed to handle massive amounts of data with an emphasis on efficient performance. DataLoader/MX includes: Efficient handling of multiple input and output streams An internal file system that overcomes file size limits, handles a multiplicity of files, and facilitates tasks such as file concatenation Sampling of input data Determination of partition boundaries Generation of test data Data extraction and loading into data marts and workstations Customizable user exits that let you tailor the program to your specific needs A library that supports common loading and maintenance functions such as data conversion, hashing, and sequential I/O The DataLoader/MX process runs as a nonprivileged batch program on NonStop OS. If part of the loading operation is done on another platform, use a product on that platform to extract the desired data and send it to the NonStop system. DataLoader/MX Applications A typical database customer might be faced with any or these challenges: Loading very large amounts (terabytes) of data Loading large amounts (gigabytes) of data within a given time Converting data to various formats Loading data that comes from multiple data sources such as tape or across communication lines from other systems Other problems such as system balancing and recovery You can use DataLoader/MX to build loading applications quickly and easily and to solve these challenges. DataLoader/MX consists of a file processing program and a DataLoader/MX Reference Manual

20 Introduction to DataLoader/MX Loading Scenario number of source files for load customization. DataLoader/MX works in conjunction with the import utility. Loading Scenario A typical load scenario might involve a load operation from a single input source into three partitions of a table. You need a DataLoader/MX process for the input stream and a DataLoader/MX process for each load operation into a specific partition. Figure 1-1 illustrates this scenario. Figure 1-1. Loading a Single Input With DataLoader/MX B C A B C B C VDT014 The initial DataLoader/MX process, (A), reads a block of records from the input stream and sends the record to the appropriate downstream DataLoader/MX process, (B), based on the key value. DataLoader/MX process B reads blocks of records and performs any special processing that might be required, such as conversions or reformatting operations. It then writes the records to its $RECEIVE file. An import process, (C), reads the records and loads them into the designated target partition. For more information, see Section 6, DataLoader/MX Examples. DataLoader/MX Reference Manual

21 Introduction to DataLoader/MX Relationship to Other Loading Tools Relationship to Other Loading Tools DataLoader/MX is designed to complement other tools that load and maintain NonStop SQL/MX databases. For example, the NonStop SQL/MX import utility is not, in itself, parallel, but can be used in conjunction with DataLoader/MX to provide a parallel load solution. In addition, DataLoader/MX can be customized to perform data format transformations as required when using import for load operations. DataLoader/MX provides features that are similar to those provided by DataLoader/MP, an another HP tool for loading databases. DataLoader/MP is designed for the Guardian environment. It supports the maintenance of SQL/MP tables and Enscribe files. DataLoader/MX has extensions to support working with SQL/MX in the OSS environment. Also, because both DataLoader/MX and DataLoader/MP use Guardian interprocessor communication (IPC) for communicating with other processes, it is possible to create a loading scenario where both are used together. Using DataLoader/MX With SQL/MP and Enscribe DataLoader/MX supports all the features supported by DataLoader/MP. This manual uses SQL/MX and import to illustrate the features of DataLoader/MX. The same functionality can be obtained by using DataLoader/MX with SQL/MP or Enscribe and SQLCI LOAD/COPY. The data format required by LOAD/COPY is different from what is required by import. The customized user exits need to be modified to convert the input data to SQLCI LOAD/COPY format rather than the import format. DataLoader/MX supports having embedded SQL statements in the user-written exit routines. These can be either SQL/MX or SQL/MP statements. The SQL error-handling functions provided use different arguments if the customized build specifies that SQL/MP SQL compilation is to be performed. SQL/MP requires the use of SQLCODE and SQLCA, while SQL/MX works using SQLSTATE. 1-3

22 Introduction to DataLoader/MX Using DataLoader/MX With SQL/MP and Enscribe 1-4

23 2 DataLoader/MX Components This section describes the file package, structure, and file system of DataLoader/MX. DataLoader/MX File Package DataLoader/MX is delivered with the object file, dataload, as well as with text files with prototypes for its utility procedures. You can copy and modify these procedures to control processing with your own version of DataLoader/MX. The DataLoader/MX package consists of these files: dataload dtload.h sexitsc.c sexitscb.cob build.sh aaaindex linkable sexitsc.o The executable object file for DataLoader/MX, ready to use with default user exits. A text file containing C prototypes for all the utility procedures in DataLoader/MX. Incorporate this file into all user exit source files by using an #include command. A text file containing the C source of the default user exits. To build a customized version of DataLoader/MX, make a copy of appropriate pieces of this file and modify the copy as needed. A text file containing the COBOL source of the default user exits. To build a customized version of DataLoader/MX, make a copy of appropriate pieces of this file, and modify the copy as needed. An OSS shell script designed to assist in creating a customized DataLoader/MX version. It does the compilation and linking and, optionally, does the SQL compile. Descriptions of data files included with DataLoader/MX. This index file and the sample input files are kept in the data subdirectory of the DataLoader/MX installation directory. These data files contain lists of common data, such as last names, male and female first names, street addresses, and more. Use these files with the Cartesian field method to generate test data. For more information, see Generating Test Data on page The relinkable object code for DataLoader/MX, complete except for the default user exits. Use this file when building a customized DataLoader/MX. The object file containing the default user exits. Any exits that are not redefined by the user are picked from this object. 2-1

24 DataLoader/MX Components DataLoader/MX Structure DataLoader/MX Structure Internally, DataLoader/MX consists of three major components the DataLoader/MX main logic, user exits, and the DataLoader/MX library. Figure 2-1 shows the structure of DataLoader/MX. Figure 2-1. Structure of DataLoader/MX DataLoader/MX Main Logic User Exits Library VST016.vsd Main Logic The main logic component performs the basic program control. It uses two other components, the DataLoader/MX library and the user exits, to perform common functions that might need to be extended by the user. User Exits Library The user exits module contains the user exit procedures described in User Exit Descriptions on page 5-4. User exit procedures are called from specific points in the main logic, allowing loading operations to be customized. The processing flowcharts in Appendix B, Processing Flowcharts, indicate where user exits are called. The DataLoader/MX library contains modules for common functions, such as file access, data conversion, and error handling, for use by both the main logic and user exits. The library is described in DataLoader/MX Library on page

25 DataLoader/MX Components DataLoader/MX File System DataLoader/MX File System To provide important loading features that are not directly supplied by the Enscribe file system, DataLoader/MX layers its own file system on top of the Guardian file system. The DataLoader/MX file system is designed to support DataLoader/MX, so some of the capabilities in the underlying Guardian file system are not used. For example, DataLoader/MX processes files sequentially, so it does not use random I/O. The DataLoader/MX file system includes these features for sequential file processing: Concatenated files. DataLoader/MX enables you to point to data files that in turn point to other data files. As a result, you can load from very large data files or build extremely large data file sets, consisting of many files. Using the INDIRECT option for input and output, you can go beyond the size limitation for single files. Full files. DataLoader/MX follows a specific strategy for file-full situations. When it encounters an error 43 or 45 on a file, DataLoader/MX writes a request to its stderr process for the name of another file to use. The new file name can have the standard interpretations and modifiers. Because there is no time-out, you can take as much time as necessary to specify another file. You can also choose not to specify another file, causing DataLoader/MX to terminate. If you specify a file and that file also fills up, DataLoader/MX prompts again for a subsequent file or files. File creation. If you enter the name of a file that does not exist, DataLoader/MX creates it for you. If the specified file name is an OSS name, it is created as an odd unstructured file. The TEXT interpretation is not added by default. If the specified file name is a Guardian name, the file is created as an EDIT file. If you do not want your output file to be a text file type, create a file of the type that you want at another OSS shell prompt, and then give the file name to DataLoader/MX. For information about treating a set of output files as a unit, see the description of the INDIRECT interpretation on page 4-7. Unstructured files. Unstructured files are considered by the DataLoader/MX file system to be simply a string of bytes. Any structure inside an unstructured file must be provided by the application. Unstructured files are important in the loading process because they can use bulk I/O, the ability to read or write in any multiple of 2 KB up to 56 KB chunks. The use of bulk I/O reduces the cost per byte of I/O, which is best for I/O intensive applications like loading. However, bulk I/O has these restrictions: Secondary partitions are not supported. Bulk I/O treats the primary partition as the only partition. If an unstructured file with secondary partitions is given to DataLoader/MX, it terminates with an error giving the name of the offending file and the number of secondary partitions it has. This restriction does not limit the ability of DataLoader/MX to process a large amount of unstructured data sequentially, because any number of (unpartitioned) unstructured files can be concatenated using the INDIRECT interpretation. 2-3

26 DataLoader/MX Components DataLoader/MX File System The file must be opened in exclusive mode. (Input files can be opened in nonexclusive mode.) Guardian files. DataLoader/MX can read and write Guardian files. DataLoader/MX determines that a file is a Guardian file if the file name begins with one of these: $ \ = /G /E If an OSS file name is being opened for output and the file does not exist, DataLoader/MX creates the file and writes to it as an unstructured file. It will not write to it as an OSS text file. If the OSS file should be created as a text file, add the TEXT interpretation after the file name. Adding the CSV and DUMP interpretations also implicitly makes the output file an OSS text file. If the output file name is a Guardian name, it creates an EDIT file if the file being opened for output does not exist. The DataLoader/MX process itself must still be run in the OSS environment, with a working directory that is not under the /G or /E directories. I/O blocking. DataLoader/MX can do I/O blocking only if the RECFORM modifier is specified. Otherwise, DataLoader/MX uses the maximum block size for I/O blocking, as listed in Table 2-1 on page 2-5, and each block is treated as if it contains one record. 2-4

27 DataLoader/MX Components DataLoader/MX File System Table 2-1. DataLoader/MX I/O Blocking Behavior Max. Block Size BLKSIZE Modifier Allowed? Max. Rec Length RECFORM= FB Allowed? Performance. DataLoader/MX efficiently uses the underlying OSS file system for OSS files and also efficiently uses the underlying Enscribe file system for Guardian files. Nowait I/O is used for everything except input from a terminal. For those types of files that support it, multiple outstanding I/Os are maintained. Unstructured disk files are processed with bulk I/O (56 KB blocks). Text file line numbering. Whenever DataLoader/MX opens a Guardian EDIT file for output, it automatically changes the line increment to.001, instead of the default increment of Consequently, an empty text file can contain 100,000,000 lines (.000,.001,.002,..., ). The number and size of extents, as well as the possible unavailability of such extents, can limit the number of lines, eventually producing an error 43 or 45 (handled as described under the Full Files entry). $RECEIVE. You can specify $RECEIVE like any other file for either input or output. DataLoader/MX matches writes to the DataLoader/MX process from another process with reads of $RECEIVE from within DataLoader/MX and reads of the 2-5 RECFORM= VB Allowed? EDIT file N.A. 1 No 239 No No No EDIT file with CSV N.A. 1 No 4072 if input No No No Process No Yes 3 Yes 5 No $RECEIVE No Yes 3 Yes 5 Yes 6 Structured N.A. 1 No 4072 No No No File Tape No Yes 3 Yes 5 Yes 6 Terminal N.A. No No No No Unstructured File Yes Yes 4 Yes 5 Yes 6 RECFORM= IBMVB Allowed? 1 The blocking is done by file system functions. 2 The specified size must be an integral multiple of 2048 between 2048 and DataLoader/MX performs I/O blocking based on the criteria specified. For output, DataLoader/MX chooses the block size to be an integral multiple of the record size that will hold as many records as possible but not exceed the value shown in the Max Block Size column in this table. For input, DataLoader/MX unblocks the data into fixed length records. A file will not contain a partial record. 4 A record transparently spans the block if the block size is not an integral multiple of the record size. 5 The block size chosen is the value shown in the Max Block Size column in this table unless otherwise specified in the BLKSIZE modifier for the unstructured file. Blocks contain an integral number of records. If the block size is not a multiple of the record size, the block is padded to the full length. For more information, see VARIN and VAROUT in the File Utility Program (FUP) Reference Manual. 6 For input only. Note. In the OSS environment, you must precede the $ with a backslash (for example, \$RECEIVE).

28 DataLoader/MX Components DataLoader/MX File System DataLoader/MX process from another process to writes to $RECEIVE. You can use $RECEIVE as input and output at the same time. There are many benefits to using $RECEIVE as a normal input or output file when you use other utilities. For example, import can get its input from a named DataLoader/MX process that has its output stream set as $RECEIVE. Also you can process an SQL/MP table in its binary form by using the SQLCI COPY command to copy the table to or from a named DataLoader/MX process. You can also receive data from FTP directly into DataLoader/MX without having to put it to a disk file first. You can do this by having the FTP client use get to move a file to a named DataLoader/MX process. When you run DataLoader/MX with $RECEIVE as its output, use the -S parameter to direct DataLoader/MX to generate statistics that include information about the buffer pool. Use these statistics to build a loading application and to decide how many DataLoader/MX processes to use. Opening nonexistent processes. DataLoader/MX is often used in an environment where a number of processes, such as DataLoader/MX and import processes, communicate with each other. In such an environment, processes being opened must exist before the processes that open them. To simplify this, the DataLoader/MX file system delays an open when the process being opened does not exist yet. If an open returns error 14, the file system delays for one second and retries the open. It repeats this 60 times for a total of a one minute delay before reporting an error on the open. You can specify a larger or smaller wait time by using the WAIT modifier. Opening busy files. DataLoader/MX is used in environments where many processes are started by the OSS shell, some started waited fashion and some nowaited. In this environment, the OSS shell can start a process before work needed from a second process has completed. To avoid potential problems, DataLoader/MX uses the delaying strategy described under the previous item whenever an open yields error 12. Speed inhibitors. Some types of files can have attributes that reduce access speed to files. DataLoader/MX checks for these situations and, if present, reports a warning message and continues processing (these attributes apply only to Guardian files): Unbuffered output disk files Output disk files with the REFRESHEOF attribute Output disk files on mirrored disks with the SERIALWRITES attribute Output disk files with the VERIFYWRITES attribute Output ENSCRIBE files with alternate keys Input or output ENSCRIBE files with block sizes less than 4096 Unblocked interprocess communication 2-6

29 3 Running DataLoader/MX This section explains how to run DataLoader/MX. It describes parameters defined by HP and gives examples of the use of these parameters. It also discusses parallelism and how to build your load application to run DataLoader/MX most effectively. Syntax of DataLoader/MX To start DataLoader/MX, run the DataLoader/MX program and give it a list of parameters. These parameters, together with defined user exits, determine what DataLoader/MX does. DataLoader/MX does not have an interactive mode. Two types of parameters can be passed to DataLoader/MX: parameters for the DataLoader/MX main logic and parameters for user exits. Defined by HP, parameters for the DataLoader/MX main logic start with a dash (-). They are discussed next. Parameters for user exits should not start with a dash. For more information, see User Exits on page 5-1. run [-RUN_OPTION=[option][-RUN_OPTION=[option]]...] dataload [-DATALOADER_OPTION[=option] [-DATALOADER_OPTION[=option]]...] DATALOADER_OPTION is: -C=num-recs -E=err-file -F=first-rec -G -H -I=in-file[(i-mod[,i-mod]...)] -L=num-errs -O=out-file[(o-mod[,o-mod]...)] -P=num-parts -Q -S=num-recs [ <CSV[=out-file ]> ] -T=num-recs [ <RESTARTFILE=restart-file [<{KEEP PURGE}>]>] -X=[1][2][3][4][C][D][I][U][Hnnn] i-mod is: BUFSIZE=bufsize CSV [ <[SEPCHAR=sepchar] [,CONTCHAR=contchar ] [,MAXRECLEN=len ]> ] DUMP [ <{HEX ASCII}> ] EBCDIC 3-1

30 Running DataLoader/MX Syntax of DataLoader/MX o-mod is: INDIRECT TEXT MAX=size NOREWIND NOUNLOAD num% NUMBUFS=numbufs RECFORM= { FB<{RECSIZE=recsize [,PADCHAR=padchar ]> } { VB[<BLKSIZE=blksize> ] } { IBMVB } PERSIST SUBTYPE=subtype-num TYPE=type-num WAIT=secs BROADCAST BUFSIZE=bufsize CSV [ <[SEPCHAR=sepchar ] [,CONTCHAR=contchar ] [,MAXRECLEN=len ]> ] DUMP [ <{HEX ASCII}> ] INDIRECT KEYRANGE TEXT MAX=size NOREWIND NOUNLOAD num% NUMBUFS=numbufs RECFORM= { FB<{RECSIZE=recsize [,PADCHAR=padchar]> } { VB[<BLKSIZE=blksize>] } WAIT=secs RUN_OPTION is an OSS run option, such as name, as described in the Open System Services Shell and Utilities Reference Manual. DATALOADER_OPTION is a parameter that controls the operation of DataLoader/MX. When using or specifying DATALOADER_OPTION: Parameters can be in any order, are not case-sensitive, must be separated from each other by at least one space, and must not contain any spaces. To avoid conflict with those parameters that might be defined in the future by HP, do not define parameters for user-defined exit functions to begin with a dash. Each parameter (defined by HP) begins with a dash (-) and is immediately followed by the name of the parameter. If the parameter has a value, the value 3-2

31 Running DataLoader/MX Syntax of DataLoader/MX must follow the parameter name immediately, separated by an equal sign (=). If additional information is associated with the value, it must follow the value and be enclosed in angle brackets (<>). The parameter and its value cannot have embedded spaces. The parameter and its value are not case-sensitive, except for any OSS file names specified. If interpretations or modifiers are present, then the value has to be enclosed within double quotes. $ or \ characters have to be escaped by adding \. -C=num-recs specifies the number of records to process. DataLoader/MX terminates normally when num-recs input records have been processed or when it reaches end of file. If this parameter is not specified, all records in the input stream are processed. -E=err-file specifies the name of the file or DEFINE to which errors and the input records for which they occur are written. If the file does not exist, it is created as an OSS text file when the first error occurs (the TEXT interpretation is implicitly added). If errfile is a Guardian file name, it is created as an EDIT file when the first error occurs. To determine if errors occurred, check for the existence of this file. If the file already exists before running DataLoader/MX, error information is appended to the file. A line indicating the date and time of the append is written first. If this parameter is not specified, the error information is written to the DataLoader/MX process stderr. -F=first-rec specifies the first record to process. To maintain compatibility with the File Utility Program (FUP), the first record is record number 0. This parameter can also be thought of as the number of records to skip. If this parameter is not specified, processing begins with the first record (which is the same as -F=0). -G causes DataLoader/MX to call GETNEXTRECORD (a user exit) to fetch the data. The record returned from GETNEXTRECORD is passed to DataLoader/MX to be processed. It might be passed to a user exit, included in partition determination (if -P was specified), or written to the output file (if -O was specified). Therefore, when you specify the -G parameter, DataLoader/MX does not fetch data directly from the input file. The -I and -G parameters interact as: 3-3

32 Running DataLoader/MX Syntax of DataLoader/MX If -G is specified and -I= is not specified, DataLoader/MX does not open an input file. When it needs a record to process, it calls the GETNEXTRECORD exit. If -I= is specified and -G is not specified, DataLoader/MX opens the input file and fetches the data to be processed. If -I= and -G are both specified, DataLoader/MX opens the input file but does not read data from it directly. When DataLoader/MX needs a record to process, it calls the GETNEXTRECORD exit. If you require it, GETNEXTRECORD can read the input file specified in -I= by calling DTLRead and passing -1 as the file number. If neither -I= nor -G are specified, DataLoader/MX behaves as if only -G had been specified. -H displays help. DataLoader/MX displays a brief description of its parameters. This display is also generated if DataLoader/MX is run with a parameter that is not recognized by DataLoader/MX or its user exits. -I=in-file [ (i-mod [,i-mod]...) ] specifies an input file to read, with optional modifiers or interpretations, or both. where i-mod is: BUFSIZE=bufsize CSV [<[SEPCHAR=sepchar] [,CONTCHAR=contchar] [,MAXRECLEN=len]>] DUMP [<{HEX ASCII}>] INDIRECT TEXT MAX=size NOREWIND NOUNLOAD num% NUMBUFS=numbufs RECFORM= { FB<RECSIZE=recsize[,PADCHAR=padchar]> } { VB[<BLKSIZE=blksize>] } { IBMVB } PERSIST SUBTYPE=subtype-num TYPE=type-num WAIT=secs in-file is the name of the file or DEFINE to be used for input. For more information, see Specifying File Names on page

33 Running DataLoader/MX Syntax of DataLoader/MX bufsize is the size of the buffers. sepchar is an alternate separation character. The default value is a comma (,). contchar is an alternate continuation character. The default value is an ampersand (&). len is the maximum record length. Its default value is 239. num% is a percent number of records. Decimal point is allowed. numbufs is the number of buffers to use. recsize is the record size. padchar is the character to be used for padding. blksize is the block size to use. subtype-num is a subtype number. type-num is a type number. secs is the number of seconds to wait. For more information about the -I and -G parameters interact, see the description of the -G parameter on page 3-3. For an in-depth discussion of modifiers and interpretations, such as commaseparated values (CSV) and MAX, used with the -I and -O parameters. For more information, see Section 4, Specifying File-Related Options for DataLoader/MX. 3-5

34 Running DataLoader/MX Syntax of DataLoader/MX -L=num-errs directs DataLoader/MX to terminate if the number of error messages reaches numerrs errors. If this parameter is not specified, DataLoader/MX continues regardless of the number of errors detected. -O=out-file[(o-mod[,o-mod]...)] specifies an output file to write, with optional modifiers or interpretations, or both. where o-mod is BROADCAST BUFSIZE=bufsize CSV [<[SEPCHAR=sepchar] [,CONTCHAR=contchar] [,MAXRECLEN=len]>] DUMP[<{HEX ASCII}>] INDIRECT TEXT KEYRANGE MAX=size NOREWIND NOUNLOAD num% NUMBUFS=numbufs RECFORM= { FB<RECSIZE=recsize [,PADCHAR=padchar]> } { VB[<BLKSIZE=blksize>] } WAIT=secs out-file is the output file. For more information about the format for specifying the file name, see Specifying File Names on page If the file does not exist, DataLoader/MX creates the file. Without modifiers, if the file name is an OSS name, DataLoader/MX creates a plain OSS file. If the file name is a Guardian name, DataLoader/MX creates an EDIT file. If -O= is not specified, DataLoader/MX does not automatically write records to an output file, but you can still write records from a user exit to any file. bufsize is the size of the buffers. sepchar is an alternate separation character. The default value is a comma (,). contchar is an alternate continuation character. The default value is an ampersand (&). 3-6

35 Running DataLoader/MX Syntax of DataLoader/MX len is the maximum record length. Its default value is 239. num% is a percent number of records to write. Decimal point is allowed. numbufs is the number of buffers to use. recsize is the record size. padchar is the character to be used for padding. blksize is the block size to use. secs is the number of seconds to wait. For more information about an in-depth discussion of modifiers and interpretations, such as CSV and MAX, used with the -I and -O parameters, see Section 4, Specifying File-Related Options for DataLoader/MX. -P=num-parts directs DataLoader/MX to estimate the partition boundary keys for num-parts number of partitions. For more information, see Using the -P Parameter on page Q runs DataLoader/MX in quiet mode. Normally, DataLoader/MX writes error and informational messages to its stderr file. DataLoader/MX also writes to its stdout file, which can be changed by OSS command-line output redirection, if the stdout is different from stderr. Specifying the -Q parameter directs DataLoader/MX not to write informational messages to stderr. Error messages are still handled normally, and informational messages are still written to stdout if it differs from stderr. Use this parameter to reduce the number of messages on the screen when running many DataLoader/MX processes simultaneously. When an error occurs, try using an earlier informational message to understand the error. If you use -Q, it is generally a good idea to redirect the stdout file to a disk file to capture all messages. If you cannot diagnose a problem by using the error message, and you 3-7

36 Running DataLoader/MX Syntax of DataLoader/MX did not redirect stdout, you used the -Q option, you can obtain more information by rerunning the job without -Q. -S=num-recs [<CSV[=out-file]>] directs DataLoader/MX to print periodic status messages. num-recs is the number of input records that are processed between status messages. <CSV[=out-file]> if specified, causes messages to appear in a comma-separated value (CSV) format suitable for loading into a spreadsheet for analysis or graphical display. If =out-file is specified, the CSV status information is written to this file. If =outfile is omitted, the status information sent to the terminal appears in CSV format. When you specify -S, if you open a DataLoader/MX process with the #CMD.STAT qualifier, it prints the current statistics. -T=num-recs [ <RESTARTFILE=restart-file [<{KEEP PURGE}>]>] directs DataLoader/MX to bracket records within HP NonStop Transaction Management Facility (TMF) transactions. If a DataLoader/MX process fails within a transaction, TMF backs out changes associated with records in the currently active transaction and returns the database to the state it was in after processing the last record in the last committed transaction. After resolving the problem that caused the failure, you can restart the operation at that point. Information to facilitate such a restart is put into restart-file. num is the number of input records per TMF transaction. If num is large, the resources consumed by database changes associated with the records in the transaction might hit a TMF limit, and the transaction could be aborted by TMF. If num is very small, the TMF and DataLoader/MX costs per transaction are excessive. Some experimentation is usually necessary to determine an appropriate value for num. restart-file is the name of the restart file, which must be a Guardian file name. DataLoader/MX generates a restart file with the name specified if the file does not already exist. This file is an audited relative file with a record size of 20. When DataLoader/MX successfully commits a transaction, it writes the last number of records read from input to this file. If an error occurs, this file contains the record number that was committed successfully. When DataLoader/MX is restarted with this restart file, the previously processed 3-8

37 Running DataLoader/MX Considerations DataLoader/MX Process records (as recorded in the restart file) are skipped. The restart file is an Enscribe file and therefore has a Guardian name rather than an OSS name. PURGE directs DataLoader/MX to purge the file containing restart information on normal termination. After abnormal termination, the file still exists. KEEP directs DataLoader/MX to never purge the file. You must purge it after determining that the load application was successful. KEEP is the default. If the output requires a transaction ID and -T= was not specified, DataLoader/MX terminates with an error. -X=exitlist specifies optional user exits to call and in what order to call them. For more information, see User Exits on page 5-1. Considerations DataLoader/MX Process In addition to the preceding DataLoader/MX parameters, user exits can have optional or required parameters. By convention, all DataLoader/MX parameters defined by HP begin with a dash, and parameters for user-defined exits do not begin with a dash. For a brief list of all parameters supplied by HP, run DataLoader/MX with the -H parameter. Enclose an argument having parentheses or angle brackets within double quotes. Use parentheses only to enclose modifiers after file names. Any other qualifying information is enclosed in angular brackets, even within file modifiers: $ dataload -t="100<restartfile=\$vol.subvol.filename>" $ dataload -i="file1(recform=fb<recsize=100>)" The restart file name is enclosed in angle brackets because restartfile=filename is not a file modifier. The entire recform=fb<recsize=100> clause is enclosed in parentheses because it is a file modifier. <recsize=100> is enclosed in angle brackets because it is qualifying information specifying the length of fixed length records. When specifying a number, if it is used in the % file modifier, it can have a decimal point. A number used in parameters cannot have a decimal point. Note. In this manual, the examples for running DataLoader/MX from the OSS command prompt use \ as the line continuation character. All instances of $ and \ characters are escaped by adding a \ to avoid shell interpretation. Arguments having other special characters (such as parentheses and angular brackets) are enclosed within double quotes to avoid shell interpretation. An & is added at the end of the command to denote that the command should run in the background. For more information about OSS, see the Open System Services User s Guide. 3-9