Interstage Shunsaku Data Manager Operator s Guide

Transcription

1 Interstage Shunsaku Data Manager Operator s Guide

2 Operator s Guide Trademarks Trademarks of other companies are used in this manual only to identify particular products or systems. Product Microsoft, Visual C++, Windows, and Windows Server Solaris Linux Java Red Hat, RPM, and all trademarks and logos based on Red Hat Interstage Trademark/Registered Trademark Registered trademark or trademarks of Microsoft Corporation in the United States and other countries. Solaris and all other trademarks and logos related to Solaris are registered trademarks or trademarks of Sun Microsystems Inc. U.S.A. in the United States and other countries, and are used under license from the said company. Registered trademark or trademark of Linus Torvalds in the United States and other countries. Java and other trademarks comprising Java are trademarks of Sun Microsystems Inc. U.S.A. in the United States and other countries. Trademarks or registered trademarks of Red Hat Inc. in the United States and other countries. Interstage is a registered trademark of Fujitsu Limited. Fujitsu documentation may contain specific technologies that are subject to the Foreign Exchange and Foreign Trade Control Law. When such specific technology is described in a document, and that document is either exported or provided to a nonresident, permission must be sought under the above law. Fujitsu Limited First Edition (May 2005) The contents of this manual may be revised without prior notice. All Rights Reserved, Copyright FUJITSU Limited 2005 ii

3 Preface Purpose of this Manual This document describes the basic functions provided by the Interstage Shunsaku Data Manager (hereafter abbreviated as Shunsaku) and explains how to set up the environment and tune operations. Target Audience This manual is intended for the following audience: Persons who use Shunsaku in a basic configuration (single-server configurations or configurations in which only searchers are allocated to separate servers). Required Knowledge This manual assumes that the reader has an understanding of the following topics: XML C language Java language Internet (basic knowledge) Windows Linux Solaris OE. iii

4 Preface Organization of this Manual This manual is organized as follows: Chapter 1 - Overview This chapter provides an overview of the Shunsaku features and functions. Chapter 2 - Design This chapter describes the Shunsaku system configuration, file structure, and design procedures. Chapter 3 - Operation This chapter describes all Shunsaku operations. Chapter 4 - Maintenance This chapter describes the actions required if errors occur during Shunsaku operation. Chapter 5 - XML Conversion Function This chapter describes how to create the text files that store data in XML format from databases and CSV files. Appendix A - Allowable Values This appendix lists the allowable Shunsaku values. Appendix B - Estimating Resources This appendix describes how to estimate the various resources used by Shunsaku. Appendix C - Notes on XML Documents This appendix gives notes concerning the data stored in XML format by Shunsaku. Appendix D - XML Conversion Function API This appendix describes the XML conversion function APIs provided by Shunsaku. Appendix E - XML Conversion Function Mapping Rules This appendix describes how to define the mapping rules used by the XML conversion function and gives examples of applying rules and conversion examples. iv

5 Table of Contents Chapter 1 Overview Overview of Shunsaku What is Shunsaku? Shunsaku Technology Shunsaku Features Easy Installation without Design or Indexes Strong Search Functions and High-speed Searches Highly Flexible System Structures High-availability Operation with Load Redistribution Shunsaku Functions Data Search Function and Update Function Data Searches Data Updates Data Sorting and Aggregates Data Import Function Data Import XML Conversion Backup and Recovery Strong GUI Tools Chapter 2 Design System Structures Process Structure Server Structure File Structure Admin Server Structure Same Server used as Admin Server and Director Server (System) Different Servers used as Admin Server and Director Server (System) Shunsaku Design Appointing an Administrator Designing the Server Structure Planning the Number of Searchers Planning the Number of Directors Planning the Number of Search Servers v

6 Table of Contents Designing the Disk Allocation Determine the Character Encoding Estimating Resources File Capacity Memory Capacity Designing the Applications Installing Chapter 3 Operation Operation Flow Starting the Management Console and Logging in Starting and Stopping Starting Stopping Creating Lookup Data Searching and Updating Data Tuning System Operation Reallocating Search Data Optimizing Director Data Adding Searchers Adding a Searcher Adding a Search Server and a Searcher Adding Directors Changing the Data Insertion Destination Director Deleting Searchers Deleting Searchers Deleting an Added Search Server and Searcher Deleting a Director Environment Setup Modifying the Search Environment Setting the Backup Destination for the Environment File Chapter 4 Maintenance Backup Director Data Backup Shunsaku Environment File Backup Recovery Director Data Recovery Shunsaku Environment File Recovery vi

7 Table of Contents Recreating Data for Searches Troubleshooting Server Errors Errors on the Application Server Chapter 5 XML Conversion Function Overview of the XML Conversion Function Converting Data from Databases to XML Format Converting Data from CSV Files to XML Format Designing the XML Conversion Function Determine the Source of Input Data Determine the Structure of the Data in XML Format Setting up the XML Conversion Function Setting up the XML Conversion Function APIs API Configuration Setting Environment Variables Creating a Mapping Rule File Designing Applications List of APIs Processing Procedure of Applications that use the XML Conversion Function Procedure for Converting Data in a Database to XML Format Procedure for Converting Data in a CSV File to XML Format Examples of API Usage Startup through to ResultSet Acquisition (Database Records) Startup through to ResultSet Acquisition (CSV File) Creation of XMLGenerator to Acquisition of Data in XML Format String Stream File DOM Tree SAX Sample Program for the XML Conversion Function Data Types Supported by the XML Conversion Function Results Table Data Types Compatible with the XML Conversion Function Correspondence between Results Table and Database Data Types CSV Formats Supported by the XML Conversion Function Character Data Linefeed Characters vii

8 Table of Contents Appendix A Allowable Values Appendix B Estimating Resources Estimating Memory Requirements... B-2 Estimate for Operation on a Single Server... B-3 Estimation when the Searcher runs on a Separate Server... B-3 Estimating Disk Requirements...B-5 Estimating Disk Space for Director Data... B-5 Estimating Disk Space for Operation Log Files and Core Files... B-6 Estimating Resources used by the XML Conversion Function... B-8 Appendix C Notes on XML Documents Format of XML Documents...C-2 XML Documents in Text Files...C-3 Notes on XML Format...C-5 Appendix D XML Conversion Function APIs List of APIs...D-2 DocumentFountain Interface...D-5 Fields of the DocumentFountain Interface...D-5 Methods of the DocumentFountain Interface...D-5 getxmldocument...d-5 hasxmldocument...d-6 setdestination...d-7 setencoding...d-8 CSVSource Class...D-9 CSVSource...D-9 setencoding...d-10 setdelimiter... D-11 setquote...d-12 setnullcharacter...d-13 setfirstrecordtocolumn...d-14 setrandommode...d-15 getresultset...d-16 setenableescapechar...d-17 GeneratorMap Class...D-18 GeneratorMap...D-18 XMLGenerator Class...D-19 XMLGenerator...D-19 createdocumentfountain...d-20 viii

9 Table of Contents XMLConverterException Class...D-21 getcode...d-21 getexception...d-22 getmessage...d-23 CSVException Class...D-24 CSVException...D-24 XMLGenException Class...D-26 XMLGenException...D-26 Error Codes Posted when the XML Conversion Function API is used...d-29 Appendix E XML Conversion Function Mapping Rules Overview... E-2 Mapping Rule File Definition Format... E-3 MappingRule... E-4 parentrule... E-5 columnrule... E-7 attributerule... E-10 documentrule... E-13 extensionrule... E-13 Mapping Rule Examples... E-16 Mapping Rule Conversion Examples... E-19 Extracting Unstructured Data in XML Format... E-19 Extracting Structured Data in XML Format... E-20 Extracting Fixed Values... E-21 Extracting Column Data as Attribute Values... E-22 Extracting Column Data as Fixed Attributes... E-23 Extracting Column Names as Attribute Values... E-24 Extracting Data in XML Format that has Multi-level Tag Structures Based on Consolidation... E-26 Extracting Binary Data... E-28 ix

10 Table of Contents x

11 Chapter 1 Overview This chapter provides an overview of Shunsaku. Overview of Shunsaku Shunsaku Features Shunsaku Functions 1-1

12 Chapter 1: Overview Overview of Shunsaku Corporations, autonomous bodies, and public organizations currently use information systems consisting of multiple business servers which accumulate more and more information each day. It is not easy to integrate the information of an entire corporation, and it is a recognized problem that corporations cannot make effective use of all the information they continue to accumulate. For example, the various business sections of a single corporate group normalize the data they use and store it in databases. However, attempts to enable the group as a whole to use of all this data are hindered by the adoption of different database products and schema structures. To make use of all of the data, it must first be normalized and systems restructured. Otherwise, only some data can be used. The result is that the group as a whole cannot effectively use of all the various business data and customer information. In addition, the handling of data in a variety of document formats, not just in simple numeric value and character string formats is becoming increasingly important in the Internet business environment. In particular, data in XML format is now used by many businesses. However, these data formats are not easy to normalize, and are problematic to make use of effectively in business. Figure 1-1 Problems of Current Business Environments 1-2

13 Overview of Shunsaku Various commercial RDBMS (Relational Database Management System) vendors now provide extended functions that handle data in XML format. However, it is difficult to design database structures for dealing with variable-length data or variable occurrence items and indexes. Accordingly, a mechanism is required for making use of information that cannot be managed by a RDBMS or that cannot be normalized at the company-wide level. Shunsaku is the revolutionary technology that solves these problems. What is Shunsaku? Shunsaku searches data that is in XML format, and manages the variable-length and variable occurrence items that are problematic for the system structures of conventional RDBMS. Shunsaku uses revolutionary technology that searches data, not indexes, directly, at high speeds. Shunsaku can search data regardless of whether it is XML, text file data, or data stored in a RDBMS. The Shunsaku high-level search functions search text data, ranging from small-scale text data to the text data held in large-scale database systems, as well as data that is difficult to search using conventional RDBMS. Shunsaku also provides superior support for future business expansion, since it can flexibly manage the addition of new businesses and new business applications. Installation of Shunsaku enables users to manage and utilize the data held in multiple business systems from one base, thus making business faster and system management more efficient. Shunsaku makes possible business systems that use data more flexibly and quickly than ever before. Shunsaku also shortens development time, reduces investment costs, and provides performance guarantees, simple operations, and scope for business expansion. Figure 1-2 What is Shunsaku? 1-3

14 Chapter 1: Overview Shunsaku Technology Shunsaku uses the following technology to enable high-speed searches of data in XML format: High-speed search algorithms High-speed search algorithm technology executes searches at a fixed speed even if there are multiple search conditions. Searches take about the same amount of time logically regardless of whether there is one condition or three conditions. High-volume traffic technology High-volume traffic technology bundles together multiple search requests from applications and processes them at the same time. This technology compares multiple search conditions while it reads through the data, and prepares the answers to all the requests by the time reading is finished. For example, the search time is logically about the same for one search request containing multiple search items as it is for multiple search requests each containing one search item. Thus, even a dramatic increase in access has little effect on the search time. Parallel search technology Parallel search technology runs multiple search engines to use hardware resources efficiently and run searches in parallel. The search time can be easily reduced simply by increasing the hardware resources. 1-4

15 Shunsaku Features Shunsaku Features Shunsaku boasts the following features: Easy Installation without Design or Indexes Strong Search Functions and High-speed Searches Highly Flexible System Structures High-availability Operation with Load Redistribution These features are described below. Easy Installation without Design or Indexes Shunsaku can be installed easily and quickly without the need for designs or indexes. Use of the Wizard to execute installation enables various types of required setups to be performed automatically, thus simplifying Shunsaku installation. In addition, after installation, Shunsaku provides a user-friendly Management Console which is viewed from a Web browser. This provides superior usability, making Shunsaku very easy to operate. Installation using the Wizard Figure 1-3 Installation using the Wizard 1-5

16 Chapter 1: Overview Superior usability with the Management Console Figure 1-4 Superior Usability with the Management Console 1-6

17 Shunsaku Features Strong Search Functions and High-speed Searches Shunsaku makes it easy to handle data that has a complicated structure. This includes data where: The number of items is not fixed Some items do not have fixed lengths Items have a hierarchical structure Most types of items can be processed by one search request. Shunsaku can also process a variety of conditions, including complicated conditions involving character strings, numeric data, and word searches targeting English words, in one search request. Figure 1-5 Shunsaku's Strong Search Functions In addition, Shunsaku enables high-speed searches regardless of the volume of data, and enables cross-section searches of data in multiple systems. High-volume traffic can be processed at high speeds because multiple search requests are processed as a group, the search results then split, and returned to the applications. 1-7

18 Chapter 1: Overview Figure 1-6 High-volume Traffic High-speed Searches Highly Flexible System Structures Since Shunsaku has no indexes, new data definitions or tuning are not required if new data items are added or search methods are modified as a result of business expansion or changing needs. Changes in business can be managed simply by adding applications. In addition, if the quantity of data becomes greater than was expected when a system was constructed, the simple addition of search engines can instantly handle the increase. High-availability Operation with Load Redistribution Shunsaku can run multiple Shunsaku search engines. Shunsaku allocates equal amounts of searchable data to each of the search engines, and searches the data in parallel. If an error occurs at one of the search engines, the data being searched by that search engine is automatically distributed between the remaining search engines. This preserves the data and enables operations to continue. This function is called the degradation function. 1-8

19 Shunsaku Functions Shunsaku Functions Shunsaku provides the following functions: Data Search Function and Update Function Data Import Function Backup and Recovery Strong GUI Tools Figure 1-7 shows the configuration of the Shunsaku functions. Figure 1-7 Shunsaku Function Configuration Data Search Function and Update Function Shunsaku uses the Shunsaku APIs to enable data searches and updates to be performed from applications. The search results can also be sorted and aggregated. Shunsaku provides both Java language and C language APIs. Refer to the Application Development Guide for details of how to perform data searches and updates from applications. 1-9

20 Chapter 1: Overview Data Searches Shunsaku receives search requests from applications, finds the data that matches the search conditions, and returns the search results to the applications. Figure 1-8 Data Search Function Data Updates During Shunsaku data updates, XML documents are added, deleted, and updated in response to requests from applications: Adding an XML document Shunsaku receives the request to add an XML document, then adds the XML document to Shunsaku. Deleting an XML document Shunsaku receives information indicating the XML document to be deleted, then deletes the specified XML document from Shunsaku. Updating an XML document Shunsaku receives information indicating the XML document to be updated and the XML document which will hold the post-update contents, then updates the contents of the specified XML document held in Shunsaku. 1-10

21 Shunsaku Functions Data Sorting and Aggregates When data is sorted or aggregated, Shunsaku receives the search request from the application, sorts or aggregates the data that matches the search conditions, then returns those results to the application. Figure 1-9 Data Aggregate Function 1-11

22 Chapter 1: Overview Data Import Function The data import function imports data in XML format to Shunsaku. Figure 1-10 Data Import Function Data Import Shunsaku Data Import imports both XML data and data extracted from databases to Shunsaku, and distributes the data to the search engine. The distributed data is expanded in the search engine memory. If there are multiple search engines, the data is distributed equally to all the search engines. XML Conversion The XML conversion function converts the data stored in CSV files to XML format data. Mapping rules that define the correspondence between the input data and the XML format output data are used during data conversion. Refer to Appendix E, 'XML Conversion Function Mapping Rules', for details of the mapping rules. The XML conversion function provides a Java API. This API is called from the application. 1-12

23 Shunsaku Functions Backup and Recovery Shunsaku can backup accumulated data during operation without stopping search or update processes from the Shunsaku API. If an error occurs for the disk that is in use, Shunsaku can use the backed up data to execute recovery. This is called the recovery function. Either the Management Console or commands are used to perform backup and recovery. Figure 1-11 Backup and Recovery In addition, if Shunsaku is set to collect update logs, backups can be made without stopping updates from the Shunsaku APIs. When operating in this way, if a disk error occurs during operation, the data that is already backed up can be used together with the update logs to recover Shunsaku to its status at any point prior to the error occurrence. Refer to 'Backup and Recovery' in the Operator's Guide (Advanced) for details. 1-13

24 Chapter 1: Overview Strong GUI Tools Shunsaku provides user-friendly GUI tools. The Shunsaku Install Wizard is simple to operate and enables one-touch installation. In addition, after installation, the required setup can be performed automatically. A Management Console is provided to operate Shunsaku. The Management Console function is viewed from the Web browser and makes it very easy for users to start and stop Shunsaku, perform backups, import data, and monitor system operations. Tuning operations for various related products are integrated from a single screen. Thus, the Management Console enables operations from a single source, making separate operations on the various related products unnecessary. Figure 1-12 Management Console Screen Example 1-14

25 Chapter 2 Design This chapter describes Shunsaku design. System Structures Shunsaku Design Designing the Applications Installing 2-1

26 Chapter 2: Design System Structures This section describes the following Shunsaku structures: Process Structure Server Structure File Structure Admin Server Structure. Shunsaku can be used with the following combinations of operating systems: Table 2-1 Shunsaku Operating System Compatibility Product Admin Server Director Server (*1) Search Server (*2) Application Server Windows Edition Shunsaku Linux Edition Shunsaku Solaris OE Edition Shunsaku *1 The server allocated to the conductor, director, and sorter is referred to as the director server *2 The server allocated to the searcher is referred to as the search server Refer to Process Structure for information on the conductor, director, sorter, and searcher. 2-2

27 System Structures Note Note the following points when designing the Shunsaku structures Figure 2-1 Internet Connection Format Where the Internet is used for connections, locate Shunsaku within the intranet in order to protect data from illegal external access. In addition, add a third network or DMZ in the firewall and position the Web server used for external access in that network. This places the Web server under the control of the firewall and prevents illegal access. The communication method used for access across the firewall is via the application server. To further protect data on the network from data leakages and corruption, use SSL or other encrypted communication for communications that are external to the application server. 2-3

28 Chapter 2: Design Process Structure Figure 2-2 shows the Shunsaku process structure. Figure 2-2 Shunsaku Process Structure Diagram Shunsaku allocates the four processes shown in the table below. The overall system constructed from these processes is known as the Shunsaku system Process Conductor Director Sorter Searcher Role Director management, receives requests from applications, issues requests to the director and sorter Searcher management and data management (fetches data that matches the search conditions) Sorts (changes the order) and aggregates the search results Evaluates search conditions The Shunsaku system is managed using the "Shunsaku system name", attached to the Shunsaku system. The conductor, director, sorter, and searcher are managed using the identifiers attached to distinguish between their operating environments. These identifiers are the "conductor identifier", "director identifier", "sorter identifier", and "searcher identifier" respectively. 2-4

29 System Structures The "Shunsaku system name" and the identifiers are alphanumeric characters of up to 8 characters. Server Structure The basic server structures for Shunsaku systems are shown below. For information on how to expand the server configuration, refer to the Operator's Guide (Advanced). System Operating with a Single Server This structure is for a Shunsaku system where all processes are allocated to one server. Figure 2-3 System Operating with a Single Server System Operating with Searchers on a Separate Server In this structure, the only Shunsaku processes that run on a separate server are the searchers. Figure 2-4 System Operating with Searchers on a Separate Server 2-5

30 Chapter 2: Design File Structure Figure 2-5 shows the Shunsaku file structure. Figure 2-5 Shunsaku File Structure Diagram 2-6

31 System Structures Table 2-2 describes the various files. Table 2-2 File Structure File type Number of files Use Allocation position System environment file Conductor environment file 1 This file holds information that specifies the Shunsaku system operating environment 1 This file holds information that specifies the conductor operating environment Shunsaku installation folder\shunsaku\etc\system\shunsakusystem Name.cfg /etc/opt/fjsvshnsk/etc/system/shunsakusystem Name.cfg (*1) Shunsaku installation folder\shunsaku\etc\conductor\conductoridentifier. cfg /etc/opt/fjsvshnsk/etc/conductor/conductor Identifier.cfg Director environment file Sorter environment file Searcher environment file 1 This file holds information that specifies the director operating environment 1 This file holds information that specifies the sorter operating environment 1 This file holds information that specifies the searcher operating environment Shunsaku installation folder\shunsaku\etc\director\directoridentifier.cfg /etc/opt/fjsvshnsk/etc/director/directoridentifier. cfg Shunsaku installation folder\shunsaku\etc\sorter\sorteridentifier.cfg /etc/opt/fjsvshnsk/etc/sorter/sorteridentifier.cfg Shunsaku installation folder\shunsaku\etc\searcher\searcheridentifier.cfg /etc/opt/fjsvshnsk/etc/searcher/searcheridentifier. cfg (*2) 2-7

32 Chapter 2: Design File type Number of files Use Allocation position Director data file Multiple Holds the data that is imported to the director The location specified during installation or specified from the Management Console Conductor log file Director log file Sorter log file Searcher log file 2 Holds the conductor operation history and error messages 2 Holds the director operation history and error messages 2 Holds the sorter operation history and error messages 2 Holds the searcher operation history and error messages The disk that installed Shunsaku The disk where /var/opt/fjsvshnsk is located The disk that installed Shunsaku The disk where /var/opt/fjsvshnsk is located The disk where Shunsaku is installed The disk where /var/opt/fjsvshnsk is located The disk where Shunsaku is installed The disk where /var/opt/fjsvshnsk is located (*2) Text file Multiple Input file used for importing data to the director Any Conductor core file Multiple Resource used to investigate the cause of conductor errors Output only if an error occurs. The disk where Shunsaku is installed The disk where /var/opt/fjsvshnsk is located Director core file Multiple Resource used to investigate the cause of director errors Output only if an error occurs. The disk that installed Shunsaku The disk where /var/opt/fjsvshnsk is located 2-8

33 System Structures File type Number of files Use Allocation position Sorter core file Multiple Resource used to investigate the cause of sorter errors Output only if an error occurs. The disk that installed Shunsaku The disk where /var/opt/fjsvshnsk is located Searcher core file Multiple Resource used to investigate the cause of searcher errors Output only if an error occurs. The disk that installed Shunsaku The disk where /var/opt/fjsvshnsk is located (*2) API environment file 1 This file holds information that specifies the operating environment of the server allocated for the application (*3) *1 If the searcher runs on a separate server, a system environment file is also allocated for the search server. *2 If the searcher runs on a separate server, this file is allocated on the search server. A searcher environment file is required for each searcher operating. *3 Refer to the Application Development Guide for details. 2-9

34 Chapter 2: Design Admin Server Structure If the Management Console is used to control Shunsaku operations, the server where the Management Console operates is referred to as the Admin Server. Two possible Admin Server configurations are shown below. Same Server used as Admin Server and Director Server (System) In this configuration, the same server is allocated for the Shunsaku system (the director server) and the Admin Server. Figure 2-6 Same Server used as Admin Server and Director Server (System) 2-10

35 System Structures Different Servers used as Admin Server and Director Server (System) In this configuration, the server allocated for the Shunsaku system (the director server) and the Admin Server are different servers. If Shunsaku is operated in conjunction with the Interstage Application Server, the Shunsaku Management Console is allocated on the Interstage Application Server Admin Server. Therefore, the Interstage Application Server and Shunsaku can both be operated from the same Management Console GUI. In this configuration, the server where the Shunsaku system is allocated is a Managed Server. Figure 2-7 Different Servers used as Admin Server and Director Server (System) To use the Management Console in the above configuration, the server allocated for the Shunsaku system needs to be registered at the Interstage Application Server Admin Server as a Managed Server. 2-11

36 Chapter 2: Design Shunsaku Design Design the Shunsaku components, the director server, and the search server. Follow the design procedures shown below. Figure 2-8 Shunsaku Design Appointing an Administrator For the administrator, select who will belong to the "Administrators" group. For the administrator, select the "Super users". 2-12

37 Shunsaku Design Designing the Server Structure This section describes how to design the server structure. Planning the Number of Searchers Determine the number of searchers on the basis of the volume of data allocated to the searchers. The data allocated to a searcher is loaded in the searcher memory. Set the number of searchers such that, based on the total amount of data stored in Shunsaku, the amount of data allocated to each server is not excessive. Considering factors such as memory usage fluctuations and search performance, about 1GB data allocation per searcher is recommended In addition, during degradation, the data held by the searcher for which the error occurs is split between the remaining searchers and loaded in memory. Thus, in addition to memory usage fluctuations and search performance, take into account the amount of data to be distributed during degradation and determine the number of searchers accordingly. Note A searcher can be added during Shunsaku operation. Hence, if the amount of data increases beyond the amount estimated during planning, a searcher can be added and operations can continue. Refer to AppendixB, Estimating Resources for more information on the amount of memory required by searchers. Note Use of many searchers within one server may waste Windows system resources such that the Shunsaku system does not operate correctly. Check the Shunsaku environment settings. If these are correct but the Shunsaku system still does not operate correctly, the Windows system resources may be being wasted. If so, reduce the number of searchers. Planning the Number of Directors Determine the number of directors based on the number of searchers and the Shunsaku method of operation. If the number of searchers exceeds the limit for one director, multiple directors are required. Up to 20 searchers can be registered for each director. Therefore, if there are more than 20 searchers, multiple directors are required. Note A director can be added during operation. If the number of searchers increases beyond the number estimated during planning, a director can be added and operation can continue. Planning the Number of Search Servers If the amount of data stored in Shunsaku exceeds the search server memory capacity, allocate multiple search servers based on the amount of data. 2-13

38 Chapter 2: Design Designing the Disk Allocation Design the disk allocation such that the director data storage directory and the backup directory are on separate disks. Refer to 'Backup'; in Chapter 4, Maintenance for details concerning backup. Determine the Character Encoding Determine the character encoding to be used. Use the same character encoding for the following: Character identifier encoding set during standard installation Search, return, and sort expressions specified in the Shunsaku C API The XML format data used as input when search data is created. Note The character encoding specified in the XML declaration is not enabled. Estimating Resources This section describes how to estimate resources. File Capacity Refer to Appendix B, Estimating Resources for information on file capacity. Memory Capacity Refer to Appendix B, Estimating Resources for information on memory capacity. 2-14

39 Designing the Applications Designing the Applications Refer to the Application Development Guide for more information on how to design applications that search and update data using Shunsaku APIs. 2-15

40 Chapter 2: Design Installing When the design work is complete, install Shunsaku. Refer to the Installation Guide for information concerning Shunsaku installation. 2-16

41 Chapter 3 Operation This chapter describes Shunsaku operation. Operation Flow Starting and Stopping Creating Lookup Data Searching and Updating Data Tuning System Operation Environment Setup 3-1

42 Chapter 3: Operation Operation Flow This section describes the flow of Shunsaku operations. Figure 3-1 Shunsaku Flow of Operations The operations listed below are performed while the system is operating. Users can view the Management Console from a Web browser, and use the Management Console to perform these operations easily: Creating Lookup Data Searching and Updating Data Reallocating Search Data Optimizing Director Data Adding a Searcher Adding Directors Changing the Data Insertion Destination Director Deleting Searchers Deleting a Director Refer to the Installation Guide for details of Management Console setup. 3-2

43 Operation Flow Starting the Management Console and Logging in This section describes how to start the Management Console and log in. Note Do not start more than one Management Console at the same time. One of the following browsers is required in order to use the Management Console: Microsoft(R) Internet Explorer 5.01 Microsoft(R) Internet Explorer 5.5 Microsoft(R) Internet Explorer 6.0 Netscape(R) 7 Follow the procedure below to start the Management Console. 1. Start the Web browser and specify the following URL: Host name: Port number: Specify the host name or IP address of the Admin Server. Specify the port number that is used for the Management Console. The port number for the Management Console is set during Interstage Application Server installation. If the port number has not been changed, its initial value is "12000". Refer to the Interstage Application Server Installation Guide for details. Note In an environment in which Interstage Application Server is installed, users can also start the Management Console from the Windows Start menu by selecting [Programs]- [Interstage]-[Application Server]-[Interstage Management Console]. 2. When the Management Console is started, the login screen is displayed. Specify the following information to log in: User name Password For the user name, specify a user ID that is registered in the operating system of the Admin Server. Log in to the Interstage Management Console as a user who belongs to the "Administrators" group. For the user name, specify a user ID that is registered in the operating system of the Admin Server. Log in to the Interstage Management Console as a super user. 3-3

44 Chapter 3: Operation Note When the Management Console is first started following installation, set the environment file backup destination address and the director data backup destination address. Refer to Setting the Backup Destination for the Environment File for information on setting the environment file backup destination. Use the following procedure to set the director data backup destination: 1. In the navigation frame, click the identifier of the director for which the backup destination is to be modified. 2. In the operations frame, click the Backup tab. Specify the Backup directory name, then click the Update settings button. 3-4

45 Starting and Stopping Starting and Stopping This section describes how to start and stop Shunsaku. Starting Follow the procedure below to start Shunsaku. This operation starts the entire Shunsaku system. 1. In the navigation frame, click the system name of the system to be started. 2. In the operations frame, click the Starting and stopping tab, then click the Start button. This starts the Shunsaku system. 3. Click the Refresh button to verify that the system has started. 3-5

46 Chapter 3: Operation Stopping Follow the procedure below to stop Shunsaku. This operation stops the entire Shunsaku system. 1. In the navigation frame, click the system name of the system to be stopped. 2. In the operations frame, click the Starting and stopping tab, then click the Stop button. This stops the Shunsaku system. 3. Click the Refresh button to verify that the system has started. 3-6

47 Creating Lookup Data Creating Lookup Data Lookup data refers to the director data managed by the director and to the search data distributed to each searcher. The input used to create this lookup data is the text files that store XML format data. Figure 3-2 shows the flow of tasks involved in creating lookup data from the text files. Figure 3-2 Creating Lookup Data Note After the data is created, always backup the director data. Refer to 'Backing up director data' in Chapter 4, Maintenance for director data backup details. 3-7

48 Chapter 3: Operation Follow the procedures below to create the lookup data from the text files. 1. On the director server, prepare the text files that store the XML format data. If there are multiple text files, join them together into one file. 'Text file XML format data',in Appendix C for details of the text file contents. Note The Shunsaku XML conversion function can be used to create the text files that store the XML format data. These files are created from the data stored in databases and CSV files, and are used as input files for importing data. Refer to Chapter 5, XML conversion function for details of the XML conversion function. Network drives and network paths cannot be specified for text files. Specify files on local disks. 2. In the navigation frame, click the director identifier. 3. In the operations frame, click the Status tab and click then click the Refresh button. Check and record the number of records. 3-8

49 Creating Lookup Data 4. In the operations frame, click the Data operations tab and specify each option, then click the Execute button. This imports the XML format data stored in the text file and creates the lookup data. Option Name Input Data Create/Delete Create data Select the radio button. Text file name Specify the absolute pathname as the name of a text file on a director server local disk. Create method/add Select the radio button. 5. In the operations frame, click the Status tab and click the Refresh button to verify that the number of records has changed. 3-9

50 Chapter 3: Operation Searching and Updating Data Once the lookup data has been created, the data can be searched and updated. Data searches and updates are executed from applications that use the Shunsaku API. Refer to the Application Development Guide for information on how to develop applications. 3-10

51 Tuning System Operation Tuning System Operation This section describes how to tune system operation. The following processes are used to tune system operation: Reallocating Search Data Optimizing Director Data Adding Searchers and Deleting Searchers Adding Directors Changing the Data Insertion Destination Director and Deleting a Director Guidelines are given below indicating when to execute the tuning operations, search data reallocation, director data optimization, and addition of searchers and directors. Search data re-allocation guideline Reallocate the search data if the data is not allocated evenly between the searchers. Director Data Optimization Guideline Check the director fragmentation rates. If the amount of wasted areas in the director data are increasing, optimize the director data file. Searcher Addition Guideline If the server performance indicates that it can process more data and faster search speeds are desired, or if increases in the volume of data are expected, add a searcher. If the amount of data increases beyond the ideal amount for one server, increase the number of search servers first, then add the searcher. Director Addition Guideline When the number of searchers exceeds the number that can be managed by one director, add a director. Reallocating Search Data When the amount of data in each searcher becomes uneven due to an increase in the number of searchers or due to deletions of large amounts of data, the search data needs to be reallocated. If the amounts of data in each searcher are not even, the Shunsaku search speed is dependent on the searcher that has the largest amount of data. Reallocation of the search data can be used to make the amount of data in each searcher equal. Figure -3 shows a tuning example in which search data is reallocated. 3-11

52 Chapter 3: Operation Figure 3-3 Reallocating Search Data 3-12

53 Tuning System Operation Follow the procedure shown in Figure 3-4 to re-allocate search data. Figure 3-4 Reallocating Search Data 1. In the navigation frame, click the director identifier. 2. In the operations frame, click the Status tab and click the Refresh button to check the data sizes. 3-13

54 Chapter 3: Operation 3. If data size is not evenly distributed across servers, click the Tuning operations tab. In Reallocate/Optimize, select Reallocate and click the Execute button to re-allocate the data. 4. Click the Status tab, and then click the Refresh button to verify the data sizes to confirm that the data is now allocated evenly. Note Data cannot be updated while the search data is being reallocated. 3-14

55 Tuning System Operation Optimizing Director Data After many deletions and updates, the amount of wasted space in the director data storage area may increase and affect the search function. Optimizing the director data removes the wasted storage area from the director data and solves the problem of deteriorating search times. Figure 3-5 Optimizing Director Data Evaluate the fragmentation rate to determine if the director data needs to be optimized. Data fragmentation refers to the condition that occurs when data that should exist as a single block becomes scattered. The access efficiency deteriorates in proportion to the degree of fragmentation. The fragmentation rate indicates the amount of unnecessary data left behind when processing is cancelled during deletion or update operations, as a percentage of the amount of required data stored in the director data file specified by the DataFileFolder parameter in the director environment file. Higher fragmentation rates indicate greater amounts of unnecessary data. Figure 3-6 shows the task flow for optimizing the director data. 3-15

56 Chapter 3: Operation Figure 3-6 Task Flow for Optimizing the Director Data 1. In the navigation frame, click the director identifier. 2. In the operations frame, click the Status tab, and click the Refresh button to check the fragmentation rate. 3-16

57 Tuning System Operation 3. If an increase in the fragmentation rate indicates that director data optimization is required, click the Tuning operations tab In Reallocate/Optimize, select the Optimize, then click the Execute button to optimize the director data. 4. Click the Status tab, and then click the Refresh button to verify that the fragmentation rate after optimization of the director data is 0%. Note Data cannot be updated while the director data is being optimized. An area twice the size of the director data file is used temporarily while the director data is being optimized. Adding Searchers If the server performance indicates that it can process more data and faster search speeds are desired, or if increases in the volume of data are expected, add a searcher. If the amount of data increases beyond the ideal amount for one server, increase the number of search servers first, then add the searcher. Adding a Searcher Deleting an Added Search Server and Searcher The maximum number of searchers that can be registered in one director is

58 Chapter 3: Operation Adding a Searcher Figure 3-7 shows the task flow for adding a searcher. Figure 3-7 Adding a Searcher 1. In the navigation frame, click the identifier of the director to which a searcher is to be added. 2. In the operations frame, click the searcher tab. Increase Number of searchers, then click the Update button to add the searcher. Alternatively, click the Update and reallocation button to re-allocate the search data at the same time. If the system is active, the added searcher is started automatically. 3-18

59 Tuning System Operation Note If the Number of searchers is increased too much, the Windows system resources may be used up and it may be impossible to start some searchers. Check the Status tab and, if some searchers have not started, reduce the "Number of searchers". Then, increase the number of search servers before adding more searchers. 3. Click the Status tab and click the Refresh button to verify that the searcher has started. Adding a Search Server and a Searcher Figure 3-8 shows the task flow for increasing the number of search servers, then adding a searcher. The number of search servers can be increased without using the Management Console. Refer to 'Adding searchers and increasing the number of search servers' in the Operator's Guide (Advanced). Figure 3-8 Adding a Search server and a Searcher 1. Install a searcher at the search server that is being added. Refer to the Installation Guide for details of how to install searchers. 3-19

60 Chapter 3: Operation 2. In the navigation frame, click the identifier of the director to which the searcher is to be added. 3. In the operations frame, click the searcher tab. Check the Create searcher on different server to director option, and in Number of searchers, specify the number of searchers to be added. Under <Add server>, specify the information for the server being added. Click the Add button to specify the server being added. Repeat this process for each search server that is to be added. The searchers specified in Number of searchers are distributed evenly between the search servers. 3-20

61 Tuning System Operation Check the Specify number of searchers on each server option in order to specify a different number of searchers for each server. The number of searchers can be specified for each of the servers displayed in the server list. 4. Click the Update button to increase the number of search servers and add the searchers. Alternatively, click the Update and reallocation button to re-allocate the search data at the same time. If the system is active, the added searcher is started automatically. 5. Click the Status tab and click the Refresh button to verify that the searcher has started. Use the procedures described in Deleting Searchers, to delete a searcher that is allocated to the dierctor server. 3-21

62 Chapter 3: Operation Adding Directors Add a director if the number of searchers exceeds the number that can be managed by one director. The maximum number of directors that can be added is 255. Figure 3-9 shows the task flow for adding a director. Figure 3-9 Adding a Director 1. In the navigation frame, click the director node 3-22

63 Tuning System Operation 2. In the operations frame, click the Create new tab, specify the required options, then click the Create button to add a director and a searcher. If the system is active, the added director and searcher are started automatically. Option Name Input Director definition Searcher definition Director identifier Data file directory name Backup directory name Backup warnings Number of searchers Server Encourage the user to execute backup when the specified number of days have elapsed since execution of the most recent backup. Backup time interval Specify the director identifier. Specify the name of the directory that stores the director data. Specify the name of the directory that stores the backup. Select the checkbox. Specify the number of days. Specify the number of searchers. Select the checkbox if the searcher is to be created on a server other than the director server. Enter the required items on the screen that is displayed if the checkbox is selected. Refer to Operation 3 of Adding a Search Server and a Searcher for details of the input items. 3-23

64 Chapter 3: Operation Notes Specify a backup directory on a different disk to the data file directory to prevent both directories being corrupted if there is a disk error. For the data file directory name and the backup directory name, specify the absolute paths for local disks on the director server. 3. In the navigation frame, click the identifier of the added director. 4. In the operations frame, click the Status tab and click the Refresh button to verify that the director and searcher have started. 3-24

65 Tuning System Operation Changing the Data Insertion Destination Director Change the insertion destination director where the application inserts data. Follow the procedure below to change the data insertion destination director. 1. In the navigation frame, click the director node. 2. In the operations frame, click the Modify tab and change the value of Director that inserts data, then click the Update button. 3-25

66 Chapter 3: Operation Deleting Searchers Use the procedures shown below to delete a searcher from a server or to delete a previously added search server and a searcher. Deleting Searchers Deleting an Added Search Server and Searcher Deleting Searchers Figure 3-10 shows the procedure used to delete a searcher from a server. Figure 3-10 Deleting Searchers 1. Stop Shunsaku. Refer to Stopping for details on stopping Shunsaku. 2. In the navigation frame, click the identifier of the director from which the searcher is to be deleted. 3-26

67 Tuning System Operation 3. In the operations frame, click the searcher tab and decrease the value of the Number of searchers option, then click the Update and reallocation button to delete the searcher. Note The total number of searchers registered to the director cannot be zero (0). 4. Start Shunsaku. Refer to Starting for details on starting Shunsaku. 5. In the navigation frame, click the identifier of the director from which the searcher was deleted. 6. Click the Status tab and click the Refresh button to check the number of searchers. If the director holds a large quantity of data and searchers are deleted, the remaining searchers may not be able to handle all the search data from the deleted searchers. Therefore, check the director status and, while the director is stopped, increase the number of searchers. Refer to Adding Searchers for the method used to add searchers. 3-27

68 Chapter 3: Operation Deleting an Added Search Server and Searcher Figure 3-11 shows the task flow for deleting a previously added search server and a searcher. Figure 3-11 Deleting an Added Search Server and a Searcher 1. Stop Shunsaku. Refer to Stopping for details on stopping Shunsaku. 2. In the navigation frame, click the identifier of the director from which the searcher is to be deleted. 3-28

69 Tuning System Operation 3. In the operations frame, click the searcher tab. If the number of searchers is not specified separately for each server, under <Configuration members>, select the search server that is to be deleted, then click the Delete button. If the number of searchers is specified separately for each server, specify zero (0) as the number of searchers for the search server that is to be deleted. 4. Click the Update and reallocation button to delete the previously added search server and the searcher. 5. Start Shunsaku. Refer to Starting for details on starting Shunsaku. 6. In the navigation frame, click the identifier of the director from which the searcher was deleted. 3-29

70 Chapter 3: Operation 7. Click the Status tab, then click the Refresh button to verify the number of searchers. If the director holds a large quantity of data and searchers are deleted, the remaining searchers may not be able to handle all the search data from the deleted searchers. Therefore, check the director status and, while the director is stopped, increase the number of searchers. Refer to Adding Searchers for the method used to add searchers. Deleting a Director Figure 3-12 shows the procedure for deleting a director from a server. Figure 3-12 Deleting a Director 3-30

71 Tuning System Operation 1. Stop Shunsaku. Refer to Stopping for details on stopping Shunsaku. 2. In the navigation frame, click the director identifier. 3. In the operations frame, click the Delete tab, then click the Delete button to delete the director and searcher. The director data is deleted at the same time. 4. Start Shunsaku. Refer to Starting for details on starting Shunsaku. 5. In the navigation frame, click the director node. 3-31

72 Chapter 3: Operation 6. In the operations frame, click the List tab, then click the Refresh button to verify that the director has been deleted. Note When a director is deleted, the lookup data that was imported to that director is also deleted. If that data is still required, use the shundexport command to export the data to an external file before deleting the director. When a director is deleted, the backup collected by the Management Console backup function is not deleted. If a backup was created, delete the backup that was saved to the backup directory. 3-32

73 Environment Setup Environment Setup This section describes the environment settings. Modifying the Search Environment Modification of the search environment changes the data search conditions. Follow the procedure below to modify the search environment. 1. In the navigation frame, click the system name. 2. In the operations frame, click the Modifying the search environment tab and specify the various options, then click the Update button to modify the search environment. Note If the system is active, the system must be restarted for the changes to take effect. 3-33

74 Chapter 3: Operation Setting the Backup Destination for the Environment File When Management Console operations are used to modify the operating environment file, the file is automatically backed up to the specified directory. Follow the procedures shown below to change the backup address. 1. In the navigation frame, click the system name. 2. In the operations frame, click the Operating environment file backup destination tab, then specify the name of the backup directory, and click the Update button to change the destination. Note For the backup directory name, specify the absolute path of a directory on the local disk of an Admin Server. Specify a backup directory name that is on a different disk to the disk allocated for the environment file to prevent both directories being corrupted if there is a disk error. 3-34

75 Chapter 4 Maintenance This chapter describes Shunsaku maintenance. Backup Recovery Recreating Data for Searches Troubleshooting 4-1

76 Chapter 4: Maintenance Backup Backup refers to saving Shunsaku operation data in order to prevent the data being lost if a disk fault or other error occurs. The following files must be backed up to enable Shunsaku to perform data recovery: Director Data Backup Shunsaku Environment File Backup Director Data To backup director data, create backups of all the files under the director data storage directory that was specified during installation. Shunsaku Environment File When Management Console operations are used to modify the Shunsaku environment file, the Management Console automatically backs up the file when it is changed. Director Data Backup Back up the director data at regular intervals. If a disk fault occurs, the backup enables Shunsaku to be recovered to the state when the backup was performed. Save the director data that is backed up by the Management Console on a different disk to that where the director data storage directory is located. Figure 4-1 shows an example in which the director data is backed up at the end of each day's operation. 4-2

77 Backup Figure 4-1 Director Data Backup Follow the procedures shown below to prepare a backup to be used if a fault occurs. 1. In the navigation frame, click the system name. 4-3

78 Chapter 4: Maintenance 2. In the operations frame of the Management Console, select the Backup tab and click the Backup button. This operation starts backup for all the directors. The backup status of each director is displayed in the Status column. Note After backup is started, director data update processing cannot be executed until backup is complete. Shunsaku Environment File Backup If a disk fault occurs at the disk that contains the storage directory for the Shunsaku environment file, Shunsaku can no longer operate. Therefore, always make a backup of the current information held in the Shunsaku environment file. When Management Console operations are used to modify the Shunsaku environment file, the Management Console automatically backs up the file as soon as it is changed. Refer to 'Setting the Backup Destination for the Environment File' in Chapter 3 for details on how to set the backup destination. The following files are backed up: System environment file Conductor environment file Director environment file Sorter environment file Searcher environment file. Note The Management Console is provided to manage the Shunsaku server (conductor, director, sorter, and searcher) resources. 4-4

79 Recovery Recovery If a disk fault or similar has affected Shunsaku operations, recovery can return Shunsaku to its state at the time of the most recent backup. This section describes the recovery procedures to use if a fault occurs on the disk where the following files are allocated: Director Data Recovery Shunsaku Environment File Recovery Director Data Recovery During operations, if a fault occurs at the disk to which the director data is allocated, Shunsaku posts message "shn30306u" to the director server event log or the system log (syslog). Figure 4-2 shows an example of recovery if, during one day's operation, a fault occurs on the disk to which the director data is allocated. Figure 4-2 Director Data Recovery Figure 4-3 shows the recovery procedures to use for the director data. Note Stop all applications before performing director data file recovery, and ensure that data searches and updates are not performed during recovery. 4-5

80 Chapter 4: Maintenance Figure 4-3 Recovery Procedures for Director Data 1. Check the system messages and determine the fault location. If a disk fault occurs during access to the directory containing the director data file, an error occurs for Shunsaku operations from the Shunsaku API or the Management Console. Check the Shunsaku messages posted to the director server event log or the system log (syslog) to determine the cause of the error. If a disk fault occurred during access to the director data file directory, Shunsaku posts the following message: shn30306u: I/O error has occurred in director data file. file=filename System call=function-name errno=error-number (director) [director-identifier] Shunsaku System Name=Shunsaku-system-name Check this message to find the name of the disk where the fault occurred or the absolute pathname of the file. 2. Enter the shundrecover command b option to declare the start of recovery. If the director is stopped, use the shunsysstart command to start Shunsaku, then enter the shundrecover command. Note If the operating system needs to be restarted during disk replacement, restart the operating system, then perform Step 3 before this step. Ensure that all applications are stopped, start Shunsaku, and then perform this step. shundrecover -s director-identifier -b 3. Switch disks, and re-create the directory where the director data is allocated. 4-6

81 Recovery 4. Use the operating system commands or the disk unit copy function to restore the director data from the backup data. copy command, or similar cp command, or similar 5. Enter the shundrecover command e option and p option to declare the end of recovery. The recovery end declaration recovers the director data to its state at the time the backup was made. shundrecover -s director-identifier -e -p Note The director data that was backed up using the Management Console is used as the backup data during recovery by the shundrecover command. Refer to Command Reference for details of the shundrecover command. Shunsaku Environment File Recovery If a fault occurs at the disk used for the directories where the Shunsaku environment file is allocated (shown below), recover the Shunsaku environment file. Shunsaku install directory\shunsaku\etc /etc/opt/fjsvshnsk/etc Figure 4-4 shows the recovery procedure to use for the Shunsaku environment file. 4-7

82 Chapter 4: Maintenance Figure 4-4 Recovery Procedure for the Shunsaku Environment File 1. Switch the disk where the fault occurred. 2. Use operating system commands to restore the backup data of the Shunsaku environment file. If the searcher runs on a separate server, restore the searcher environment file on the relevant search server. copy command, ftp command, or similar cp command, ftp command, or similar Notes Refer to 'Setting the backup address for the environment file' in Chapter 3, Operation, to check the environment file backup destination. Environment file backup locations The Management Console saves the backup data to the following specified locations: Environment file System environment file Conductor environment file Director environment file Sorter environment file Searcher environment file Storage location Environment file backupdestination folder\servername\system\shunsaku-system-name.cfg Environment file backupdestination folder\servername\conductor\conductor-identifier.cfg Environment file backupdestination folder\servername\director\director-identifier.cfg Environment file backupdestination folder\servername\sorter\sorter-identifier.cfg Environment file backupdestination folder\servername\searcher\ searcher-identifier.cfg 4-8

83 Recovery Environment file System environment file Conductor environment file Director environment file Sorter environment file Searcher environment file Storage location Environment file backup destination directory/server-name/system/shunsaku-systemname.cfg Environment file backup destination directory/server-name/conductor/conductoridentifier.cfg Environment file backup destination directory/server-name/director/directoridentifier.cfg Environment file backup destination directory/server-name/sorter/sorter-identifier.cfg Environment file backup destination directory/server-name/searcher/searcheridentifier.cfg Environment file restore locations Backup data is restored to the following locations: Environment file System environment file Conductor environment file Director environment file Sorter environment file Searcher environment file Storage location Shunsaku install folder\shunsaku\etc\system\shunsaku systemname.cfg Shunsaku install folder\shunsaku\etc\conductor\conductoridentifier.cfg Shunsaku install folder\shunsaku\etc\director\director-identifier.cfg Shunsaku install folder\shunsaku\etc\sorter\sorter-identifier.cfg Shunsaku install folder\shunsaku\etc\searcher\searcheridentifier.cfg 4-9

84 Chapter 4: Maintenance Environment file System environment file Conductor environment file Director environment file Sorter environment file Searcher environment file Storage location /etc/opt/fjsvshnsk/etc/sorter/sorter-identifier.cfg /etc/opt/fjsvshnsk/etc/system/shunsaku systemname.cfg /etc/opt/fjsvshnsk/etc/conductor/conductoridentifier.cfg /etc/opt/fjsvshnsk/etc/director/directoridentifier.cfg /etc/opt/fjsvshnsk/etc/searcher/searcheridentifier.cfg 4-10

85 Recreating Data for Searches Recreating Data for Searches If changes are made to the data that is the source of the data stored in Shunsaku, the searchable data is re-created in order to reflect those changes at Shunsaku. Follow the procedures below to re-create the data for searches. 1. Prepare at the director server the text file that stores the XML format data. If there are multiple text files, join them together into one file. Refer to Appendix C, Notes Concerning XML Documents, for details of the contents of the text files. Note A network drive or network path cannot be specified for a text file. Specify a file on a local disk. 2. In the navigation frame of the Management Console, click the director identifier. 3. In the operations frame of the Management Console, select the Status tab and click the Refresh button. Check and record the number of records. 4-11

86 Chapter 4: Maintenance 4. In the operations frame of the Management Console, select the Data operations tab and specify the items shown below. Then, click the Execute button to import the XML format data that is stored in the text file and to create the data for searches. Item name Data Create / Delete Create data Text file name Replace Input Select the radio button. Enter the name of a text file on the director server local disk. Select the radio button. 5. In the operations frame of the Management Console, select the Status tab and click the Refresh button, then verify that the number of records has changed. 4-12

87 Troubleshooting Troubleshooting This section describes how to deal with errors. Server Errors Errors on the Application Server Server Errors If an error occurs that prevents further use of Shunsaku, the administrator must remove the cause of the error and restart Shunsaku. Figure 4-5 shows the procedures from error occurrence to recovery. Figure 4-5 Procedure from Error to Recovery 1. Refer to the user action indicated for the Shunsaku message posted to the event log or system log (syslog) of the server where the error occurred. Take the required action to remove the cause of the error. Refer to the Messages manual for details of the action required in response to a message. 2. When the cause of the error has been removed, restart the system. Refer to 'Starting' and 'Stopping', in Chapter 3 for details concerning starting and stopping the system. 4-13

88 Chapter 4: Maintenance 3. If an error occurs for a searcher, the system automatically switches from post-degradation usage to normal usage when the system is restarted. However, the search data that was distributed amongst other searchers during degradation is not returned to its pre-degradation state. Therefore, re-allocate the search data. Note Refer to 'Re-allocating Search Data', in Chapter 3 for details concerning search data reallocation. Data cannot be updated while search data re-allocation is in progress. Errors on the Application Server If an Application Server fails due to a power failure or similar, the connection that was established by the application remains connected and occupies Shunsaku resources. Therefore, forcibly disconnect the application s connection in order to release the Shunsaku resources occupied by that connection and restart work. Figure 4-6 shows the procedures from failure of an Application Server to recovery. Figure 4-6 Procedures from failure of a Application Server to recovery 1. In the navigation frame of the Management Console, click Connection to access the connection operations frame. 4-14

89 Troubleshooting 2. In the operations frame, select the List tab and click the Refresh button, then check the connection status. 3. Select the IP address of the failed server, then click the Disconnect button to disconnect the application s connection. 4. Restart the application and resume work. 4-15

90 Chapter 4: Maintenance 4-16

91 Chapter 5 XML Conversion Function This chapter explains the function that converts data to XML format. Overview of the XML Conversion Function Designing the XML Conversion Function Setting up the XML Conversion Function Designing Applications Examples of API Usage Data Types Supported by the XML Conversion Function CSV Formats Supported by the XML Conversion Function 5-1

92 Chapter 5: XML Conversion Function Overview of the XML Conversion Function The XML conversion function converts data stored in a database or CSV file to XML format. Data conversion uses mapping rules that define the correspondence between the input data and the data in XML format that is to be output. Refer to Appendix E, XML Conversion Function Mapping Rules for details on mapping rules. The XML conversion function provides Java APIs that are invoked and used by an application. The following conceptual diagram shows how the XML conversion function is organized. Figure 5-1 Structural Overview for XML Conversion Function Converting Data from Databases to XML Format The XML conversion function converts the results returned from a database to data in XML format according to specified mapping rules. The results returned from a database can be obtained as a ResultSet object. The ResultSet object is in the form of a single two-dimensional table made up of rows and columns that is independent of the database schema or the search method used. This two-dimensional table is referred to as a results table. The XML conversion function converts this ResultSet object to data in XML format according to specified conversion rules. Conversion rules are created by the user as mapping rules based on XML format. They must be prepared in advance in the form of a mapping rule file. The mapping rules include conversion rules that specify which column of a results table corresponds to which element or attribute of data in XML format. 5-2

93 Overview of the XML Conversion Function Refer to Appendix E, XML Conversion Function Mapping Rules for details on the mapping rules. Figure 5-2 Mapping Results Returned from a Database to Data in XML Format Converting Data from CSV Files to XML Format The XML conversion function converts the results returned from the data specified in a CSV file to data in XML format according to specified mapping rules. The XML conversion function also supports the various delimiters, quotation marks and null characters that are used in CSV files. The results returned from the data specified in a CSV file can be obtained as a ResultSet object. The ResultSet object is in the form of a single two-dimensional table made up of rows and columns. This two-dimensional table is referred to as a results table. 5-3

94 Chapter 5: XML Conversion Function Designing the XML Conversion Function To use the XML conversion function, the following information must be specified: Determine the Source of Input Data Determine the Structure of the Data in XML Format Determine the Source of Input Data Select either a database or a CSV file as the source of the input data. Determine the Structure of the Data in XML Format To specify the structure of the data in XML format that is to be output to a text file, the following information must be determined: The number of data in XML format items Whether to output an XML declaration The hierarchical relationship between the data in XML format The correspondence between the columns in the results table and the data in XML format The elements of the data in XML format The attributes of the data in XML format. Notes Hierarchical structure of data in XML format Data in XML format consists of a single root element under which sub-elements and element data exist. These elements are all related and their relationships can be expressed as a parentchild relationship or a sibling relationship, etc. In a hierarchical relationship, higher-level tags are parents and lower-level tags are children. Elements on the same level that possess the same parent are siblings. If some simple data in XML format were represented as a hierarchical relationship, it could be expressed as shown in the following figure. 5-4

95 Designing the XML Conversion Function Figure 5-3 Sample of Hierarchical Structure of Data in XML Format Associating parent and child elements According to the mapping rules, elements within the data in XML format to be output that possess element data are referred to as column elements and elements that possess other elements are referred to as parent elements. In the following figure, PERSON and DETAIL are parent elements, and the five subordinate elements NUM, NAME, JOB, LOCNUM and LOC are column elements. Figure 5-4 Example of Parent Elements and Column Elements 5-5

96 Chapter 5: XML Conversion Function Figure 5-5 Hierarchy of Parent Elements and Column Elements from Previous Example Determine the Output Format of the Data in XML format Specify the output format of the data in XML format that the application will receive from the XML conversion function. The following output formats can be selected: String Stream File DOM Tree SAX Note The data must be in the form of a file for Shunsaku to create the data for searches. 5-6

97 Setting up the XML Conversion Function Setting up the XML Conversion Function This section explains how to set up the XML conversion function. Setting up the XML Conversion Function APIs This section explains how to set up the XML conversion function APIs. API Configuration This section explains how the XML conversion function APIs are configured. The following APIs are created when the Shunsaku APIs are installed: XML Conversion Function API Package XML Processor Note To use an application that uses the XML conversion function APIs, a JDBC driver that is compatible with the database to be used is needed in addition to the above API package. The user must provide the JDBC driver. XML Conversion Function API Package When the Shunsaku APIs are installed, a jar file such as the following is created as the API package to be used with the XML conversion function. Shunsaku installation folder\shunsaku\lib\xmlconv.jar /opt/fjsvshnsk/lib/xmlconv.jar XML Processor When the Shunsaku APIs are installed, the following two jar files are created as an XML processor, compatible with the Java API for XML Processing (JAXP). Shunsaku installation folder\shunsaku\lib\xml-apis.jar Shunsaku installation folder\shunsaku\lib\xercesimpl.jar /opt/fjsvshnsk/lib/xml-apis.jar /opt/fjsvshnsk/lib/xercesimpl.jar 5-7

98 Chapter 5: XML Conversion Function Setting Environment Variables Set the environment variables required for using the XML conversion function APIs: Add the following jar files to the CLASSPATH environment variable: xmlconv.jar xml-apis.jar xercesimpl.jar JDBC driver jar file If XML processors other than xercesimpl.jar are specified in the CLASSPATH, xercesimpl.jar must be specified before these XML processors. Examples of how to set these environment variables are shown below. In these examples, Oracle 8i/9i JDBC driver for JDK 1.3 (classes12.zip) is used as the JDBC driver. Example To install Shunsaku APIs in C:\Program Files\Interstage Shunsaku: SET CLASSPATH=C:\Program Files\Interstage Shunsaku\lib\classes12.zip;%CLASSPATH% SET CLASSPATH=C:\Program Files\Interstage Shunsaku\lib\xercesImpl.jar;%CLASSPATH% SET CLASSPATH=C:\Program Files\Interstage Shunsaku\lib\xmlapis.jar;%CLASSPATH% SET CLASSPATH=C:\Program Files\Interstage Shunsaku\lib\xmlconv.jar;%CLASSPATH% Example 1 For bash, Bourne shell or Korn shell: CLASSPATH=/opt/FJSVshnsk/lib/classes12.zip:$CLASSPATH CLASSPATH=/opt/FJSVshnsk/lib/xercesImpl.jar:$CLASSPATH CLASSPATH=/opt/FJSVshnsk/lib/xml-apis.jar:$CLASSPATH CLASSPATH=/opt/FJSVshnsk/lib/xmlconv.jar:$CLASSPATH export CLASSPATH Example 2 For C shell: setenv CLASSPATH /opt/fjsvshnsk/lib/classes12.zip:${classpath} setenv CLASSPATH /opt/fjsvshnsk/lib/xercesimpl.jar:${classpath} setenv CLASSPATH /opt/fjsvshnsk/lib/xml-apis.jar:${classpath} setenv CLASSPATH /opt/fjsvshnsk/lib/xmlconv.jar:${classpath} 5-8

99 Setting up the XML Conversion Function Creating a Mapping Rule File A mapping rule file must be created. Refer to Appendix E, XML Conversion Function Mapping Rules for details on how to define a mapping rule file. 5-9

100 Chapter 5: XML Conversion Function Designing Applications The following figure shows the input and output objects of the XML conversion function. Figure 5-6 XML Conversion Function I/O Objects The ResultSet object is an instance of the java.sql.resultset interface implementation class. The other objects are instances of classes provided by the XML conversion function. The results table used as the input source of the conversion is the ResultSet object. The XMLGenerator object uses the GeneratorMap object and the ResultSet object as input to create a DocumentFountain object, which is a collection of data in XML format. The user extracts the desired data from the DocumentFountain object. Note that because a CSVResultSet object cannot be obtained directly from a CSV file, it is necessary to write an application that creates a CSVSource object. The ResultSet is obtained from this. The input and output objects used when creating a ResultSet object from a CSV file are shown in the following figure. Figure 5-7 I/O Objects used when Creating a ResultSet Object from a CSV File 5-10

101 Designing Applications The CSVResultSet object implements the java.sql.resultset interface. It can be used as input for XMLGenerator. XMLGenerator can extract data in XML format in a variety of formats according to specified mapping rules. It can also extract partial instead of complete data in XML format, and split a single input results table into multiple sets of output data in XML format. Data in XML format can be extracted in the following output formats: String Stream File DOM Tree SAX Note Data must be in the form of a file for Shunsaku to create data for searches. List of APIs The XML conversion function provides APIs written in Java. The following table lists the APIs that are available. Table 5-1 APIs Written in Java Package Interface Inheritance Method Function com.fujitsu. xmlconverter DocumentFountain None getxmldocument Obtains a single data in XML format object that is to be created. hasxmldocument Checks whether any XML data created still exists. setdestination Sets the output destination for the data in XML format to be created when the output format is a Stream, File or SAX. setencoding Sets the encoding of the data in XML format to be created. com.fujitsu. xmlconverter CSVSource java.lang. Object CSVSource Creates an instance of CSVSource. setencoding Sets the character encoding of data stored in a CSV file. setdelimiter Sets the delimiter for data stored in a CSV file. setquote Sets the quotation mark character for data stored in a CSV file. 5-11

102 Chapter 5: XML Conversion Function Package Interface Inheritance Method Function setnullcharacter Sets the null character for data stored in a CSV file. setfirstrecordto Column Specifies whether to process the record in the first row of data stored in a CSV file as the column name. setrandommode Specifies whether to change the method used to maintain data in the CSV ResultSet from sequential reading to memory loading. getresultset Obtains ResultSet from the data stored in a CSV file. setenableescape Char Specifies the method used to enter delimiter characters, quotation marks, and the backslash (\) as data. GeneratorMap java.lang. Object GeneratorMap Creates an instance of GeneratorMap. XMLGenerator java.lang. Object XMLGenerator Creates an instance of XMLGenerator. createdocument Fountain Creates an instance of DocumentFountain. XMLConverterException java.lang. Object java.lang. Throwable java.lang. Exception getcode getexception getmessage Obtains the error code. Obtains the internal exception. Obtains the error message. CSVException java.lang. Object CSVException Creates an instance of CSVException. java.lang. Throwable java.lang. Exception com.fujitsu. xmlconverter. XMLConverter Exception 5-12

103 Designing Applications Package Interface Inheritance Method Function XMLGenException` java.lang. Object XMLGenException Creates an instance of XMLGenException. java.lang. Throwable java.lang. Exception com.fujitsu. xmlconverter. XMLConverter Exception 5-13

104 Chapter 5: XML Conversion Function Processing Procedure of Applications that use the XML Conversion Function This section explains the flow of processing performed by applications that use the XML conversion function. Refer to Appendix D, XML Conversion Function APIs for details on each class and method. Procedure for Converting Data in a Database to XML Format The following figure shows the procedure used to convert data in a database to XML. Figure 5-8 Procedure used to Convert Database Data to XML Format 1. Connect to database Send a request to connect to the database and establish the connection. 2. Create ResultSet object Use SQL to perform database searches and obtain the ResultSet object. 3. Create GeneratorMap object Specify a mapping rules file and create the GeneratorMap object. 4. Create XMLGenerator object Create the XMLGenerator object. 5-14

105 Designing Applications 5. Create DocumentFountain object Invoke the document creation method for the XMLGenerator object, createdocumentfountain(). When doing so. specify the ResultSet object and the GeneratorMap object, the output format and the information needed for the output format as arguments. This will cause the DocumentFountain object to be created and returned to the application. Note that the output format of the data in XML format extracted from the DocumentFountain object is determined at this time. 6. Set character encoding used during output Invoke the character encoding setting method for the DocumentFountain object, setencoding(), from the application. Specify the character encoding of the data in XML format to be created. When no character encoding is specified, the use of UTF-8 is assumed by default. 7. Output data in XML format Invoke the document extraction method for the DocumentFountain object, getxmldocument(), Receive the data in XML format in one of the following formats: String Stream File DOM tree SAX. More than one set of data in XML format can be created from a single ResultSet object by invoking the document extraction method getxmldocument() more than once. The data in XML format will be created and returned to the application in sequence. 5-15

106 Chapter 5: XML Conversion Function Procedure for Converting Data in a CSV File to XML Format The following figure shows the procedure used to convert data in a CSV file to XML format. Figure 5-9 Procedure used to Convert CSV File Data to XML Format 1. Create CSVSource object. Create the CSVSource object from the specified CSV file name. 2. Set CSV file information Specify the following items for the created CSVSource object: Character encoding setting Read the CSV file according to the specified character encoding and convert the content of the read CSV file to String type. Delimiter setting Set the delimiter used within the CSV file. The default is the comma (,). Quotation mark character setting Set the quotation mark character used within the CSV file. By default, no quotation mark character is specified. Setting the method for entering delimiters, quotation marks, and "\" as data Set the method used to enter delimiter characters, quotation marks, and the backslash (\) as data. The default is to add a backslash (\) before each of the characters. 5-16

107 Designing Applications Null character setting Set the character that will be used within the CSV file to express a null value. The default value is "" (blank character). Column name setting Specify if the first row of a CSV file should be processed as the column name. By default, the first row is not designated as the column name. Data retention method setting Set the method used to maintain data in CSVResultSet. Either of the following two methods can be selected: i) Sequential reading Memory consumption is small, but performance may be poor in some situations. ii) Memory loading Performance is good, but a large amount of memory may be consumed. The default method is sequential reading. 3. Create CSVResultSet object Obtain CSVResultSet from the created CSVSource object. 4. Create GeneratorMap object Specify a mapping rules file and create the GeneratorMap object. 5. Create XMLGenerator object Create the XMLGenerator object. 6. Create DocumentFountain object Invoke the document creation method for the XMLGenerator object, createdocumentfountain(). When doing so, specify the ResultSet object and the GeneratorMap object, the output format, and the information needed for the output format. This will cause the DocumentFountain object to be created and returned to the application. Note that the output format of the data in XML format extracted from the DocumentFountain object is determined at this time. 7. Set character encoding used during output Invoke the character encoding setting method for the DocumentFountain object, setencoding(), from the application. Specify the character encoding of the data in XML format to be created. When no character encoding is specified, the use of UTF-8 is assumed by default. 8. Output data in XML format Invoke the document extraction method for the DocumentFountain object, getxmldocument(). Receive the data in XML format in one of the following formats: String Stream File DOM tree SAX. More than one set of data in XML format can be created from a single ResultSet object by invoking the document extraction method, getxmldocument(), more than once. The data in XML format will be created and returned to the application in sequence. Notes Refer to CSV Formats Supported by the XML Conversion Function for the CSV formats that are supported by the XML conversion function. 5-17

108 Chapter 5: XML Conversion Function Examples of API Usage This section provides examples of applications that use the XML conversion function APIs. Figure 5-10 Processing Procedure for Applications using the XML Conversion Function Startup through to ResultSet Acquisition (Database Records) The following example shows how to connect to a database and obtain all the column data from a specified table as the ResultSet object. Oracle9i is used as the database in this example. A table must already exist in the database before the operations shown in this example can be performed. // Test program for XMLconverter // import java.sql.drivermanager; import java.sql.connection; import java.sql.resultset; import java.sql.statement; import oracle.jdbc.oracledriver; 5-18

109 Examples of API Usage import com.fujitsu.xmlconverter.xmlgenerator; import com.fujitsu.xmlconverter.generatormap; import com.fujitsu.xmlconverter.documentfountain; public class Conv_Test { public static void main(string[] args) { ResultSet rs = null; try { Class.forName("oracle.jdbc.driver.OracleDriver"); Connection dbcon=null; dbcon = DriverManager.getConnection( "jdbc:oracle:thin:@xx.yy.zz.fujitsu.com:1521:db", "username","password" ); Statement stmt = dbcon.createstatement( ResultSet.TYPE_SCROLL_SENSITIVE, ResultSet.CONCUR_UPDATABLE); rs = stmt.executequery("select * FROM PERSON"); } catch(exception e) { e.printstacktrace(); return; } Startup through to ResultSet Acquisition (CSV File) The following example shows how data stored in a specified CSV file is obtained as the ResultSet object. In this example, the first row of the CSV file is not set as the column name. Also, the CSV file contains lines in which delimiters, quotation marks, or backslashes (\) are entered as data, as shown below. Lines where delimiters, quotation marks, or backslashes (\) are entered as data // Test program for XMLconverter // import java.sql.connection; import java.sql.resultset; import java.sql.statement; import com.fujitsu.xmlconverter.xmlgenerator; import com.fujitsu.xmlconverter.generatormap; import com.fujitsu.xmlconverter.documentfountain; import com.fujitsu.xmlconverter.csvsource; public class Conv_Test { public static void main(string[] args) { ResultSet rs = null; try { // Create CSVSource object CSVSource csvsource = new CSVSource("source.csv") ; csvsource.setencoding("sjis"); // Encoding setting 5-19

110 Chapter 5: XML Conversion Function csvsource.setdelimiter(','); // Delimiter setting csvsource.setquote('\"'); // Quotation mark character setting csvsource.setenableescapechar(false); // // Setting method for entering delimiters, quotation marks, and "\" as data csvsource.setfirstrecordtocolumn(false); // Column name setting ( default ) // Obtain ResultSet object rs = csvsource.getresultset(); } catch(exception e) { e.printstacktrace(); return; } // Obtain ResultSet Creation of XMLGenerator to Acquisition of Data in XML Format The following example shows how the APIs are used, from the creation of XMLGenerator through to acquisition of data in XML format. String The following example extracts and outputs data in XML format in String format, after XMLGenerator has performed a conversion based on a ResultSet object that has been obtained from data stored in either a database or CSV file. XMLGenerator gen = null; GeneratorMap map = null; DocumentFountain df = null; try { gen = new XMLGenerator(); map = new GeneratorMap("map01.xml"); df = gen.createdocumentfountain( DocumentFountain.TYPE_STRING,rs, map ); df.setencoding("sjis"); } catch(exception e) { e.printstacktrace(); return; } String ResultStr = null; try { while(df.hasxmldocument()) { ResultStr = (String)df.getXMLDocument(); System.out.println("Result:"+ResultStr); } rs.close(); // If the input is from a database, connection release is also required. 5-20

111 Examples of API Usage } } } catch(exception e) { e.printstacktrace(); return; } Stream The following example extracts and outputs data in XML format in Stream format, after XMLGenerator has performed a conversion based on a ResultSet object that has been obtained from data stored in either a database or CSV file. import java.io.file; import java.io.fileoutputstream; : XMLGenerator gen = null; GeneratorMap map = null; DocumentFountain df = null; try { gen = new XMLGenerator(); map = new GeneratorMap("map01.xml"); df = gen.createdocumentfountain( DocumentFountain.TYPE_STREAM, rs, map ); df.setencoding("sjis"); } catch(exception e) { e.printstacktrace(); return; } FileOutputStream ResultStream = null; try { while(df.hasxmldocument()) { ResultStream = new FileOutputStream(new File("filename")); df.setdestination(resultstream); df.getxmldocument(); ResultStream.close(); } rs.close(); // If the input is from a database, connection release is also required. } catch(exception e) { e.printstacktrace(); return; } } } 5-21

112 Chapter 5: XML Conversion Function File The following example extracts and outputs data in XML format as a file after XMLGenerator has performed a conversion based on a ResultSet object that has been obtained from data stored in either a database or CSV file. XMLGenerator gen = null; GeneratorMap map = null; DocumentFountain df = null; try { gen = new XMLGenerator(); map = new GeneratorMap("map01.xml"); df = gen.createdocumentfountain( DocumentFountain.TYPE_FILE, rs, map ); df.setencoding("sjis"); } catch(exception e) { e.printstacktrace(); return; } try { while(df.hasxmldocument()) { df.setdestination("filename"); df.getxmldocument(); } rs.close(); // If the input is from a database, connection release is also required. } catch(exception e) { e.printstacktrace(); return; } } } DOM Tree The following example extracts data in XML format as a DOM tree for use in an application, after XMLGenerator has performed a conversion based on a ResultSet object that has been obtained from data stored in either a database or CSV file. import org.w3c.dom.document; : XMLGenerator gen = null; GeneratorMap map = null; DocumentFountain df = null; try { gen = new XMLGenerator(); map = new GeneratorMap("map01.xml"); df = gen.createdocumentfountain( DocumentFountain.TYPE_DOM, rs, map ); df.setencoding("sjis"); } catch(exception e) 5-22

113 Examples of API Usage { e.printstacktrace(); return; } Document docroot = null; try { while(df.hasxmldocument()) { docroot = (Document)df.getXMLDocument(); // // This section specifies application processing that uses docroot received as a DOM tree. } rs.close(); // If the input is from a database, connection release is also required. } catch(exception e) { e.printstacktrace(); return; } } } SAX The following example extracts data in XML format 4as SAX for use in an application, after XMLGenerator has performed a conversion based on a ResultSet object that has been obtained from data stored in either a database or CSV file. The user should set an instance of a class that implements the org.xml.sax.contenthandler interface as the argument of the setdestination() method. In the following example, the SAXhandler class implements the org.xml.sax.contenthandler interface separately. import org.xml.sax.helpers.defaulthandler; : XMLGenerator gen = null; GeneratorMap map = null; DocumentFountain df = null; try { gen = new XMLGenerator(); map = new GeneratorMap("map01.xml"); df = gen.createdocumentfountain( DocumentFountain.TYPE_SAX,rs, map ); df.setencoding("sjis"); } catch(exception e) { e.printstacktrace(); return; } try { while(df.hasxmldocument()) { // // This section specifies application processing received as a SAX event. SAXhandler sh = new SAXhandler(); 5-23

114 Chapter 5: XML Conversion Function df.setdestination(sh); df.getxmldocument(); } rs.close(); // If the input is from a database, connection release is also required. } catch(exception e) { e.printstacktrace(); return; } } } class SAXhandler extends DefaultHandler { : } Sample Program for the XML Conversion Function The sample program for the XML conversion function is provided on the Shunsaku CD-ROM. CD-ROM drive:\sample /directory where CD-ROM is mounted/sample Note Refer to Readme.txt, located in the same directory as the sample program for a detailed description of the sample program for the XML conversion function. 5-24

115 Data Types Supported by the XML Conversion Function Data Types Supported by the XML Conversion Function The column data in the results table is expressed using SQL. The SQL types are defined using java.sql.types. If the SQL types in the results table are not supported by the XML conversion function, the XML conversion function will report an exception during the conversion process. Results Table Data Types Compatible with the XML Conversion Function The following table shows SQL types in the results table that can be used with the XML conversion function. Table 5-2 Compatibility of Data Types in the Results Table with the XML Conversion Function SQL type in results table Description Compatibility ARRAY Array object - BIGINT 64-bit integer BINARY (*1) Fixed-length binary BIT 1-bit integer BLOB (*1) Large binary CHAR Fixed-length character string CLOB Large character string DATE Date DECIMAL Decimal DISTINCT User-defined type - DOUBLE FLOAT Double precision floating-point number Double precision floating-point number INTEGER 32-bit integer JAVA_OBJECT Java-specific object - LONGVARBINARY (*1) Variable-length binary LONGVARCHAR Variable-length character string NULL Null

116 Chapter 5: XML Conversion Function SQL type in results table Description Compatibility NUMERIC Decimal OTHER Undefined type - REAL Single precision floating-point number REF OID container - SMALLINT 16-bit integer STRUCT Structure object - TIME Time TIMESTAMP Date and time TINYINT 8-bit integer VARBINARY (*1) Variable-length binary VARCHAR Variable-length character string represents data types supported - represents data types not supported *1 binary data 5-26

117 Data Types Supported by the XML Conversion Function Correspondence between Results Table and Database Data Types The following table shows the correspondence between the SQL types that can be extracted to the results table and data types used in various databases. Table 5-3 Correspondence between SQL Types for Result Table and Database Data Types SQL type that can Data type in database be extracted to results table PostgreSQL Oracle SQL Server ARRAY BIGINT int8 - bigint BINARY bytea - binary, timestamp BIT bool - bit BLOB - BLOB - CHAR bpchar CHAR, NCHAR char, nchar, uniqueidentifier CLOB - CLOB, NCLOB - DATE date DATE - DECIMAL - - smallmoney, money DISTINCT DOUBLE float8, money - - FLOAT float4 - - INTEGER int4, serial, oid - int JAVA_OBJECT LONGVARBINARY - LONG RAW image LONGVARCHAR - LONG text, ntext NULL NUMERIC numeric, decimal NUMBER numeric OTHER REAL - - real, float (5) 5-27

118 Chapter 5: XML Conversion Function SQL type that can Data type in database be extracted to results table PostgreSQL Oracle SQL Server REF SMALLINT int2 - smallint STRUCT TIME time - - TIMESTAMP timestamp, timestampz, datetime TIMESTAMP, TIMESTAMP (6) datetime, smalldatetime TINYINT - - tinyint VARBINARY - RAW varbinary VARCHAR text, varchar, lztext VARCHAR2, NVARCHAR varchar, nvarchar, sql_variant - represents data types not supported 5-28

119 CSV Formats Supported by the XML Conversion Function CSV Formats Supported by the XML Conversion Function This section describes the CSV formats that are supported by the XML conversion function. Character Data The two methods shown below can be used to enter delimiter characters, quotation marks, and the backslash (\) as character data. Select the method in accordance with the argument value specified in the setenableescapechar() of the CSVSource class. If the argument value is "false": Method for entering delimiter characters as data Enclose the data in quotation marks. Method for entering quotation marks as data Enclose the data in quotation marks, and enter two consecutive quotation marks for a quotation mark character that is to be processed as data. If a quotation mark character is entered partway through the data, the quotation mark character can also be entered provided that the data as a whole is not enclosed in quotation marks. Method for entering "\" as data Enter the backslash (\) as is. If the argument value is "true": Method for entering delimiter characters as data Enclose the data in quotation marks, or add a backslash (\) before the delimiter character. Method for entering quotation marks as data Add a backslash (\) before the quotation mark character that is to be processed as data. If a quotation mark character is entered partway through the data, the quotation mark character can also be entered provided that the data as a whole is not enclosed in quotation marks. Method for entering "\" as data Add a backslash (\) before the backslash that is to be processed as data. The example below shows how delimiter characters, quotation marks, and the backslash (\) are entered as data. (This example assumes that the comma (,) was specified as the delimiter character.) Example To enter $1,200, He said "Hello.", and \100 as data, enter these in the CSV file as follows: If the CSVSource class setenableescapechar() argument value is "false": "$1,200", "He said ""Hello.""", "\100" 5-29

120 Chapter 5: XML Conversion Function If the CSVSource class setenableescapechar() argument value is "true: (*1) "$1,200", "He said \"Hello.\"", \\100 *1 Instead of "$1,200", $1\,200 can also be entered. Linefeed Characters Linefeed characters (CR, LF, or CRLF) included in data that is enclosed in quotation marks are output to XML format data as is, even if the output destination is an attribute value or an element. The linefeed characters (CR, LF, or CRLF) in data enclosed in quotation marks are not converted to character references ( ). From XML 1.0, if XML format data that includes linefeed characters is given to an XML processor, the linefeed characters are converted as follows: If the data is specified in an element: If linefeed characters (CR, LF, or CRLF) are included in an element, all linefeed characters are converted to LF (0x0A). If the data is specified in an attribute value: If linefeed characters (CR, LF, or CRLF) are included in an attribute value, all linefeed characters are converted to blanks (0x20). 5-30

121 Appendix A Allowable Values The following table shows the values allowed for Shunsaku. Table A-1 Shunsaku Permitted Values Item name Number of directors that can be registered to one conductor Number of searchers that can be registered to one director Volume of data that can be registered to one searcher Shunsaku system name Conductor identifier Director identifier Sorter identifier Searcher identifier Host name of director server Host name of search server Length per line in environment file Name of director data file storage directory Allowable values Up to 255 Up to 20 Up to 2 gigabytes Up to 8 bytes (alphanumeric characters) Up to 8 bytes (alphanumeric characters) Up to 8 bytes (alphanumeric characters) Up to 8 bytes (alphanumeric characters) Up to 8 bytes (alphanumeric characters) Up to 127 bytes Up to 127 bytes Up to 4095 bytes (excluding linefeed characters) Up to 220 bytes A-1

122 Appendix A: Allowable Values A-2

123 Appendix B Estimating Resources This appendix describes how to estimate Shunsaku resources. Estimating Memory Requirements Estimating Disk Requirements Estimating Resources used by the XML Conversion Function B-1

124 Appendix B: Estimating Resources Estimating Memory Requirements This section describes how to estimate the memory resources for the Shunsaku server. Estimate for Operation on a Single Server Estimation when the Searcher runs on a Separate Server First, determine the base values for estimations. Seven types of base value are used. The formula characters shown in the following table are used to represent base value items in the formulae included in the explanations below. Table B-1 Base Values for Estimating Memory Resources Item Total number of records Standard record length Number of directors Number of searchers Maximum number of simultaneous connections to the conductor Maximum number of responses Query expression length Formula character N L D S C R Q Explanation This is the total number of units of data stored in Shunsaku. This is the average length of each of the data records stored in Shunsaku. This is calculated in bytes. To obtain this value, divide the total amount of data by the total number of records. This is the number of directors. This is the number of searchers. This is the maximum number of connections to Shunsaku at one time. This is the maximum number of units that can be exported by a single search. This is specified when a search request is issued, and is the average length of the search expression, return expression, and sort expression. This is calculated in bytes. Use these base values in the following formulae to calculate the memory requirements. B-2

125 Estimating Memory Requirements Estimate for Operation on a Single Server Use the following formula to estimate the amount of memory used at the director server: Amount of memory used at server (bytes) = C x ( x R + 2 x N + 10 x S + L x R) + C x D x ( Q x L + L x R) + 90 x N x L x D x S + L x N Calculation Example Total number of records (N): Standard record length (L): 1000 Number of directors (D): 1 Number of searchers (S): 2 Maximum number of simultaneous connections to the conductor (C): 100 Maximum number of responses (R): 50 Query expression length (Q): 1000 When these values are used for the calculation, the amount of memory used is about 1.27 gigabytes. Amount of memory used at the director server = x ( x x x x 50) x 1 x ( x x 50) + 90 x x x x x = (bytes) = 1.27 (gigabytes) Estimation when the Searcher runs on a Separate Server Use the following formula to estimate the amount of memory used at the director server: Amount of memory used at the director server (bytes) = C x ( x R + N + L x R) + C x D x ( Q x L + L x R) + 80 x N x L x D Use the following formula to estimate the amount of memory used at the search server. For the base values of the number of searchers and the number of data units, use the values from the search server. Amount of memory used at the search server (bytes) = x S + C x (10 x S + N ) + 10 x N + L x N Calculation Example Total number of records (N): Standard record length (L): 1000 Number of directors (D): 1 Number of searchers (S): 2 Maximum number of simultaneous connections to the conductor (C) : 100 Maximum number of responses (R): 50 Query expression length (Q): 1000 B-3

126 Appendix B: Estimating Resources When these values are used for the calculation, the amount of memory used at the director server is about 765 megabytes, and the amount used at the search server is about 535 megabytes. Amount of memory used at the director server = x ( x x 50) x 1 x ( x x 50) + 80 x x x 1 = (bytes) = 765 (megabytes) Amount of memory used at the search server = x x (10 x ) + 10 x x = (bytes) = 535 (megabytes) B-4

127 Estimating Disk Requirements Estimating Disk Requirements This section describes how to estimate Shunsaku server disk requirements. Estimates are required for the following: Director data Operation log files and core files Estimating Disk Space for Director Data This section describes how to estimate space for director data. Director data is allocated to the disk specified during setup. First, determine the base values for estimations. Two types of base value are used. The formula characters shown in the following table are used to represent base value items in the formula included in the explanation below. Table B-2 Base Values for Estimating Director Data Space Item Total number of records Standard record length Formula character N L Explanation This is the total number of units of data stored in Shunsaku. This is the average length of each of the data records stored in Shunsaku. This is calculated in bytes. To obtain this value, divide the total amount of data by the total number of records. Use these base values in the following formula to calculate the disk requirements. Use the following formula to estimate the amount of space used by the director data on its allocated disk: Disk space used for director data (bytes) = N x (70 + L ) x 2.6 Calculation Example Total number of records (N): Standard record length (L): 1000 When these values are used for the calculation, the disk space used is about 1.30 gigabytes. Disk space used for director data= x ( ) x 2.6 = (bytes) = 1.30 (gigabytes) B-5

128 Appendix B: Estimating Resources Estimating Disk Space for Operation Log Files and Core Files This section describes how to estimate disk space for the operation log files and core files. Operation log files and core files are allocated on the disk where Shunsaku is installed. Operation log files and core files are allocated on the disk where /var/opt/fjsvshnsk is located. First, determine the base values for estimations. Three types of base value are used. The formula characters shown in the following table are used to represent base value items in the formula included in the explanation below. Table B-3 Base Values for Estimating Operation Log File and Core File Space Item Number of directors Number of searchers Memory estimate Formula character D S M Explanation This is the number of directors. This is the number of searchers. This is the estimate of memory used calculated from the memory resource estimates. The unit used for this value is megabytes. Use the base values in the following formulae to calculate the disk requirements. Use the following formula to estimate the amount of disk space used on the disk where the operation log files and core files are allocated: Disk space for operation log files and core files (megabytes) = x D x S + M Calculation Example Number of directors (D): 1 Number of searchers (S): 2 Memory estimate (M): 1300 When these values are used for the calculation, the disk space used is about 2.25 gigabytes. Disk space for operation log files and core files = x x = 2300 (megabytes) = 2.25 (gigabytes) B-6

129 Estimating Disk Requirements Use the following formula to estimate the amount of disk space used for the director server operation log files and core files. For the memory estimate base value, use the value estimated for the director server. Disk space for the director server operation log files and core files (megabytes) = x D + M Use the following formula to estimate the amount of disk space used for the search servers operation log files and core files. For the number of searchers and memory estimate base values, use the values for that search server. Disk space for search server operation log files and core files (megabytes) = 200 x S + M Calculation Example Number of directors (D): 1 Number of searchers (S): 2 director server memory estimate (M): 765 Search server memory estimate (M): 535 When these values are used for the calculation, the director server disk space used is about 1.33 gigabytes, and the search server disk space used is about 935 megabytes. Disk space used for director server operation log files and core files = x = 1365 (megabytes) = 1.33 (gigabytes) Disk space used for search server operation log files and core files = 200 x = 935 (megabytes) B-7

130 Appendix B: Estimating Resources Estimating Resources used by the XML Conversion Function This section describes how to estimate the resources required when the XML conversion function is used. Use the following formula to estimate the size of the data in XML format output when one record of a results table is converted to one item of data in XML format that has no tag layer or attributes: Table B-4 Estimation Formula Definition unit Variable factors Size (bytes) The data in XML format that is output Number of bytes per column Number of columns per record Number of records Number of bytes in tag name XML declaration number-of-records x ( XML-declaration + ( number-of-columns-per-record + 1 ) x ( number-of-bytes-per-column + 2 x numberof-bytes-in-tag-name + 5 ) ) Calculation Example An example calculation of an estimate for the size of the data in XML format that is output is shown below. If the variable factors are: Number of bytes per column: 30 (bytes) Number of columns per record: 20(columns) Number of records: 1,000 (records) Number of bytes in tag name: 20 (bytes) XML declaration: 21 (bytes) Output Data in XML Format 1,000 x (21 + (20 + 1) x ( x )) =1,596,000 bytes (about 1.6MB) B-8

131 Appendix C Notes on XML Documents This appendix provides notes on XML Documents stored in Shunsaku. It also gives details on searching. Format of XML Documents XML Documents in Text Files Notes on XML Format C-1

132 Appendix C: Notes on XML Documents Format of XML Documents The XML Documents stored in Shunsaku consist of well-formed XML. A well-formed XML document is a document that meets the standards listed below in order to conform to XML document specifications. In addition, only the body of XML Documents can be stored without an XML declaration, DTD, or similar at the start. Only one root element present. If there is a start tag, an end tag is also required. Tag nesting is entered correctly. Shunsaku regards characters starting with "<" as the start tag for the root element, and handles all data up to the corresponding end tag as one XML Document. In XML Documents, characters such as blanks, tags, linefeed characters, and so on, are handled without processing. Figure C-1 Format of XML Documents In the example above, all data between the "<!-" comment and the root element end tag "</doc>" is handled as an XML Document. Comments after the end tag are not included in the XML Documents. Similarly, if there is a line feed character after the end tag, the line feed character is not included in the XML Document. C-2

133 XML Documents in Text Files XML Documents in Text Files Multiple XML Documents can be stored one after the other in a single text file. In this case, when Shunsaku imports XML Documents directly from a text file, all of the XML Documents can be imported in one operation. Shunsaku delimits each document when storing multiple XML Documents in a single file. Note After the root element end tag, the next XML declaration and any comments entered with "<!-" are handled as the next XML document. Shunsaku ignores line feed characters, blank lines, and any characters other than XML declarations and comments that are entered after the root element end tag and before the root element start tag of the next XML Document. Example 1 Shunsaku imports XML Documents, in which comments are included before the root element start tag and after the end tag, in one file: Document A <!-- Starting A --> <A> <B>aaam</B> </A> <!-- End A --> Document B <!-- Starting B --> <A> <C>bbbm</C> </A> <!-- End B --> When the above two documents, which are stored in one file, are imported, Shunsaku assumes that the end of Document A is where it detects the root element end tag </A>. Therefore, when Shunsaku exports the entire Document B, the <!--End A--> of Document A is included in the returned data. Export of entire Document A <!-- Starting A --> <A> <B>aaam</B> </A> C-3

134 Appendix C: Notes on XML Documents Export of entire Document B <!-- End A --> <!-- Starting B --> <A> <C>bbbm</C> </A> Example 2 Multiple XML documents are imported to Shunsaku as one file and a character string is included before the root element start tag and after the end tag: Document A Document A Start <A> <B>aaam</B> </A> Document A End Document B Document B Start <A> <C>bbbm</C> </A> Document B End When the above two documents are stored in one file and the import operation is executed, Shunsaku assumes that Document A and Document B are from <A> to </A> and ignores other characters. Therefore, when Document A and Document B are exported, Shunsaku returns only from <A> to </A> as documents. Export of entire Document A <A> <B>aaam</B> </A> Export of entire Document B <A> <C>bbbm</C> </A> C-4

135 Notes on XML Format Notes on XML Format Note the following points concerning the XML format of XML Documents stored in Shunsaku: Prefix parts, such as the XML declaration and DTD, cannot be specified as search or sort targets. Attributes, name space values, comments, and processing commands cannot be specified as search or sort targets. Character strings entered in CDATA sections can be specified as search or sort targets. Since the tags entered in CDATA sections are handled as character strings, these tags can also be specified as search or sort targets. The characters that express a CDATA section can be specified as sort targets. When an application adds or modifies an XML document, or when the Management Console or a command is used to add an XML document, Shunsaku does not check that it is a valid XML document. Use the character encoding specified in either of the following for XML documents stored in Shunsaku: The character identifier code set during standard installation The character encoding specified in the CharacterCode parameter in the system environment file C-5

136 Appendix C: Notes on XML Documents C-6

137 Appendix D XML Conversion Function APIs This chapter describes the APIs provided by the XML conversion function. List of APIs DocumentFountain Interface CSVSource Class GeneratorMap Class XMLGenerator Class XMLConverterException Class CSVException Class XMLGenException Class Error Codes Posted when the XML Conversion Function API is used D-1

138 Appendix D: XML Conversion Function APIs List of APIs The XML conversion function provides APIs written in Java. These APIs are listed in the following tables. Table D-1 Interfaces provided in APIs Package Interface Inheritance Method Function com.fujitsu.xmlc onverter DocumentFountain None getxmldocument hasxmldocument setdestination setencoding Obtains a single XML data object that is to be created. Checks whether any XML data created still exists. Sets the output destination for the XML data to be created when the output format is a stream, file or SAX. Sets the encoding of the XML data to be created. Table D-2 Classes provided in APIs Package Class Inheritance Method Function com.fujitsu.xml converter CSVSource java.lang.object CSVSource setencoding setdelimiter setquote setnullcharacter Creates an instance of CSVSource. Sets the character encoding of data used in a CSV file. Sets the delimiter for data stored in a CSV file. Sets the quotation mark character for data used in a CSV file. Sets the null character for data used in a CSV file. D-2

139 List of APIs Package Class Inheritance Method Function Generator Map XMLGener ator XMLConver terexceptio n CSVExcept ion setfirstrecordto Column setrandommode getresultset setenableescap echar Specifies whether to process the record in the first row of data in a CSV file as the column name. Specifies whether to change the method used to maintain data in the CSV ResultSet from sequential reading to memory loading. Obtains ResultSet from the data contained in a CSV file. Specifies the method for entering delimiter characters, quotation marks, and the backslash (\) as data. java.lang.object GeneratorMap Creates an instance of GeneratorMap. java.lang.object java.lang.object java.lang.throwa ble java.lang.excepti on java.lang.object java.lang.throwa ble XMLGenerator createdocument Fountain getcode getexception getmessage CSVException Creates an instance of XMLGenerator. Creates an instance of DocumentFountain. Obtains the error code. Obtains the internal exception. Obtains the error message. Creates an instance of CSVException. java.lang.excepti on com.fujitsu.xmlco nverter.xmlcon verterexception D-3

140 Appendix D: XML Conversion Function APIs Package Class Inheritance Method Function XMLGenEx ception` java.lang.object java.lang.throwa ble XMLGenExcepti on Creates an instance of XMLGenException. java.lang.excepti on com.fujitsu.xmlco nverter.xmlcon verterexception D-4

141 DocumentFountain Interface DocumentFountain Interface The DocumentFountain interface provides methods for obtaining data in XML format. DocumentFountain instances are generated by the XMLGenerator class. The fields and methods provided by the DocumentFountain interface are listed below. Fields of the DocumentFountain Interface The following table lists the fields of the DocumentFountain Interface. Table D-3 Fields of the DocumentFountain Interface Field Name public static final int TYPE_STRING public static final int TYPE_STREAM public static final int TYPE_FILE public static final int TYPE_DOM public static final int TYPE_SAX Meaning Indicates that the output format is a string. Indicates that the output format is a stream. Indicates that the output format is a file. Indicates that the output format is a DOM object. Indicates that the output format is a SAX event. Methods of the DocumentFountain Interface This section describes methods of the DocumentFountain Interface. getxmldocument Function This method gets the generated XML as a single object. Syntax public java.lang.object getxmldocument() throws XMLGenException; Return Value Returns the generated XML as a single object. Parameters None. Exceptions XMLGenException This is output if an error occurs during XML data creation. Refer to "Messages" for error code and error message details. D-5

142 Appendix D: XML Conversion Function APIs hasxmldocument Function This method determines if XML data has been generated. Syntax public boolean hasxmldocument() throws XMLGenException; Return Value Returns information on whether XML data has been generated. true if XML has been generated. false if XML has not been generated Parameters None. Exceptions XMLGenException This is output if an error occurs when ResultSet is executed. Refer to "Messages" for error code and error message details. D-6

143 DocumentFountain Interface setdestination Function Syntax This method sets the destination for the XML being generated when the output format is a stream, a file, or SAX events. public void setdestination( java.lang.object adestination /* (1) Output destination */ ) throws XMLGenException; Return Value None. Parameters (1) Output destination Specify the object that will be the output destination for the XML. Exceptions XMLGenException This is output if something other than an object is specified as the parameter. Refer to "Messages" for error code and error message details. D-7

144 Appendix D: XML Conversion Function APIs setencoding Function Syntax This method sets the character encoding for the XML being generated. public void setencoding( java.lang.string aencoding /* (1) Character encoding */ ); Return Value None. Parameters (1) Character encoding Specify the character encoding for the XML. Setting UTF-8 GB2312 big5 KSC5601 Meaning UTF-8 GB2312 (for Chinese language) big5 (for Chinese language) KSC5601 (for the Hangul alphabet (Korean)) Exceptions None. Note on setencoding If this method is not used, the default character encoding is UTF-8. D-8

145 CSVSource Class CSVSource Class The CSVSource class generates a CSV ResultSet object from data stored in a CSV file. CSVSource Function Syntax This method creates an instance of CSVSource. public CSVSource( java.lang.string afilename /* (1) Path to the CSV file */ ); Return Value None. Parameters (1) Path to the CSV file Specifies the path to the CSV file. Exceptions None. D-9

146 Appendix D: XML Conversion Function APIs setencoding Function Syntax This method sets the character encoding of data in the CSV file. public void setencoding( java.lang.string aencoding /* (1) Character encoding */ ); Return Value None. Parameters (1) Character encoding Specify the character encoding of data in the CSV file. Setting UTF-8 GB2312 big5 KSC5601 Meaning UTF-8 GB2312 (for Chinese language) big5 (for Chinese language) KSC5601 (for the Hangul alphabet (Korean)) Exceptions None. Note on setencoding If this method is not used, the character encoding specified for the server where the XML conversion function APIs are located will be used by default. D-10

147 CSVSource Class setdelimiter Function Syntax This method sets the delimiter character of data in the CSV file. public void setdelimiter( char adelimiter /* (1) Delimiter character */ ) throws CSVException; Return Value None. Parameters (1) Delimiter character Specify the delimiter character specified for data in the CSV file. Exceptions CSVException This is output if one of the following characters is set as the delimiter character: \\ The space character The same character as the quotation mark character The same character as the null character. Refer to "Messages" for error code and error message details. Note on setdelimiter If this method is not used, default delimiter character is a comma. D-11

148 Appendix D: XML Conversion Function APIs setquote Function Syntax This method sets the quotation mark character of data in the CSV file. public void setquote( char aquote /* (1) Quotation mark character */ ) throws CSVException; Return Value None. Parameters (1) Quotation mark character Specify the quotation mark character specified for data in the CSV file. Exceptions CSVException This is output if one of the following characters is set as the quotation mark character: \\ The space character The same character as the delimiter character The same character as the null character. Refer to "Messages" for error code and error message details. Note on setquote If this method is not used, no quotation mark character will be set. D-12

149 CSVSource Class setnullcharacter Function Syntax This method sets the null character of data in the CSV file. public void setnullcharacter( java.lang.string anullcharacter /* (1) Null character */ ) throws CSVException; Return Value None. Parameters (1) Null character Specify the null character specified for data in the CSV file. Exceptions CSVException This is output if one of the following characters is set as the null character: \\ The same character as the delimiter character. The same character as the quotation mark character. Refer to "Messages" for error code and error message details. Note on setnullcharacter If this method is not used, the default null character is a space character ( ). D-13

150 Appendix D: XML Conversion Function APIs setfirstrecordtocolumn Function Syntax This method specifies whether the record in the first row of data in a CSV file should be interpreted as the column name. public void setfirstrecordtocolumn( boolean aflag /* (1) Recognize as column name */ ) Return Value Parameters None. (1) Recognize as column name Specifies whether to recognize the record in the first row of data stored in a CSV file as the column name. true - to interpret as the column name false - will not be interpreted as the column name. Exceptions None. Note on setfirstrecordtocolumn If this method is not used, the record in the first row of data in a CSV file will not be interpreted as the column name. Also, a character string specified with setnullcharacter() of the CSVSource class cannot be used as a row name. D-14

151 CSVSource Class setrandommode Function Syntax This method specifies if the method used to maintain data in the CSV ResultSet should be changed from sequential reading to memory loading. public void setrandommode( boolean aflag /* (1) Load into memory */ ) Return Value Parameters None. (1) Load into memory Specifies whether to change the method used to maintain data in the CSV ResultSet from sequential reading to memory loading. true to change from sequential reading to memory loading false will not change. Exceptions None. Note on setrandommode If this method is not used, data in the CSV ResultSet will be read sequentially. D-15

152 Appendix D: XML Conversion Function APIs getresultset Function This method generates a ResultSet from the data in the CSV file. Syntax public java.sql.resultset getresultset() throws CSVException; Return Value Returns the ResultSet object that has been generated. Parameters None. Exceptions CSVException This is output if an error occurs when the ResultSet object is created. Refer to "Messages" for error code and error message details. D-16

153 CSVSource Class setenableescapechar Function Syntax Specifies the method used to enter delimiter characters, quotation mark characters, and the backslash (\) as data public void setenableescapechar( boolean aflag /* (1)Data entry method specification for delimiter characters, quotation mark characters, and the backslash (\) */ ) Return value Parameters Exceptions None. (1) Data entry method for delimiter characters, quotation mark characters, and the backslash (\) Specifies the method used to enter delimiter characters, quotation mark characters, and the backslash (\) as data. Two methods are available for entering the above characters as data. The entry method used depends on the value of the aflag parameter. as follows: False Use the following entry methods for the various characters: True Delimiter characters - enter the delimiter character enclosed in quotation marks. Quotation mark characters - enter two consecutive quotation mark characters. Backslash (\) - enter the backslash (\) by itself. Add a backslash (\) before the character. Refer to CSV Formats Supported by the XML Conversion Function in Chapter 5 for details on how to enter delimiter characters, quotation mark characters, and the backslash (\) as data. None. Note If this method is not used, operation is the same as when "true" is specified in the arguments. D-17

154 Appendix D: XML Conversion Function APIs GeneratorMap Class The GeneratorMap class expresses mapping rules. GeneratorMap Function Syntax This method creates an instance of GeneratorMap. public GeneratorMap( java.lang.string amappingrulefilename /* (1) Path to the mapping rule file */ ) throws XMLGenException; Return Value None. Parameters (1) Path to the mapping rule file Specify the path to the mapping rule file. Exceptions XMLGenException This is output if an error occurs during the I/O process. Refer to "Messages" for error code and error message details. D-18

155 XMLGenerator Class XMLGenerator Class The XMLGenerator class creates a DocumentFountain object. XMLGenerator Function This method creates an instance of XMLGenerator. Syntax public XMLGenerator(); Return Value None. Parameters None. Exceptions None. D-19

156 Appendix D: XML Conversion Function APIs createdocumentfountain Function Syntax This method creates an instance of DocumentFountain. public DocumentFountain createdocumentfountain( int anoutputtype, /* (1) Output format */ java.sql.resultset aresultset, /* (2) ResultSet object */ GeneratorMap ageneratormap /* (3) Mapping rule object */ ) throws XMLGenException; Return Value Returns the generated DocumentFountain instance. Parameters (1) Output format Specify the output format for the XML. Setting TYPE_STRING TYPE_STREAM TYPE_FILE TYPE_DOM TYPE_SAX Meaning The output format is a string. The output format is a stream. The output format is a file. The output format is DOM. The output format is SAX. (2) ResultSet object Specify the ResultSet object. (3) Mapping rule object Specify the mapping rule object. Exceptions XMLGenException This is output if an error occurs when a DocumentFountain instance is generated. Refer to "Messages" for error code and error message details. D-20

157 XMLConverterException Class XMLConverterException Class The XMLConverterException class is a parent class for the following exceptions: com.fujitsu.xmlconverter.xmlgenexception com.fujitsu.xmlconverter.csvexception. getcode Function This method obtains an error code. Syntax public java.lang.string getcode(); Return Value Returns an error code. Refer to "Messages" for error code and error message details. Parameters None. Exceptions None. D-21

158 Appendix D: XML Conversion Function APIs getexception Function This method obtains an internal exception. Syntax public java.lang.throwable getexception(); Return Value Returns an exception object. Null will be returned if no exception object exists. Parameters None. Exceptions None. D-22

159 XMLConverterException Class getmessage Function This method obtains an error message. Syntax public java.lang.string getmessage(); Overrides This method overrides the getmessage method of the java.lang.throwable class. Return Value Returns an error message string. Refer to "Messages" for error code and error message details. Parameters None. Exceptions None. D-23

160 Appendix D: XML Conversion Function APIs CSVException Class The CSVException class outputs exceptions that occur when the CSVSource class is operating or when CSV ResultSet is being generated. CSVException Function Syntax Format 1 This method creates an instance of CSVException. public CSVException( java.lang.string code /* (1) Error code */ ); Format 2 public CSVException( java.lang.string code, /* (1) Error code */ java.lang.string arg /* (2) Error message parameter */ ); Format 3 public CSVException( java.lang.string code, /* (1) Error code */ java.lang.exception e, /* (2) Exception that caused an error */ java.lang.string arg /* (3) Error message parameter */ ); Format 4 public CSVException( java.lang.string code, /* (1) Error code */ java.lang.exception e /* (2) Exception that caused an error */ ); Format 5 public CSVException( java.lang.string code, /* (1) Error code */ java.lang.string[] param /* (2) Error message parameter array */ ); D-24

161 CSVException Class Return Value None. Parameters Specify according to the following: Format 1 (1) Error code An error code is output. Format 2 (1) Error code An error code is output. (2) Error message parameter An error message parameter is output. Format 3 (1) Error code An error code is output. (2) Exception that caused an error The exception that caused an error is output. (3) Error message parameter An error message parameter is output. Format 4 (1) Error code An error code is output. (2) Exception that caused an error The exception that caused an error is output. Format 5 (1) Error code An error code is output. (2) Error message parameter array An error message parameter array is output. Exceptions None. D-25

162 Appendix D: XML Conversion Function APIs XMLGenException Class The XMLGenException class outputs exceptions that occur during processing of the following classes and interfaces: DocumentFountain interface GeneratorMap class XMLGenerator class XMLGenException Function Syntax Format 1 This method creates an instance of XMLGenException. public XMLGenException( java.lang.string code /* (1) Error code */ ) ; Format 2 public XMLGenException( java.lang.string code, /* (1) Error code */ java.lang.string arg /* (2) Error message parameter */ ) ; Format 3 public XMLGenException( java.lang.string code, /* (1) Error code */ java.lang.exception e, /* (2) Exception that caused an error */ java.lang.string arg /* (3) Error message parameter */ ) ; Format 4 public XMLGenException( java.lang.string code, /* (1) Error code */ java.lang.exception e /* (2) Exception that caused an error */ ); D-26

163 XMLGenException Class Format 5 public XMLGenException( java.lang.string code, /* (1) Error code */ java.lang.throwable err /* (2) Exception that caused an error */ ) ; Format 6 public XMLGenException( java.lang.string code, /* (1) Error code */ java.lang.string[] param /* (2) Parameter array */ ); Return Value None. Parameters Specify according to the following: Format 1 (1) Error code An error code is output. Format 2 (1) Error code An error code is output. (2) Error message parameter An error message parameter is output. Format 3 (1) Error code An error code is output. (2) Exception that caused an error The exception that caused an error is output. (3) Error message parameter An error message parameter is output. Format 4 (1) Error code An error code is output. (2) Exception that caused an error The exception that caused an error is output. D-27

164 Appendix D: XML Conversion Function APIs Format 5 (1) Error code An error code is output. (2) Exception that caused an error The exception that caused an error is output. Format 6 (1) Error code An error code is output. (2) Error message parameter array An error message parameter array is output. Exceptions None D-28

165 Error Codes Posted when the XML Conversion Function API is used Error Codes Posted when the XML Conversion Function API is used Reference This section describes the error codes and messages posted when the XML conversion function API is used. "{0}" and "{1}" in messages indicates a variable character string. Messages posted from the XML processor are sometimes displayed in variable character strings. Refer to List of error messages posted by the XML processor, for information concerning XML processor messages posted when the XML conversion function is used. Table D-4 Error codes and messages posted when the XML conversion function API is used Error Code Message Action Required XCV0003 XCV0006 XCV0007 XCV0008 XCV0009 XCV0010 XCV0011 XCV0012 XCV0013 XCV0014 XCV0015 XCV0016 The specified data type is not supported. The specified output destination object is invalid. The specified encoding is not supported. The ResultSet is null. The specified output format is not supported. The required argument was not passed when the error code was specified. The class ({0}) cannot be found. The mapping rule schema is incorrect. Detail: {0} No data in ResultSet. Detail: {0} An error occurred during ResultSet operation. Detail: {0} Failed to create XML document. Detail: {0} The class is not found. Detail: {0} Correct the data type. Check the specified output destination. Specify a valid encoding. Correct such that this message is not invoked when ResultSet is empty. Correct to an output format that is supported. Check the XML conversion function API. Check the Java execution environment. Check the URL of the specified database. Check the mapping rules and the database table contents. Remove the fundamental cause of the error as indicated in the details information. Remove the fundamental cause of the error as indicated in the details information. Check if the Java option classpath and the environment variable CLASSPATH are set correctly. D-29

166 Appendix D: XML Conversion Function APIs Error Code Message Action Required XCV0017 XCV0018 XCV0019 XCV0020 XCV0021 XCV0022 XCV0023 XCV0024 XCV0025 XCV0026 XCV0027 XCV0028 XCV0029 Failed to analyze the Mapping Rule. Detail: {0} There is no XML document to be output. More than one documentrule element is defined in the mapping rule file. The mapping rule id ({0}) is duplicated. The mapping rule root element (id: {0}) is duplicated. The value of the parent attribute in a mapping rule ({0}) is invalid. The controller-arg attribute is not defined in a mapping rule ({0}). Either both colname and colno are specified in a mapping rule ({0}), or neither is specified. Duplicate elements are been defined in a mapping rule ({0}). The value of the attrof attribute in a mapping rule ({0}) is invalid. Duplicate attributes are defined in a mapping rule ({0}). Either the attrof attribute is not specified in a mapping rule ({0}), or the parent attribute is specified. The value of the extender attribute in a mapping rule ({0}) is invalid. The mapping rules contain an error. Correct the mapping rules as indicated in the details information. Check the mapping rules and the database table contents. Define only one documentrule tag. Make the id unique. Define only one root element. Check the specification content for the parent tag. When the controller attribute is KEY, specify controller-arg. Specify either colname or colno. Delete the duplicate element. Correct the attrof attribute. Define the attribute that has multiple definitions only once. Correct the extension rule (extensionrule). If attrname is used, set a value other than STATIC for the extender attribute. D-30

167 Error Codes Posted when the XML Conversion Function API is used Error Code Message Action Required XCV0030 XCV0032 XCV0034 XCV0035 XCV0041 XCV0042 XCV0043 XCV0044 XCV0045 XCV0046 XCV0047 The extender-arg attribute is not defined in a mapping rule ({0}). The value of the controller-arg attribute in a mapping rule ({0}) is invalid. The value of the handler-arg attribute in a mapping rule ({0}) is invalid. The encoding specified in the handler-arg attribute in a mapping rule ({0}) is not supported. The value of the handler attribute in a mapping rule ({0}) is invalid. The value of the controller attribute in a mapping rule ({0}) is invalid. Either the tagname attribute or the attrname attribute must be specified. If the tagname attribute is specified, the parent attribute must also be specified. If the tagname attribute is specified, the attrof attribute must not be specified. If the attrname attribute is specified, the parent attribute must not be specified. If the attrname attribute is specified, the attrof attribute must also be specified. If STATIC is set for extender, define extender-arg. Specify the correct value for controller-arg. Specify the correct value for handler-arg. Correct the handler-arg encoding specification. Specify the correct value. Specify the correct value. Specify either the tagname attribute or the attrname attribute. Specify the parent attribute. Delete the attrof attribute. Delete the parent attribute. Specify the attrof attribute. D-31

168 Appendix D: XML Conversion Function APIs Error Code Message Action Required XCV0048 XCV0049 XCV0050 XCV0051 XCV0053 XCV0054 XCV0056 XCV0057 XCV0058 XCV0059 XCV0060 XCV0061 The tagname attribute in a mapping rule ({0}) cannot be omitted if the column name is not defined in the record on the first row of the CSV file. The attrname attribute in a mapping rule ({0}) cannot be omitted if the column name is not defined in the record on the first row of the CSV file. A empty string cannot be specified as the column name. No column name is defined in the CSV file. The specified column number ({0}) exceeds the number of columns in the CSV file. The column name in the first line does not contain the specified column name ({0}). Failed to close the CSV file. Failed to read from the CSV file. If the column name is not defined in the record on the first row of the CSV file, the colname attribute cannot be used in the mapping rules. When the first row is not used as the column name, the default mapping rules cannot be used. {0} cannot be used as a delimiter. {0} cannot be used as a quotation mark. Specify the tagname attribute, or set the column names in the first row. Specify the attrname attribute, or set the column names in the first row. Specify characters other than an empty string ("") and the character string specified with setnullcharacter() of the CSVSource class for the column name. Define the column name, or correct the settings such that column names are not in the first row. Correct the mapping rule. Correct the mapping rule. This is an error state. Restart the application. Check the CSV file path and access authority. Specify colno rather than colname. If the default mapping rules are to be used, specify the first row as being column names. Change the specified delimiter character. Change the specified quotation mark character. D-32

169 Error Codes Posted when the XML Conversion Function API is used Error Code Message Action Required XCV0062 XCV0063 XCV0064 XCV0065 XCV0066 XCV0067 XCV0068 XCV0069 XCV0070 XCV0071 XCV0072 XCV0073 XCV0074 {0} cannot be used as a null string. The number of data columns in the CSV file is not {0}. Escaped data ({0}) exists. Data is not enclosed in quotation marks ({0}). The method ({0}) is not supported. If the data itself includes quotation marks, they must be immediately preceded by "\\". The space character cannot be used as a delimiter. The space character cannot be used as a quotation mark. The space character cannot be used as a null string. The same character "{0}" cannot be specified as both a delimiter and a quotation mark. The same character "{0}" cannot be specified as both a delimiter and a null string. The same character "{0}" cannot be specified as both a quotation mark and a null string. CSV file analysis failed. Row {0}. Detail: {1} Specify a character other than the null character. Match to the specified number of columns. The data preceded by an escape sequence/character cannot be handled. Remove this data from the data to be converted. Enclose each data item in the CSV file in quotation marks, or change the quotation mark specification. The specified method cannot be used because CSVResultSet does not support it. Escape using the escape character "\". Specify a character other than a blank. Specify a character other than a blank. Specify a character other than a blank. Specify a different character for each. Specify a different character for each. Specify a different character for each. Correct the CSV file as indicated in the details message. (*1) D-33

170 Appendix D: XML Conversion Function APIs Error Code Message Action Required XCV0075 XCV0076 XCV0077 XCV0111 XCV0120 XCV0121 XCV0122 XCV0123 XCV0124 XCV0125 XCV0126 XCV0127 If the data itself includes quotation marks, they must be immediately preceded by a quotation mark. Column name ({2}) of columun number ({0}) is already defined in column number({1}). The ResultSetMetaData method ({0}) is not supported. The value of the attrname attribute in a mapping rule ({0}) is a empty string. The output destination is not specified. Failed to write to file. Detail: {0} Failed to write to stream: Detail: {0} Failed to output the XML document using SAX. Detail: {0} Failed to read from CSV file. Detail: {0} Failed to read data from the CSV file. Row {0}, column {1} Row {0} in the CSV file was left space characters. Failed to parse XML data in column that data type is "BINARY". Row {0}, column {1} If the data includes a quotation mark character, enter two consecutive quotation mark characters. Set the first row of the CSV file such that column names are not duplicated. The specified method cannot be used because it is not supported by CSVResultSet MetaData. Specify a character other than an empty string (""). Specify the output destination filename. Check that the file path exists and that it is possible to write to the file. An operating system disk or memory error may have occurred. Try restarting the operating system. An operating system disk or memory error may have occurred. Try restarting the operating system. Check that the CSV file exists and that it is possible to read from the CSV file. Check the CSV file column data that is indicated in the details information. (*1) Delete rows containing only blanks from the CSV file. Check whether or not the binary-type column data specified in the mapping rule is well-formed XML. (*1) *1 If an exception is caused by the data stored in a CSV file, the meanings of the row numbers in the posted error codes varies as described below. D-34

171 Error Codes Posted when the XML Conversion Function API is used For error code "XCV0125", if the first row is used for column names, the row number is one less than the row number in "For XCV0125" in the above figure. For error code "XCV0127", the row number is the same as in "For XCV0125". List of error messages posted by the XML processor The table below lists the error messages posted by the XML processor if the XML conversion function is used. Table D-5 error messages posted by the XML processor Message The version is required in the XML declaration. A pseudo attribute name is expected. cvc-elt.1: Cannot find the declaration of element 'XXXXX'. Explanation The version attribute is not specified in the XML declaration. Example: <?xml?> The XML declaration is incorrect. Example: <?xml > Tag 'XXXXX' is not a mapping rule tag but is in the root tags. D-35

172 Appendix D: XML Conversion Function APIs Message XML document structures must start and end within the same entity. cvc-complex-type.2.4.a: Invalid content was found starting with element 'XXXXX'. One of '{"":parentrule, "":columnrule, "":attributerule, "":documentrule, "":extensionrule}' is expected. cvc-complex-type.3.2.2: Attribute 'YYY' is not allowed to appear in element 'XXX'. cvc-complex-type.4: Attribute 'YYY' must appear on element 'XXX'. cvc-datatype-valid.1.2.1: 'XXX' is not a valid value for 'YYY' cvc-enumeration-valid: Value 'no' is not facet-valid with respect to enumeration '[yes]'. It must be a value from the enumeration. cvc-enumeration-valid: Value 'nop' is not facet-valid with respect to enumeration '[empty, omit]'. It must be a value from the enumeration. Explanation A MappingRule tag is not closed. An invalid tag is defined in the MappingRule tag elements. The invalid attribute 'YYY' is defined in the mapping rule 'XXX' tag. Mandatory attribute 'YYY' is not in mapping rule 'XXX'. 'XXX' is not a valid value for type element 'YYY'. An attribute value other than 'yes' is specified for the hide attribute of the columnrule tag. An attribute value other than 'empty' or 'omit' is specified for the null attribute of the columnrule tag. D-36

173 Appendix E XML Conversion Function Mapping Rules This chapter describes the XML conversion function mapping rules. Overview Mapping Rule File Definition Format Mapping Rule Examples Mapping Rule Conversion Examples E-1

174 Appendix E: XML Conversion Function Mapping Rules Overview Mapping rules define the structure of the XML data output to a text file by the XML conversion function. These rules define the following: The elements of XML data The attributes of XML data The correspondence between these elements/attributes and columns in the results table The hierarchical structure of the XML data generated Mapping rules are defined in mapping rule files. E-2

175 Mapping Rule File Definition Format Mapping Rule File Definition Format Mapping rules are marked up as XML, using the following tags: Table E-1 Tags used in Mapping Rule File Tag name Description Multiple use Required item? Remarks MappingRule parentrule columnrule attributerule documentrule extensionrule This tag is the highest order tag of all the mapping rule tags. Specifies rules relating to parent elements. Specifies rules relating to items. Specifies rules relating to attributes. Specifies rules when no XML declaration is defined for the data to be output in XML format. Specifies rules relating to additional information. No Required - Yes Required Hereafter, rules defined by the parentrule tag are referred to as parent rules. Yes Optional Hereafter, rules defined by the columnrule tag are referred to as column rules. Yes Optional Hereafter, rules defined by the attributerule tag are referred to as attribute rules. No Optional Hereafter, rules defined by the documentrule tag are referred to as document rules. Yes Optional Hereafter, rules defined by the extensionrule tag are referred to as extension rules. Tag Consolidation When the XML conversion function converts records from the ResultSet object to XML format, multiple elements with the same hierarchical structure can be grouped under a common parent element. This process is called consolidation. Note Do not use characters that cannot be used with XML tags or attribute names when setting the tagname and attrname attributes of the columnrule or attributerule tags. Otherwise, the XML data output will not be well-formed XML, and could not be used as an input file for Shunsaku. If characters that cannot be used in XML tag names and attribute names are used in the column names, the column names cannot be used as XML tag names and attribute names. In this case, the tagname attribute name of the columnrule tag and the attrname attribute name of the attributerule tag must be specified. E-3

176 Appendix E: XML Conversion Function Mapping Rules MappingRule Description This tag is the highest order tag out of all the mapping rule tags. Format <MappingRule>... </MappingRule> Attributes None. Elements parentrule tag Can be specified more than once. Required. columnrule tag Can be specified more than once. Optional. attributerule tag Can be specified more than once. Optional. documentrule tag Cannot be specified more than once. Optional. extensionrule tag Can be specified more than once. Optional. Note on the MappingRule Tag If unnecessary elements or attributes are defined in the Mapping Rule tag, an error will occur when the XML conversion function tries to convert the data to XML format, and an exception will be thrown. E-4

177 Mapping Rule File Definition Format parentrule Description Format This tag specifies rules relating to parent elements. <parentrule id="attribute value" tagname="attribute value" parent="attribute value" controller="attribute value" controllerarg="attribute value" /> Attributes Table E-2 Attributes Attribute name Description Required item? Id Specifies the mapping rule identifier (*1). This attribute must be specified. Tagname Parent controller Specifies the name of the XML tag converted by the parentrule tag. This attribute must be set. If spaces ("") are specified as the value for this attribute, the opening and closing parentrule tags will not be output. Specifies the mapping rule identifier of the parent rule for a parent tag that exists in the parentrule tag. This attribute need not be specified if the XML data that is output by the parentrule tag is the root element of the XML data. Otherwise it must be specified. Additionally, a parentrule tag with a mapping rule identifier specified by this attribute must be defined before defining the parentrule tag for which the parent tag is specified. To specify consolidation under the parentrule tag, set "KEY" as the value of the controller attribute. Required Required Conditional Optional Records where the column value for the element specified by the controller-arg attribute is the same as for a record that has already been processed are consolidated in a single parent element. Duplicate column elements under this parent element are eliminated, and those that remain become siblings. Consolidation is performed only if the controller attribute is specified. E-5

178 Appendix E: XML Conversion Function Mapping Rules Attribute name Description Required item? controller-arg Specifies the elements to be consolidated, in addition to the mapping rule identifier for the attribute or column elements to be consolidated. For each record, the consolidation process checks if the column name indicated by the specified mapping rule identifier is the same as any other record that has already been processed. If any such records are found, they will be consolidated. Multiple columns can be checked using commas to separate each mapping rule identifier. Optional *1 Mapping rule identifiers are used to identify mapping rules. Each mapping rule identifier must be unique within a mapping rule file. Elements None. Notes on the parentrule Tag Mapping rule files must always include one parent rule for the root element that does not specify the parent attribute. If "KEY" is specified as the value of the controller attribute, parents (parentrule) under the specified parentrule tag are grouped together under a single parent if the elements under the specified columnrule tag have the same value. When KEY is specified, it is necessary to specify a columnrule mapping rule identifier for the controller-arg attribute. If more than one columnrule mapping rule identifier is specified for the controller-arg attribute, consolidation will occur only when all specified element values are the same. If the columnrule tags under the parentrule tag that specifies the KEY attribute contain element values not specified by the controller-arg attribute, only the last of these element values will be stored during consolidation. Element values in results tables must be sorted using the consolidated column values. E-6

179 Mapping Rule File Definition Format columnrule Description Format This tag specifies rules relating to column elements. <columnrule id="attribute value" { colname="attribute value" colno="attribute value" } tagname="attribute value" parent="attribute value" hide="attribute value" null="attribute value" handler="attribute value" handler-arg="attribute value" /> Attributes Note Specify either the colname or the colno attribute. Table E-3 Attributes Attribute Name Description Required Item? Id Specifies the mapping rule identifier (*1). This attribute must be specified. colname colno tagname parent hide Specifies the name of the column in the results table that is to be extracted as the element of the columnrule tag (*2). Either the colname attribute or the colno attribute must be specified. Specifies the number of the column in the results table to be extracted as the element of the columnrule tag. Elements are extracted from the column specified in this attribute. Specify an integer starting from 1. Either the colname attribute or the colno attribute must be specified. Specifies the name of the XML tag converted by the columnrule tag. If the tagname attribute is not specified, the column names in the results table are used as tag names (*2). If spaces ("") are specified as the value of this attribute, the opening and closing columnrule tags will not be output. Specifies the mapping rule identifier for the parent rule that is the parent of the columnrule tag. This attribute must be set. Also, the parentrule tag with the mapping rule identifier specified by this attribute must be defined before defining the columnrule tag that is specified. Specifying this attribute as yes will prevent the element corresponding to the columnrule tag from being output as XML data. This attribute does not need to be set if the element corresponding to the columnrule tag is to be output as XML data. Required Conditional Conditional Conditional Required Optional E-7

180 Appendix E: XML Conversion Function Mapping Rules Attribute Name Description Required Item? null handler handler-arg Specifies the format used to output the element if the column value in the results table is null. Either of the following values can be specified: - empty: Outputs the column element as an empty element. - omit: Omits the column element. If this attribute is not specified, the default value is empty. If the data stored in the column of the results table is binary data, it will be converted to the appropriate character encoding and output as data in XML format. Specify BINARYDATA in this attribute if the column data in the results table is binary data, and specify in the handler-arg attribute the character encoding and the data type to be converted. This attribute does not need to be specified if the column data in the results table is not binary data. Specifies the character encoding for the data stored in the column of the results table and the type of data that will be converted to this encoding. Separate the data type and the character encoding by a comma. - Data type For the data type, specify either of the following as the format of the data that is to be converted to the character encoding: TEXT: Specify this value to treat the data as text when generating the output XML. Entity reference conversion processing (*3) will be performed. XML: Specify this value to treat the data as XML when generating the output XML. Entity reference conversion processing will not be performed. If this value ( XML ) is specified, the data stored in the column of the results table must be data in XML format. - Character encoding For the character encoding, specify the name of a Java character encoding. The value specified here will be passed to the various Java methods that require the name of a Java character encoding as an argument. If this attribute is not specified, the default values are TEXT for the data type and UTF-8 for the character encoding. Optional Optional Optional *1 Mapping rule identifiers are used to identify mapping rules. Each identifier must be unique within a mapping rule file. E-8

181 Mapping Rule File Definition Format *2 When converting from a CSV file, the CSVSource class setfirstrecordtocolumn() method can be used to handle the data in the first row as column names. *3 The entity references that can be used with the XML conversion function are as follows: Table E-4 XML Tags Entity References Symbol Represented < < > > & & &apos; " Elements None. Note on the columnrule Tag If the column specified does not exist in the results table, the rule specified by the columnrule tag is void. The columnrule tags cannot use colname and colno attributes in the same mapping rule file. If both types of attribute are specified, an error will occur. Example: In the following example, an error will occur because both colname and colno attributes are used in the same mapping rule file. <columnrule id="c1" colname="a" tagname="num" parent="p1" /> <columnrule id="c2" colno="2" tagname="name" parent="p1" /> The same tagname attribute value can be used for different columns in the results table. Example: In the following example, the same tagname attribute value is specified for two different columns in the results table, so no error will occur. <columnrule id="c1" colno="1" tagname="city" parent="p1" /> <columnrule id="c2" colno="2" tagname="city" parent="p1" /> If more than one columnrule tag is specified for the same column in the results table, different values must be specified for the tagname attribute of each tag. Otherwise, an error will occur. Example 1: In the following example, an error will occur because the same tagname attribute value is specified for the same column in the results table. <columnrule id="c1" colno="1" tagname="city" parent="p1" /> <columnrule id="c2" colno="1" tagname="city" parent="p1" /> E-9

182 Appendix E: XML Conversion Function Mapping Rules Example 2: In the following example, different tagname attribute values are specified for the same column in the results table, so no error will occur. <columnrule id="c1" colno="1" tagname="city" parent="p1" /> <columnrule id="c2" colno="1" tagname="sales region" parent="p1" /> attributerule Description Format This tag specifies rules for converting column values in the XML data to attributes. <attributerule id="attribute value" { colname="attribute value" colno="attribute value" } attrname="attribute value" attrof="attribute value" null="attribute value" handler="attribute value" handlerarg="attribute value" /> Attributes Note Specify either the colname attribute or the colno attribute. Table E-5 Attributes Attribute name Description Required item? Id Specifies the mapping rule identifier (*1). This attribute must be specified. colname colno attrname attrof Specifies the name of the column in the results table that is to be extracted as the element of the attributerule tag. Either the colname attribute or the colno attribute must be specified. Specifies the number of the column in the results table that is to be extracted as the element of the attributerule tag. Specify an integer starting from 1. Either the colname attribute or the colno attribute must be specified. Specifies the attribute name for the attributerule tag. If the value of this attribute is not specified, the column name in the results table will be set as the attribute value(*2). Spaces ("") cannot be specified as the value of this attribute. Specifies the mapping rule identifier for the column rule specified in the attributerule tag. This attribute must be set. Required Conditional Conditional Required Required E-10

183 Mapping Rule File Definition Format Attribute name Description Required item? null handler handler-arg Specifies the format used to output the attribute if the column value is null. Either of the following values can be specified: - empty: Outputs the column value as a space(""). - omit: Omits the column value. If this attribute is not specified, the default value is empty. If the data stored in the column of the results table is binary data, it will be converted to the appropriate character encoding and output as data in XML format. Specify BINARYDATA in this attribute if the column data in the results table is binary data, and specify the character encoding and the data type to be converted in the handler-arg attribute. This attribute does not need to be specified if the column data in the results table is not binary data. Specifies the character encoding for the data stored in the column of the results table, and the type of data that will be converted to this encoding. Separate the data type and the character encoding with a comma. - Data type To treat the data as text when generating the output XML, specify TEXT as the data type. Entity reference conversion processing (*3) will be performed. - Character encoding For character encoding, specify the name of a Java character encoding. The value specified here will be passed to the various Java methods that require the name of a Java character encoding as an argument. If this attribute is not specified, the default values are TEXT for the data type and UTF-8 for the character encoding. Optional Optional Optional *1 Mapping rule identifiers are used to identify mapping rules. Each identifier must be unique within a mapping rule file. *2 When converting from a CSV file, the CSVSource class setfirstrecordtocolumn() method can be used to handle the data in the first row as column names. *3 The entity references that can be used with the XML conversion function are as follows: E-11

184 Appendix E: XML Conversion Function Mapping Rules Table E-6 XML Tags Entity References Symbol Represented < < > > & & &apos; " Elements None. Note on the attributerule Tag Attribute values cannot be used as keys for searching data. The attributerule tags cannot use both colname and colno attributes in the same mapping rule file. If both types of attribute are specified, an error will occur. Example: In the following example, an error will occur because the attributerule tags use both colname and colno attributes in the same mapping rule file. <attributerule id="a1" colname="b" attrname="num" attrof="c1" /> <attributerule id="a2" colno="3" attrname="name" attrof="c1" /> The colname and colno attributes cannot be specified simultaneously in the same mapping rule file. If this is the case, an error will occur. Example: In the following example, an error will occur because both the colname and colno attributes are used in the same mapping rule file. <columnrule id="c1" colname="a" tagname="name" parent="p1" /> <attributerule id="a1" colno="2" attrname="num" attrof="c1" /> <attributerule id="a2" colno="3" attrname="name" attrof="c1" /> The mapping rule identifier for the column rule specified by the attrof attribute must have already been defined earlier in the mapping rule file. If more than one attributerule tag is specified for the same column rule, different attribute names must be specified for each tag. Otherwise, an error will occur. Example: In the following example, an error will occur because the same attribute name is specified for two different attributerule tags. <columnrule id="c1" colname="b" tagname="name" parent="p1" /> <attributerule id="a1" colname="a" attrname="city" attrof="c1" /> <attributerule id="a2" colname="c" attrname="city" attrof="c1" /> E-12

185 Mapping Rule File Definition Format documentrule Description Format This tag specifies rules that apply when no XML declaration is defined for the data to be output in XML format. <documentrule id="attribute value" def="no" /> Attributes Table E-7 Attributes Attribute Name Description Required Item? Id Specifies the mapping rule identifier (*1). This attribute must be specified. Required *1 Mapping rule identifiers are used to identify mapping rules. Each identifier must be unique within a mapping rule file. Elements None. Notes on the documentrule tag Only one documentrule tag can be specified in each mapping rule file. There is no need to specify the documentrule tag when an XML declaration is to be defined for the XML data to be output. extensionrule Description Format This tag specifies rules relating to additional information. <extensionrule id="attribute value" tagname="attribute value" attrname="attribute value" } { parent="attribute value" attrof="attribute value" } extender="attribute value" extenderarg="attribute value" /> Notes Specify either the tagname attribute or the attrname attribute. Specify either the parent attribute or the attrof attribute. E-13

186 Appendix E: XML Conversion Function Mapping Rules Attributes Table E-8 Attributes Attribute Name Description Required Item? id Specifies the mapping rule identifier (*1). This attribute must be specified. tagname attrname parent attrof extender To add an element, specify the tag name for the additional element. If a space ("") is specified as the value of this attribute, the opening and closing extensionrule tags will not be output. Either the tagname or the attrname attribute must be specified. To add an attribute, specify the name for the additional attribute. Spaces ("") cannot be specified as the value of this attribute. Either the tagname or the attrname attribute must be specified. When adding an element, specify the mapping rule identifier for the parent element of the additional element. This attribute must be specified if the tagname attribute has been specified. The mapping rule having the mapping rule identifier specified here must be defined before the mapping rule for the additional element. When adding an attribute, specify the mapping rule identifier for the additional attribute. This attribute must be specified if the attrname attribute has been specified. The mapping rule having the mapping rule identifier specified here must be defined before the mapping rule for the additional attribute. Specifies how to get additional attribute information. The following values can be specified for this attribute: - STATIC: Required Conditional Conditional Conditional Conditional Required Specify this value if the additional information is to be a string, in which case the actual value of the string is specified in the extender-arg attribute. - EMPTY: Specify this value if the additional information is to be an empty element. This value cannot be specified in combination with the attrname attribute. If this attribute is not specified, the default value is empty. E-14

187 Mapping Rule File Definition Format Attribute Name Description Required Item? extender-arg Specifies the string to be used as the additional attribute information. This attribute must be specified if the value of the extender attribute is STATIC. Conditional Elements *1 Mapping rule identifiers are used to identify mapping rules. Each mapping rule identifier must be unique within a mapping rule file. None. Note on extensionrule None. E-15

188 Appendix E: XML Conversion Function Mapping Rules Mapping Rule Examples The following example shows how to convert one row of the results table to a set of data in XML format. Figure E-1 Conversion of one Row of the Results Table to a Set of Data in XML Format The XML conversion function processes the results table line by line, and so the record set in the first row of the results table will be converted when the mapping rule definitions are applied. The XML conversion procedure used in the example above is explained below. (Rule 1 to Rule 4 below refer to the mapping rules from the figure above.) 1. From Rule 1, the root tag for the XML data (that will be output to a text file) will be PERSON. 2. From the tagname and parent attribute specification in Rule 2, the PERSON tag will have a child element called NUM. E-16

189 Mapping Rule Examples 3. From the colname attribute specification in Rule 2, the value of column A of the target record set will be obtained. 4. The value of column A 001 obtained in step 3 will be output as the element of the NUM tag in the XML data. 5. Rule 3 will be applied in the same way as steps 2 to Rule 4 will be applied in the same way as steps 2 to 4. Notes The id attribute for each definition tag Each definition tag must specify an id attribute that is a unique identifier within the mapping rule file. These identifiers are used with the parent attributes of other tags to specify parentchild relationships in hierarchical structures. The hierarchical structure in the following figure shows the parent-child relationships for this mapping. Figure E-2 Hierarchical Structure Showing parent-child Relationship E-17

190 Appendix E: XML Conversion Function Mapping Rules Multi-layer structures of data in XML format can be expressed by placing parent tags under other parent tags. Refer to Extracting Structured Data in XML Format for details of multi-layer structures for XML data. E-18

191 Mapping Rule Conversion Examples Mapping Rule Conversion Examples The examples below show how mapping rules are used to generate XML data from results tables. Extracting Unstructured Data in XML Format The following example shows how to extract unstructured XML data. Figure E-3 Results Table The following mapping rule extracts data from the specified column of the first row in the results table, and generates XML data as an element of the PERSON root tag. Mapping Rule <?xml version="1.0"?> <MappingRule> <parentrule id="p1" tagname="person" /> <columnrule id="c1" colname="a" tagname="num" parent="p1" /> <columnrule id="c2" colname="b" tagname="name" parent="p1" /> <columnrule id="c3" colname="c" tagname="job" parent="p1" /> </MappingRule> This mapping rule outputs the following XML data: Figure E-4 XML Output Data E-19

192 Appendix E: XML Conversion Function Mapping Rules Extracting Structured Data in XML Format The following example shows how to extract structured XML data. Figure E-5 Results Table The following mapping rule extracts data from the specified column of the first row in the results table, and generates XML data with a multi-level structure under the PERSON root tag. Mapping Rule <?xml version="1.0"?> <MappingRule> <parentrule id="p1" tagname="person" /> <columnrule id="c1" colname="a" tagname="num" parent="p1" /> <columnrule id="c2" colname="b" tagname="name" parent="p1" /> <columnrule id="c3" colname="c" tagname="job" parent="p1" /> <parentrule id="p2" tagname="detail" parent="p1" /> <columnrule id="c4" colname="e" tagname="locnum" parent="p2" /> <columnrule id="c5" colname="f" tagname="loc" parent="p2" /> </MappingRule> E-20

193 Mapping Rule Conversion Examples This mapping rule outputs the following XML data: Figure E-6 XML Output Data Extracting Fixed Values The following example shows how to extract fixed values that do not exist in the results table as XML data. Figure E-7 Results Table The following mapping rule generates XML data where fixed values that do not exist in the results table are specified as the attributes of the PERSON root tag. This example also shows how mapping rules can be used to specify XML namespaces in the root tag. E-21

194 Appendix E: XML Conversion Function Mapping Rules Mapping Rule <?xml version="1.0"?> <MappingRule> <parentrule id="p1" tagname="person" /> <extensionrule id="e1" attrname="xmlns:info" attrof="p1" extender="static" extender-arg=" <columnrule id="c1" colname="a" tagname="info:num" parent="p1" /> <columnrule id="c2" colname="b" tagname="info:name" parent="p1" /> <columnrule id="c3" colname="c" tagname="info:job" parent="p1" /> </MappingRule> This mapping rule outputs the following XML data: Figure E-8 XML Output Data Extracting Column Data as Attribute Values The following example shows how to extract XML data by using column data as attribute values. Figure E-9 Results Table The following mapping rule generates XML data in where a specific column value in the results table is used as the attribute value for a particular tag. This mapping rule shows an example of how to set an attribute named LOC for the JOB tag, and to use values in column F as the values for this attribute. E-22

195 Mapping Rule Conversion Examples Mapping Rule <?xml version="1.0"?> <MappingRule> <parentrule id="p1" tagname="person" /> <columnrule id="c1" colname="a" tagname="num" parent="p1" /> <columnrule id="c2" colname="b" tagname="name" parent="p1" /> <columnrule id="c3" colname="c" tagname="job" parent="p1" /> <attributerule id="a1" colname="f" attrname="loc" attrof="c3" /> </MappingRule> This mapping rule outputs the following XML data: Figure E-10 XML Output Data Extracting Column Data as Fixed Attributes The following example shows how to extract XML data by using column data as fixed attributes. Figure E-11 Results Table The following mapping rule generates XML data where a specific column value in the results table is used as the fixed attribute name for a particular tag. This mapping rule also shows how to make the following fixed assignments: Column A for the NUM tag element Column D for the NAME tag element Column E for the JOB tag element Column B for the name attribute of the NUM tag E-23

196 Appendix E: XML Conversion Function Mapping Rules Column C for the name attribute of the NAME tag Column F for the name attribute of the JOB tag Mapping Rule <?xml version="1.0"?> <MappingRule> <parentrule id="p1" tagname="person" /> <columnrule id="c1" colname="a" tagname="num" parent="p1" /> <attributerule id="a1" colname="b" attrname="name" attrof="c1" /> <columnrule id="c2" colname="d" tagname="name" parent="p1" /> <attributerule id="a2" colname="c" attrname="name" attrof="c2" /> <columnrule id= "c3" colname="e" tagname="job" parent="p1" /> <attributerule id="a3" colname="f" attrname="name" attrof="c3" /> </MappingRule> This mapping rule outputs the following XML data: Figure E-12 XML Output Data Extracting Column Names as Attribute Values The following example shows how to extract XML data by using column names as attribute values. Figure E-13 Results Table The following mapping rule generates XML data by using the specified column of the results table directly as the attribute value of a particular tag. E-24

197 Mapping Rule Conversion Examples This mapping rule also shows how to make the following fixed assignments: The value of column A for the NUM tag element, and the column name for the value of the name attribute of this tag The value of column D for the NAME tag element, and the column name for the value of the name attribute of this tag The value of column E for the JOB tag element, and the column name for the value of the name attribute of this tag The value of column B to the value of the value attribute of the NUM tag The value of column C to the value of the value attribute of the NAME tag The value of column F to the value of the value attribute of the JOB tag Mapping Rule <?xml version= "1.0"?> <MappingRule> <parentrule id="p1" tagname="person" /> <columnrule id="c1" colname="a" tagname="num" parent="p1" /> <extensionrule id="e1" attrname="name" attrof="c1" extender="static" extender-arg="a" /> <attributerule id="a1" colname="b" attrname="value" attrof="c1" /> <columnrule id="c2" colname="d" tagname="name" parent="p1" /> <extensionrule id="e2" attrname="name" attrof="c2" extender="static" extender-arg="d" /> <attributerule id="a2" colname="c" attrname="value" attrof="c2" /> <columnrule id="c3" colname="e" tagname="job" parent="p1" /> <extensionrule id="e3" attrname="name" attrof="c3" extender="static" extender-arg="e" /> <attributerule id="a3" colname="f" attrname="value" attrof="c3" /> </MappingRule> This mapping rule outputs the following XML data: Figure E-14 XML Output Data E-25

198 Appendix E: XML Conversion Function Mapping Rules Extracting Data in XML Format that has Multi-level Tag Structures Based on Consolidation The following example shows how to extract XML data that has multi-level tag structures based on consolidation. Figure E-15 Results Table The following mapping rule extracts data from the specified column of the first row in the results table, uses it as the element for the PERSON tag, and then repeats this process for each row in the results table. However, records with the same value for columns A, B, and C are consolidated in the resulting XML output. Mapping Rule <?xml version= "1.0"?> <MappingRule> <parentrule id="p1" tagname="person" controller="key" controllerarg="c1,c2,c3"/> <columnrule id="c1" colname="a" tagname="num" parent="p1"/> <columnrule id="c2" colname="b" tagname="name" parent="p1"/> <columnrule id="c3" colname="c" tagname="job" parent="p1"/> <parentrule id="p2" tagname="detail" parent="p1"/> <columnrule id="c4" colname="e" tagname="locnum" parent="p2"/> <columnrule id="c5" colname="f" tagname="loc" parent="p2"/> </MappingRule> E-26

199 Mapping Rule Conversion Examples This mapping rule outputs the following XML data: Figure E-16 XML Output Data and Hierarchical Structure E-27

200 Appendix E: XML Conversion Function Mapping Rules Extracting Binary Data The following example shows how to extract XML data from results tables that include binary data. Note Refer to 'Data Types Supported by the XML Conversion Function' in Chapter 5, for details on the compatibility between the data types that can be used with the XML conversion function and binary data in the results table. In the results table below, column D contains binary data. Figure E-17 Results Table The following mapping rule generates XML data that includes binary data from this results table. Mapping Rule <?xml version="1.0"?> <MappingRule> <parentrule id="p1" tagname="rep" /> <columnrule id="c1" colname="a" tagname="num" parent="p1" /> <columnrule id="c2" colname="b" tagname="date" parent="p1" /> <columnrule id="c3" colname="c" tagname="horoscope" parent="p1" /> <columnrule id="c4" colname="d" tagname="script" parent="p1" handler="binarydata" handler-arg="xml" /> </MappingRule> E-28

201 Mapping Rule Conversion Examples This mapping rule outputs the following XML data: Figure E-18 XML Output Data E-29