NuGenesis 8 SDMS Instrument Agents Installation and User s Guide

Transcription

1 NuGenesis 8 SDMS Instrument Agents Installation and User s Guide /Revision A Copyright Waters Corporation 2014 All rights reserved

2 General Information Copyright notice 2014 WATERS CORPORATION. PRINTED IN THE UNITED STATES OF AMERICA AND IN IRELAND. ALL RIGHTS RESERVED. THIS DOCUMENT OR PARTS THEREOF MAY NOT BE REPRODUCED IN ANY FORM WITHOUT THE WRITTEN PERMISSION OF THE PUBLISHER. The information in this document is subject to change without notice and should not be construed as a commitment by Waters Corporation. Waters Corporation assumes no responsibility for any errors that may appear in this document. This document is believed to be complete and accurate at the time of publication. In no event shall Waters Corporation be liable for incidental or consequential damages in connection with, or arising from, its use. For the most recent revision of this document, consult the Waters Web site (waters.com). Trademarks Waters, MassLynx, NuGenesis, NuGenesis SDMS, and THE SCIENCE OF WHAT S POSSIBLE. are registered trademarks of Waters Corporation, and NuGenesis VISION and OpenLynx are trademarks of Waters Corporation. Adobe, Acrobat, Acrobat Reader, and PostScript are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States and/or other countries. Analyst is a registered trademark of Applied Biosystems. Bruker BioSpin XWIN-NMR, Bruker Daltronics XMASS, and TopSpin are trademarks of Bruker Corporation. ChemStation is a registered trademark of Agilent Technologies. Microsoft and Windows are registered trademarks of Microsoft Corporation in the United States and/or other countries. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. UNIX is a registered trademark of The Open Group. ii June 15, 2014, Rev. A

3 XML is a registered trademark of the Massachusetts Institute of Technology, European Research Consortium for Informatics and Mathematics, or Keio University on behalf of the World Wide Web Consortium. Other registered trademarks or trademarks are the sole property of their owners. Customer comments Waters Technical Communications department invites you to tell us of any errors you encounter in this document or to suggest ideas for otherwise improving it. Please help us better understand what you expect from our documentation so that we can continuously improve its accuracy and usability. We seriously consider every customer comment we receive. You can reach us at tech_comm@waters.com. Contacting Waters Contact Waters with enhancement requests or technical questions regarding the use, transportation, removal, or disposal of any Waters product. You can reach us via the Internet, telephone, or conventional mail. Waters contact information: Contacting medium Information Internet The Waters Web site includes contact information for Waters locations worldwide. Visit Telephone and fax From the USA or Canada, phone HPLC, or fax For other locations worldwide, phone and fax numbers appear in the Waters Web site. Conventional mail Waters Corporation 34 Maple Street Milford, MA USA June 15, 2014, Rev. A iii

4 Waters NuGenesis SDMS Instrument Agents software Intended use Use the Waters NuGenesis SDMS Instrument Agents software to manage the storage and conversion of scientific data, and the automated extraction of metadata for use in the NuGenesis SDMS data repository. Safety information See the operator s guides of the instruments or devices associated with this software product for information on how to safely operate and maintain them. iv June 15, 2014, Rev. A

5 Table of Contents Copyright notice... ii Trademarks... ii Customer comments... iii Contacting Waters... iii Waters NuGenesis SDMS Instrument Agents software... iv Intended use... iv Safety information... iv 1 NuGenesis SDMS Instrument Agents software... 7 What s new in this product release... 8 Introduction... 8 Using NuGenesis SDMS Instrument Agents software and NuGenesis ELN... 9 Configuring NuGenesis SDMS projects... 9 Configuring file capture templates... 9 Configuring views Installing NuGenesis SDMS Instrument Agents software Prerequisites Installing the NuGenesis SDMS Instrument Agents software Testing the connection to the NuGenesis SDMS database Uninstalling the NuGenesis SDMS Instrument Agents software Setting up NuGenesis SDMS Instrument Agents software NuGenesis SDMS Instrument Agents software Configuring agents to run automatically Monitoring agent activity June 15, 2014, Rev. A v

6 Managing instrument agents Configuring capture methods Setting up agents to capture folders Folder capture restrictions Configuring basic folder capture agent options Configuring a Read folders from file system agent Configuring a Read folders recurs from file system agent Configuring a Read modified folders from file system agent Configuring a Read modified folders from file system, identified by CRC agent Setting up agents to capture individual files Configuring basic file capture agent options Configuring a Read files from file system agent (single file import) Configuring a Read files from file system agent (attach files to single file import) Configuring a Read files from file system agent (capture UNIFI data) Configuring Read modified files from file system identified by Archive Bit agent Configuring Read modified files from file system identified by CRC agent Data parsing Using preprocessing to modify data Example configurations AB-MDS Sciex Analyst Agilent ChemStation Bruker X-WIN NMR (plus TopSpin) TA Instruments Thermal Data Varian NMR Waters MassLynx Waters OpenLynx A Regular Expression short reference B Regular Expression Parsing short reference vi June 15, 2014, Rev. A

7 1 NuGenesis SDMS Instrument Agents software This chapter describes the NuGenesis SDMS Instrument Agents software, and explains how to configure file data projects for use with the agents. Contents: Topic Page What s new in this product release... 8 Introduction... 8 Configuring NuGenesis SDMS projects... 9 June 15, 2014, Rev. A 7

8 1 NuGenesis SDMS Instrument Agents software What s new in this product release NuGenesis 8 Data Adapters Release 1 provides these product improvements, new features, and enhancements to the NuGenesis SDMS Instrument Agents software: A new instrument agent, which collects modified files based on the set archive bit. Another new instrument agent, which collects modified files based on their CRC checksum. This replaces the former modified files agent. Introduction The Waters NuGenesis SDMS Instrument Agents software offers an alternate file capture strategy that enables you to capture file data from many sources and store those data in the NuGenesis SDMS database. The agents perform these functions: Search a configured location for folders and files that fit criteria you specify. Copy eligible files from the source location and apply processing options, including parsing for metadata in file names and file content, converting to standard format, and optional preprocessing. Store the data in a transfer folder that you configure. The NuGenesis SDMS archive agent moves the data from the transfer folder to a file data project, where you can view them as records in NuGenesis VISION. Note: The NuGenesis SDMS Instrument Agents software provides a separate file capture strategy from the NuGenesis SDMS data adapters. If you are not planning to use the features provided by the agents, you do not need to install the software. NuGenesis SDMS Instrument Agents software enables you to configure methods to capture and extract metadata from files and folders. The agents work in conjunction with the NuGenesis Technologies Corporation generic data adapter, which you set up in adapter proxy mode. An agent scans a specified source location for eligible file groups. If eligible file groups are found, the agent calls the NuGenesis SDMS Software Development Kit (SDK), which creates the required XML file that describes the collection, and copies the files to a transfer folder. The NuGenesis SDMS archive agent scans the 8 June 15, 2014, Rev. A

9 Configuring NuGenesis SDMS projects transfer folder, reads the XML file, copies the files to the Pending folder, extracts the metadata, and deletes the files from the transfer folder. Using NuGenesis SDMS Instrument Agents software and NuGenesis ELN If you are using NuGenesis SDMS with NuGenesis Electronic Laboratory Notebook (ELN), and you want to use agents to automatically capture files, you can use either NuGenesis ELN agents (that move captured data to the NuGenesis ELN inbox) or NuGenesis SDMS Instrument Agents software (that moves captured data to the NuGenesis SDMS database). You can install both agent programs, but only one can be running at any time. Note: If you installed both agents, name the exported XML configuration files so that you can easily identify which agent the configuration belongs to. The agent types are only compatible with their originating application. Important: The four agents that appear after installation are NuGenesis ELN Agents. Do not use these agents in the NuGenesis SDMS Instrument Agents software. Configuring NuGenesis SDMS projects NuGenesis SDMS Instrument Agents software operates in conjunction with NuGenesis SDMS file data projects. An agent scans a specified location for files and folders that meet capture criteria. When files are found, the agent calls the NuGenesis SDMS adapter proxy, which creates a group, extracts metadata, and prepares an XML file for use by the NuGenesis SDMS archive agent. See also: NuGenesis SDMS Data Adapters Reference Guide for information about how the adapter proxy works. Configuring file capture templates Waters recommends that you configure only a single command when using the adapter proxy, to ensure that files are captured and moved to the database as quickly as possible. June 15, 2014, Rev. A 9

10 1 NuGenesis SDMS Instrument Agents software Configure a generic template command with these settings. NuGenesis SDMS Instrument Agents software file capture template settings: Template Setting Value Vendor NuGenesis Technologies Corporation Product Generic Revision 0 Adapter proxy Selected Scan path UNC path to share specified in the Instrument Administration > Interface Configuration dialog box > SDMS Interface tab > Exchange Folder field Scan sub-folders Disabled when Adapter proxy selected Grouping Disabled when Adapter proxy selected File pattern Disabled when Adapter proxy selected Source OS Disabled when Adapter proxy selected Ignore archive bit Disabled when Adapter proxy selected Reset archive bit Disabled when Adapter proxy selected Adhere to the following rules when specifying tag markers from the project: For tags supplied by the adapter proxy, enter the tag marker exactly as shown on the tag label. For tags supplied by the template, select the desired tag marker. Configuring views When you configure an agent, you select the project in which the captured files are stored. You also select a view to restrict the tags that users can modify in the agent configuration. Create NuGenesis SDMS project views that contain only the tags that you want filled by the agent using file name or file content parsing. 10 June 15, 2014, Rev. A

11 2 Installing NuGenesis SDMS Instrument Agents software This chapter describes how to install the NuGenesis SDMS Instrument Agents software. Contents: Topic Page Prerequisites Installing the NuGenesis SDMS Instrument Agents software Uninstalling the NuGenesis SDMS Instrument Agents software June 15, 2014, Rev. A 11

12 2 Installing NuGenesis SDMS Instrument Agents software Prerequisites The NuGenesis SDMS Instrument Agents software requires an Oracle client to connect to the database, and NuGenesis SDMS file capture modules to provide the required application extensions. See also: NuGenesis SDMS Installation and Configuration Guide for information on installing these components. Each agent installation can connect to file data projects on a single NuGenesis SDMS server and run multiple agents. However, to store captured data in multiple projects, you need multiple installations of NuGenesis SDMS file capture components, because each installation can move files to only one project. To store captured data on multiple servers, your computer must have agents installed for each server. Requirement: Install NuGenesis SDMS file capture components on all computers where you install the NuGenesis SDMS Instrument Agents software, because the file capture modules installation includes programs and application extensions required by the agents. Then, operate the two complementary agents from the same computer. Waters recommends that you limit each installation to a single server and project. NuGenesis SDMS Instrument Agents software works in conjunction with the adapter proxy feature of the NuGenesis SDMS generic template, which you run with a single command line, to remove captured files from the transfer folder as quickly as possible. When you install the NuGenesis SDMS Instrument Agents software, you specify the NuGenesis ELN database where the agent configurations are stored. That is the database that the agents installation always uses, regardless of which NuGenesis SDMS server you specify when logging in to the agents program. Installing the NuGenesis SDMS Instrument Agents software You use NuGenesis SDMS Instrument Agents software to capture data and results files from the instruments. For access to preconfigured agents, contact your Waters Field Service Engineer. 12 June 15, 2014, Rev. A

13 Installing the NuGenesis SDMS Instrument Agents software The Waters Conversion Engine software, called Morpheus, is installed in the same location as the agents software. Note: The default password for the NuGenesis ELN system user, elnprod, is dor314nle. You install the agents and connect to the NuGenesis ELN database where the agent configurations will be stored. To install the NuGenesis SDMS Instrument Agents software: 1. Insert the NuGenesis 8 Data Adapters installation media. Note: Expect installation to begin automatically. If it does not do so, click Start > Run, browse to the CD drive, and click setup.exe. 2. From the installation menu, click Install SDMS 8 Data Adapters. 3. From the NuGenesis 8 SDMS Data Adapters menu, click Install SDMS Instrument Agents, and then follow the prompts. 4. In the Instrument Agents Installation Type dialog box, specify whether to install the agent software as a Windows service. Recommendation: If you plan to configure the agents and have them run continuously in the background on the computer when you are not logged in to the server, install the agent software as a Windows service. Important: When you install the instrument agents software as a Windows service, set Local Server Name to <Local Servers>. This is the default setting in the installer. 5. If you are installing the agents software as a Windows service, in the SDMS Instrument Agent Authentication dialog box, enter the following credentials: User Name: ngprojmgr Password: manager1 Alternative: You can enter a different user name and password as long as the account has the necessary permissions to start the agent. With this user type, you can also log in to the agent configuration screen. 6. In the Database Server Connection dialog box, specify the computer name for the NuGenesis ELN server you will use to maintain the agent configuration settings, the port number (typically the default value, 1521), and the database service name (SID). June 15, 2014, Rev. A 13

14 2 Installing NuGenesis SDMS Instrument Agents software Recommendation: If you are using more than one database, specify the NuGenesis SDMS database that you upgraded. In the Database User Configuration dialog box, specify the credentials for the NuGenesis ELN user elnprod, and then click Next. Result: The system attempts to connect to the specified database. After connecting successfully, click Next to validate the database structure and continue. Otherwise, click Previous, and verify that you specified the database and user information correctly. If database validation fails, click Previous, cancel the installation, and verify that you applied all required database upgrades in the correct order. If validation continues to fail, contact Waters Technical Support. 7. Follow the prompts to complete the installation. 8. Save a copy of the NG80JNI.dll file in the SDMS_Java_Toolkit_Fix folder on the installation media, and paste it to: %SYSTEMROOT%\System32 for 32-bit systems %SYSTEMROOT%\SysWOW64 for 64-bit systems Testing the connection to the NuGenesis SDMS database To test the connection to the NuGenesis SDMS database: 1. If you installed the agents as a Windows service, stop the Agents service. Rationale: You must stop the service to access the Instrument Agents configuration window. 2. Click Start > Programs, select the folder where you installed the agents, and then click Start SDMS Instrument Agent. 3. In the SDMS login window, enter an SDMS user name and password, for example: User Name: ngprojmgr Password: manager1 Result: If you successfully open the Instrument Agents software, the Instrument Agents window appears. 4. If you choose to configure the agents at this time, you must restart the Agents service upon completion. 14 June 15, 2014, Rev. A

15 Uninstalling the NuGenesis SDMS Instrument Agents software Uninstalling the NuGenesis SDMS Instrument Agents software To uninstall the NuGenesis SDMS Instrument Agents software: 1. From the Control Panel, click Add or Remove Programs, and select Waters SDMS InstrumentAgents. 2. Click Change/Remove, and follow the prompts to uninstall the program. Tip: There might be some files that the uninstaller cannot remove. These files need to be deleted manually. June 15, 2014, Rev. A 15

16 2 Installing NuGenesis SDMS Instrument Agents software 16 June 15, 2014, Rev. A

17 3 Setting up NuGenesis SDMS Instrument Agents software This chapter describes how to use the Waters NuGenesis SDMS Instrument Agents software. Contents: Topic Page NuGenesis SDMS Instrument Agents software Managing instrument agents Configuring capture methods Setting up agents to capture folders Setting up agents to capture individual files Data parsing Using preprocessing to modify data Example configurations June 15, 2014, Rev. A 17

18 3 Setting up NuGenesis SDMS Instrument Agents software NuGenesis SDMS Instrument Agents software When you start the NuGenesis SDMS Instrument Agents software, any previously started agents run in the background and scan specified locations, to capture eligible files. You can see only agents made available on the computer. To start the NuGenesis SDMS Instrument Agents software: 1. Click Start > All Programs > Waters SDMS Instrument Agents > Start SDMS Instrument Agent. Alternative: If you created a shortcut on the desktop at installation, click the icon. 2. In the SDMS Logon dialog box, specify a NuGenesis SDMS server, user name and password, and then click OK. Result: The Instrument Agents dialog box appears. The first time you start the program, the dialog box is blank, because there are no configured agents authorized on the server. If you already made agents available on this computer, they are displayed. To refresh the list of agents, click. If no agents are available, see To make an agent available: on page 22. When you close the Instrument Agents dialog box, running agents continue to run in the background. Closing this dialog box does not shut down the program. You can open the dialog box again by right-clicking the task bar icon (the eye near the computer clock), and selecting Show Agents. To shut down the NuGenesis SDMS Instrument Agents software: Right-click the agents icon in the task bar (the eye near the computer clock), click Close, and then click Yes. To start and stop agents: Perform these actions to start and stop agents. Each agent has its own set of controls. To start an agent, in the Instrument Agents dialog box, click for that agent. 18 June 15, 2014, Rev. A

19 NuGenesis SDMS Instrument Agents software To stop a running agent, click for that agent. You can also perform these actions: Click for an agent to view messages while that agent scans. Click for an agent to view properties for that agent. You can review the agent configuration tabs, but you cannot make any changes. Configuring agents to run automatically Once you have manually started an agent, you can configure it to automatically start whenever the computer is restarted. To configure agents to run automatically: 1. In the Instrument Agents dialog box, select an agent from the list of agents in the left pane, and then click Setup > Agent properties. 2. In the New Instrument Agent dialog box, select Auto run enabled, to start the agent whenever the computer is restarted. Note: Clear this setting to prevent the agent from starting after a computer restart, even if it was running when you restarted the computer. Monitoring agent activity When working with modified files and folders, you can monitor an agent to identify the files and folders waiting for capture. To monitor agent activity: 1. In the Instrument Agents dialog box, click Tools > File modified editor. 2. In the Instruments list, select an agent, and click OK. Result: The software displays the date and time that the agent uses to determine capture eligibility. June 15, 2014, Rev. A 19

20 3 Setting up NuGenesis SDMS Instrument Agents software 3. Review the date and time, and perform these actions, if necessary: To see the folders or files that are waiting for capture, click View Files. To remove the current files and folders from consideration, click Clear Files. Modify the date and time, so that the files are eligible based on the capture criteria, and then click Save. Note: If the file or folder modified date attribute is earlier than the specified date and time displayed in the editor, the agent will not recognize the files as candidates. Managing instrument agents You manage agents in the Instrument Administration dialog box by specifying how folders and files are captured, where they are to be moved, and how to extract tag information. Optionally, you can specify programs to modify data before capture (preprocessing), set up to be sent in case of errors, and determine how to convert to a standard format. For all agents, specify a scan folder where the agent searches for eligible files and folders to be captured, and a transfer folder where the agent moves eligible files and folders. The NuGenesis SDMS archive agent constantly searches the transfer folder for files, and moves those files to the NuGenesis SDMS database. To manage agents: 1. In the Instrument Agents dialog box, click Setup > Instrument Administration. 2. Identify the status of existing agents by their colored buttons, which indicate one of these states: Approved You can capture data only with approved versions of agents. An agent can have only one approved version. Modifiable You can edit the configuration. An agent can have only one modifiable version. 20 June 15, 2014, Rev. A

21 Managing instrument agents Historical You can view previously approved, but now obsolete versions. You can also create a new agent based on an historical version. Locked You can lock an approved version of an agent. No data are captured by locked agents. When you unlock a locked agent, it reverts to an approved version. Note: You can restrict the view in the tree pane by clicking one or more of the buttons in the tool bar. You can also filter the view by typing text in the filter field. As you type text, agents disappear from the view as they are filtered out. 3. Perform these additional actions, as needed, to manage the agents: To create a new agent, click Create new instrument. To modify the description of a modifiable agent version, click the agent, and then click Edit selected instrument. In the Edit Instrument Agent dialog box, modify the description, and then click OK. To delete an historical version of an agent, click the agent, and then click Delete selected instrument. If there are no historical versions associated with the agent, the icon is unavailable. You can only delete historical versions. To create a modifiable version of an agent, click the agent, and then click Create new version. You can only create a modifiable version of an approved agent or a historical version, provided no modifiable version already exists. To clear all configuration information from a modifiable agent version, click the agent, and then click Reset configuration. All agent settings are immediately cleared. To approve a modifiable version, click the agent, and then click Approve selected instrument approved agents.. You can only capture files with June 15, 2014, Rev. A 21

22 3 Setting up NuGenesis SDMS Instrument Agents software To lock an approved agent, click the agent, and then click Lock selected instrument. Lock an agent to prevent data capture in the event of an unexpected computer or network problem. For scheduled database maintenance, specify agent run times within the configuration. To unlock a locked agent, click the agent, and then click Unlock selected instrument. The agent is returned to the approved state. To test a modifiable version of an agent, click the agent, and then click Run test mode. In the test mode dialog box, start the agent. View the list of files that will be captured and tag markers that will be extracted. When the test is complete, stop the agent. Note: When you test an agent, the folders and files are deleted from the source location, as if they were captured. If you are not using save and error folders, the files and folders are lost. If you are using save and error folders, the agent moves the captured folders to the Save folder. The Error folder is never used in test mode. Making agents available on computers You can configure an agent on one computer and make it available on other computers. You make an agent available on a computer by listing that computer s name or IP address in the Location(s) of Selected Agent table in the Instrument Administration dialog box. To make an agent available: 1. In the Instrument Agents dialog box, click Setup > Instrument Administration. 2. In the Instrument Administration dialog box, select the agent. 3. In the Available on area, click, and specify either the name of the computer (Node name) on which to use the selected agent, or the IP Address of the computer. 22 June 15, 2014, Rev. A

23 Configuring capture methods 4. Click OK to save, and repeat steps for any additional computers. Note: To modify an available computer name or IP address, select a row and click Edit instrument host. Configuring capture methods Follow these instructions to specify how instrument agents capture files. Then refer to Setting up agents to capture folders on page 25 or Setting up agents to capture individual files on page 35, and follow the procedure for your specific agent type to configure the source location, capture criteria, and metadata parsing. To configure capture methods: 1. In the Instrument Agents dialog box, click Setup > Instrument Administration. 2. In the Instrument Administration dialog box, configure a new agent or modify an existing one. To configure a new agent: 1. Click Create new instrument. 2. In the Create New Instrument Agent dialog box, enter a name (ID) and description for the agent, and then click OK. 3. Click Configure selected version. Result: The agent configuration wizard appears. If you need to return to the agent configuration wizard later, click Setup. To modify an agent: 1. Select an approved or historical agent, and then click Create new version. June 15, 2014, Rev. A 23

24 3 Setting up NuGenesis SDMS Instrument Agents software 2. Select a modifiable version, and then click Configure selected version. Result: The agent configuration wizard appears. If you need to return to the agent configuration wizard later, click Setup. 3. On the Record Creation tab, click Store data in SDMS. 4. On the Result Reading tab, specify how you want the agent to scan. To have the agent capture all files in a folder, or capture files with specific extensions, click Read files from file system. On the Result Import tab, click either Import single files or Attach files to single file. The Import single files option enables you to capture either all files, or files of a single extension type. The Attach files to single file option provides extra flexibility, enabling you to capture files of a specific pattern in addition to the extension. To have the agent capture a complete folder and its subfolders, based on the folder modification date attribute, the creation of a certain file extension within the folder, or the creation of a certain number of files within the folder, click Read folders from file system. On the Result Import tab, click Import Folders. To have the agent capture a complete folder and its subfolders, based on the creation of a file-name fragment within a subfolder, click Read folders recurs from file system. On the Result Import tab, click Import Folders Recurs. To have the agent capture all files in a folder based on the time since the folder was last modified, while also leaving the folder in its original location, click Read modified folders from file system. On the Result Import tab, click Import Folders, or Import Folders Recurs. To have the agent capture all files in a folder based on selected trigger and ignore files, while also leaving the folder in its original location, click Read modified folders from file system, identified by CRC. On the Result Import tab, click Import Folders, or Import Folders Recurs. To have the agent capture files based on the file modification date attribute, while also leaving the folder in its original location, click Read modified files from file system identified by Archive Bit. On 24 June 15, 2014, Rev. A

25 Setting up agents to capture folders the Result Import tab, click Import single file or Attach files to single file. 5. On the Preprocessor tab, specify whether you want to use external programs to modify data prior to capture by the agent. Process with external programs Select to run one or two programs as preprocessors. Use RPT splitter Select to separate the RPT file into files for each sample so that you can access individual sample information. None Select for no preprocessing. See also: Using preprocessing to modify data on page On the Conversion tab, specify whether you want to convert captured file data to a standard format that you will specify later. 7. On the Parsing tab, specify the type of parsing to apply to the results (you can select more than one type). 8. On the Preferences tab, specify whether you want to use preferences that you will specify later (including data management of the scan folder, save folder, and error folder), and whether you want the agent to send notification if processing errors occur (you can select both). 9. When satisfied, click Done, and configure the details as described in Setting up agents to capture folders on page 25 or Setting up agents to capture individual files on page 35 for the type of agent you are setting up. You can set up agents to capture folders or files under different conditions. The configuration tabs change depending on the agent type. The following sections describe the generic agent types and how to set up their interface configuration tabs. Setting up agents to capture folders Follow the directions in this section to configure agents to capture folders. See also: Setting up agents to capture individual files on page 35 Note: When you set up or modify agent details, the tabs you see depend on the selections you made in the agent configuration wizard. If you do not see a tab that is required to modify the agent, you need to open the agent configuration June 15, 2014, Rev. A 25

26 3 Setting up NuGenesis SDMS Instrument Agents software wizard, click Setup, and repeat the procedure To configure capture methods: on page 23. Folder capture restrictions The folder capture function captures all subfolders of the target folder, including empty folders, ensuring that data structures are replicated exactly when restored. A folder capture agent configured to capture folders based on the folder modification date attribute ( Read folders from file system and Read modified folders from file system ), but without zipping the folder, is subject to the following restriction. The agent captures all files in the scan folder and any subfolders as a group that appears as a single record in NuGenesis VISION. However, the folder paths are not preserved. When you restore the files from NuGenesis VISION, they are all restored to <drive>:\<restore path root folder>\<scan path root folder>. Requirement: To preserve the file paths, you must configure the agent to zip the folder. Zipping the folder also allows for duplicate file names in the folder structure, because the entire folder is zipped and then moved to the transfer folder as a single file. A folder capture agent configured to capture all files based on the appearance of a file-name fragment ( Read folders recurs from file system ) is subject to these restrictions: Only the folder in which the specified file-name fragment appears is captured, along with any subfolders. Higher level folders, even though they are within the scan path, are not captured. If configured to capture without zipping the folder, the agent cannot capture files if there are duplicate file names within the folder structure. Configuring basic folder capture agent options Every folder capture agent has a set of options in common. When you configure a folder capture agent, you need to set these options as part of the agent configuration. 26 June 15, 2014, Rev. A

27 Setting up agents to capture folders To configure basic folder capture agent options: 1. In the Instrument Agents dialog box, click Setup > Instrument Administration. 2. In the left-hand pane, select the agent. 3. If you want to keep your existing agent while you configure a new one, click Create new version then select your new version. to create a new modifiable version, and 4. Click Configure selected version. 5. On the SDMS Interface tab, specify the NuGenesis SDMS project and view. 6. Click the browse button, and then browse to the UNC share where the agent will place the captured files. Note: This is also the path you must specify as the scan path in the NuGenesis SDMS file capture template command. 7. On the Result Reading tab, specify these settings: Scan Folder Folder that the agent scans. Save Folder If you select Use Save & Error Folder, the folder to which captured files are moved after they have been copied to the transfer folder you specified in step 6 above. Error Folder If you select Use Save & Error Folder, the folder to which captured files are moved, if the copy to the transfer folder you specified in step 6 above fails. Note: If you do not select Use Save & Error Folder, files are removed from the source location immediately after being copied to the transfer folder. If a problem occurs after this step, the files can be lost. To minimize the risk of losing files, use the save and error folders. You can manually delete the contents after verifying that the data have been successfully moved to the NuGenesis SDMS repository. Alternatively, you can specify data management settings on the General Preferences tab to automatically clear these folders. Polling interval Select how often (in seconds or minutes) you want to scan the folder for candidate files. The agent archives all June 15, 2014, Rev. A 27

28 3 Setting up NuGenesis SDMS Instrument Agents software available candidates, and then monitors the polling interval before scanning for new archive candidates. Rule: The polling interval specified on the General Preferences tab overrides the polling interval specified in this field. 8. On the General Preferences tab, specify these settings: Cleanup Number of days after which you want to delete files from the scan, save, and error folders. You do not need to specify a value for the Input directory (the scan folder). If you want to delete files securely after successful import to NuGenesis SDMS, select Enable Safe Deletion. If you selected Use Save & Error Folders, specify a cleanup period for those folders. Notes: Enabling this option decreases performance because the data have to be temporarily restored and checked for consistency. The Enable Safe Deletion option deletes files from their original file location, as well as the save folder. It does not apply to the Error folder. In the SDMS Administrator, you must configure the _SecureDeletionUID_ tag. This tag is used to hold a unique identifier, created by the agent, to identify files belonging to a single archive unit. During deletion, the instrument agent uses this identifier to retrieve the files from NuGenesis SDMS. Basic polling interval Specifies the period (in seconds or minutes) at which the agent polls the scan folder for eligible files. The agent archives all available candidates, and then monitors the polling interval before scanning for new archive candidates. Rule: The polling interval specified in this field overrides the polling interval specified on the Result Reading tab. Define startups and shutdowns Click this button, and in the Timer Configuration dialog box, specify times when the agent is active and inactive. Doing so ensures that the agent does not copy files at inconvenient times (for example, during times when the database is shut down for regular maintenance or backing up). To activate the timer, select the check box. Note: The other items in the General Preferences tab are not used in this release. 28 June 15, 2014, Rev. A

29 Setting up agents to capture folders 9. On the Notification tab (visible only when you select Use notification on the Preferences tab of the agent configuration wizard), specify the actions the agent takes in case of an error, including recipients, text, and whether the agent should stop when an error occurs. Note: The Instrument-Agents User is not used in this release. 10. On the Preprocessor tab, specify one or two programs that will process files before they are parsed and saved. To use a program, select Enable Preprocessor. In the Executable field, specify the full path to the program. In the Parameters field, specify the command line options to use in processing. In the Input Extension and Output Extension fields, specify the extensions of the input file and output file. Optionally, specify whether you want to save the processed data. See also: Using preprocessing to modify data on page 55 for more information about using the preprocessor feature. 11. On the Converter tab, perform these actions: Select the application you are using. Specify.zip for the extension of the file to be converted, because the conversion program (Morpheus.exe) can process only a single file. Folders must always be processed as a zip file. If you selected save every file on the Folder Import tab, the folders are nevertheless zipped for conversion. In the Target Format for Conversion field, select JCAMP or Properties. Select Properties to convert data to ASCII text files containing only the data, as key-value pairs. No curve data are included in ASCII files. Verify that the executable Morpheus.exe is selected. By default, Morpheus.exe is located in the folder where you installed the application. 12. On the Parsing tabs, specify how to extract metadata. See also: Data parsing on page 47 June 15, 2014, Rev. A 29

30 3 Setting up NuGenesis SDMS Instrument Agents software Configuring a Read folders from file system agent You set up this agent by selecting the Read folders from file system option on the Result Reading tab, and the Import Folders option on the Result Import tab of the agent configuration wizard. See also: To configure capture methods: on page 23 Set up this type of agent to capture a complete folder and its subfolders. You can trigger a capture based on the appearance of a folder in the scan path, the creation of a certain file extension, or the creation of a certain number of files. The folder is captured and then deleted from the source location. Tip: If the data that you are capturing are created as a set of files with different extension types, ensure that the file type created last is the one that you specify as the trigger file extension. This ensures that all data within the set are captured. To set up a Read folders from file system agent configuration: 1. Configure the basic options, as described in To configure basic folder capture agent options: on page On the Folder Import tab, specify these settings: When to start the capture process. You can specify the amount of time, in seconds, to wait before capturing the folder, the number of files that must appear in the folder, or you can specify a file extension together with how long you want to wait after the extension is found. Whether or not to zip the folder, and if so, how to zip (use compression, include the parent folder name in the zip file s name, and the extension, which you must specify as.zip ). If you do not zip the folder, you save all the files as a group that produces a single record in NuGenesis VISION. Note: If you capture the individual files, the folder structure is not preserved. If you need to preserve the folder structure, you must zip the folder. 30 June 15, 2014, Rev. A

31 Setting up agents to capture folders Configuring a Read folders recurs from file system agent You set up this agent by selecting the Read folders recurs from file system option on the Result Reading tab, and the Import Folders Recurs option on the Result Import tab of the agent configuration wizard. See also: To configure capture methods: on page 23 Set up this type of agent to capture a folder and its subfolders based on the appearance of a specific file-name fragment (the trigger file). You can specify anything from a complete file name to just its extension. The specified folders are captured and deleted from the source location. To set up a Read folders recurs from file system agent configuration: 1. Configure the basic options, as described in To configure basic folder capture agent options: on page On the Folder Import tab, specify these settings: The file-name fragment that starts (triggers) the capture. Specify a complete file name (for example Bond_Type.rpt), an extension, or the last portion of a file name (for example _Type.rpt). The number of folder levels above the trigger file that should be included in the capture, specified by the Start capture on folder level relative (upward) to Trigger File option. How long to wait after finding the file s name fragment before capturing the folder. Whether or not to zip the folder, and if so, how to zip (use compression, include the parent folder name in the zip file s name, and the extension, which you must specify as.zip ). If you don t zip the folder, you save all the files as a group that produces a single record in NuGenesis VISION. Configuring a Read modified folders from file system agent You set up this agent by selecting the Read modified folders from file system option on the Result Reading tab and the Import Folders or Import Folders Recurs option on the Result Import tab of the agent configuration wizard. See also: To configure capture methods: on page 23 Important: Do not use this type of agent for multi-level folders unless you select the option to save the folder as a zip file. If you select to save individual June 15, 2014, Rev. A 31

32 3 Setting up NuGenesis SDMS Instrument Agents software files, such a folder structure can only be captured once, because all the files in the folder are copied to the top level folder (which is the level below the scan folder). As a result, the folder is not captured again, even if you modify files within it. The next scan results in errors, because there are now duplicate file names in the folder structure. Set up this type of agent to capture folders based on the folder s modified date attribute, but leave it in the original location. The folder is initially captured when you start the agent, provided that the folder modification date attribute is earlier than the agent start time. When you modify the folder (by adding, or deleting a file, or modifying a file using a program that updates the folder s modification date attribute), the folder is captured again whenever it meets the time criteria you specify. Requirement: Test this agent thoroughly. Some types of file modification do not affect the folder s modified date attribute. For example, when you edit a text file in Notepad or WordPad, the folder s modified date attribute remains unchanged, and therefore, the folder is not captured again. Conversely, some products, such as Microsoft Word, create temporary, backup, or lock files: for example, in the case of Microsoft Word, a file named <~$lename>.doc is created when you open <filename>.doc. The creation of these files changes the folder s modified date attribute. When configuring time settings, make sure that you leave enough time to enable users to work with and close the files, so that folders are not captured based on these temporary backup or lock files. To set up a Read modified folders from file system agent configuration: 1. Configure the agent as described in To set up a Read folders from file system agent configuration: on page 30, or To set up a Read folders recurs from file system agent configuration: on page On the General Preferences tab, specify this additional setting: Cleanup The number of days after which you want to delete files from the Input directory (scan folder). If the folders are repeatedly modified, do not specify a period for the Input Directory. Alternatively, you can specify a period long enough to ensure that the folders and files are no longer needed. 3. On the Result Reading tab, specify these additional settings: The time criteria that a file must meet to cause the folder capture. Note: Modified in the last means that the file modified date must be at least n days/hours/minutes old. Older than means that the 32 June 15, 2014, Rev. A

33 Setting up agents to capture folders creation date must be at least n days/hours/minutes old. A folder s capture requires that both criteria are met. If necessary, the number of subfolder levels to ignore. Note: The subfolder levels are counted down from the first subfolder of the scan folder. (The scan folder is excluded.) For example, if you set this value to 3, the scan folder s fourth subfolder is the highest-level folder scanned. In the example below, setting the ignore value to 2 instructs the agent to ignore the year and month subfolder levels of the scan folder Data. Only the Samplen.raw folders are scanned. This configuration eliminates the need to create a new agent each month, because the variable folder names (year and month) are ignored. Configuring a Read modified folders from file system, identified by CRC agent You configure this agent by selecting the Read modified folders from file system, identified by CRC option on the Result Reading tab, and the Import Folders or Import Folders Recurs option, on the Result Import tab of the agent configuration wizard. See also: To configure capture methods: on page 23 The agent captures a folder whenever files of a specified type change (based on their modified date attribute). The initial capture takes place when you start the agent. Then, when a file meeting the trigger criteria changes, the folder is recaptured. The captured folder remains in its original location. June 15, 2014, Rev. A 33

34 3 Setting up NuGenesis SDMS Instrument Agents software To set up a Read modified folders from file system, identified by CRC agent configuration: 1. Configure the agent as described in To set up a Read folders from file system agent configuration: on page 30, or To set up a Read folders recurs from file system agent configuration: on page On the General Preferences tab, specify this additional setting: Cleanup The number of days after which you want to delete files from the Input directory (scan folder). If the folders are repeatedly modified, do not specify a period for the Input Directory. Alternatively, you can specify a period long enough to ensure that the folders and files are no longer needed. 3. On the Result Reading tab, specify these additional settings: Scan Folder Folder that the agent scans. Polling definition Select how often (in seconds or minutes) you want to scan the folder for updated files. If necessary, the number of subfolder levels to ignore. Note: The subfolder levels are counted down from the first subfolder of the scan folder. (The scan folder is excluded.) For example, if you set this value to 3, the scan folder s fourth subfolder is the highest-level folder scanned. In the example below, setting the ignore value to 2 instructs the agent to ignore the year and month subfolder levels of the scan folder Data. Only the Samplen.raw folders are scanned. This configuration eliminates the need to create a new agent each month, because the variable folder names (year and month) are ignored. 34 June 15, 2014, Rev. A

35 Setting up agents to capture individual files Trigger files Specify the file names and extensions to monitor for changes. When a file meeting the trigger criteria changes, the agent re-archives the entire folder. Trigger files can reside in any level of the target folder structure. Tip: To increase the speed of the capture operation, specify trigger files. When no trigger files are specified, the capture agent checks every file in the target folder, reducing performance. Ignore files Specify the file names and extensions to ignore. The agent disregards these files when determining whether the folder is a candidate for archiving. Ignore files can reside in any level of the target folder structure. Tip: This feature is especially useful when capturing MassLynx software data because MassLynx software modifies the _History.inf file whenever the data set is opened, regardless of whether the file is modified. If desired, you can ignore the _History.inf file. Tip: You do not need to specify Trigger files and Ignore files. When you specify trigger files, all other files are ignored. Important: When configuring the Folder Modified CRC agent, make sure that you set the archiving frequency higher (faster) than the deletion frequency, to avoid accidental deletion of files that have not been archived yet. When Secure Deletion is enabled, the agent only deletes files that are already archived and stored in the agent s internal secure deletion database. Setting up agents to capture individual files Follow the directions in this section to configure agents to capture individual files. See also: Setting up agents to capture folders on page 25 Configuring basic file capture agent options Every file capture agent has a set of options in common. When you configure a file capture agent, you need to set these options as part of the agent configuration. June 15, 2014, Rev. A 35

36 3 Setting up NuGenesis SDMS Instrument Agents software To configure basic file capture agent options: 1. In the Instrument Agents dialog box, click Setup > Instrument Administration. 2. In the left-hand pane, select the agent. 3. If you want to keep your existing agent while you configure a new one, click Create new version then select your new version. to create a new modifiable version, and 4. Click Configure selected version. 5. On the SDMS Interface tab, specify the NuGenesis SDMS project and view. 6. Click the browse button, and then browse to the UNC share where the agent will place the captured files. Note: This is also the path you must specify as the scan path in the NuGenesis SDMS file capture template command. 7. On the Result Reading tab, specify these settings: Scan Folder Folder that the agent scans. Save Folder If you select Use Save & Error Folder, the folder to which captured files are moved after they have been copied to the transfer folder you specified in step 6 above. Error Folder If you select Use Save & Error Folder, the folder to which captured files are moved, if the copy to the transfer folder you specified in step 6 above fails. Note: If you do not select Use Save & Error Folder, files are removed from the source location immediately after being copied to the transfer folder. If a problem occurs after this step, the files can be lost. To minimize the risk of losing files, use the save and error folders. You can manually delete the contents after verifying that the data has been successfully moved to the NuGenesis SDMS repository. Alternatively, you can specify data management settings on the General Preferences tab to automatically clear these folders. Polling interval Select the frequency, in seconds or minutes, that the software scans the folder for candidate files. The agent archives 36 June 15, 2014, Rev. A

37 Setting up agents to capture individual files all available candidates, then monitors the polling interval before scanning for new archive candidates. Rule: The polling interval specified on the General Preferences tab overrides the polling interval specified in this field. 8. On the General Preferences tab, specify these settings: Cleanup Number of days after which you want to delete files from the scan, save, and error folders. You do not need to specify a value for the Input directory (the scan folder). If you want to delete files securely after successful import to NuGenesis SDMS, select Enable Safe Deletion. If you selected Use Save & Error Folders, specify a cleanup period for those folders. Notes: Enabling this option decreases performance because the data have to be temporarily restored and checked for consistency. The Enable Safe Deletion option will delete files from their original file location, as well as the save folder. It does not apply to the Error folder. Ignore Number of Subfolders Number of subfolder levels to ignore. Note: The subfolder levels are counted down from the first subfolder of the scan folder. (The scan folder is excluded.) For example, if you set this value to 3, the scan folder s fourth subfolder is the highest-level folder scanned. In the example below, setting the ignore value to 2 instructs the agent to ignore the year and month subfolder levels of the scan folder Data. Only the Samplen.raw folders are scanned. This June 15, 2014, Rev. A 37

38 3 Setting up NuGenesis SDMS Instrument Agents software configuration eliminates the need to create a new agent each month, because the variable folder names (year and month) are ignored. Basic polling interval Specifies the period (in seconds or minutes) at which the agent polls the scan folder for eligible files. The agent archives all available candidates then monitors the polling interval before scanning for new archive candidates. Rule: The polling interval specified in this field overrides the polling interval specified on the Result Reading tab. Define startups and shutdowns Click this button, and in the Timer Configuration dialog box, specify times when the agent is active and inactive. Doing so ensures that the agent does not copy files at inconvenient times (for example, during times when the database is shut down for regular maintenance or backing up). To activate the timer, select the check box. Note: The other items in the General Preferences tab are not used in this release. 9. On the Notification tab (visible only when you select Use notification on the Preferences tab of the agent configuration wizard), specify the actions the agent takes in case of an error. These actions include ing recipients, ing text, and whether the agent should stop when an error occurs. Requirement: You must specify text; otherwise, is not sent when an error occurs. Note: The Instrument-Agents User is not used in this release. 38 June 15, 2014, Rev. A

39 Setting up agents to capture individual files 10. On the Preprocessor tab, specify one or two programs that will process files before they are parsed and saved. For each program, perform these actions: To use a program, select Enable Preprocessor. In the Executable field, specify the full path to the program. In the Parameters field, specify the command line options to use in processing. In the Input Extension and Output Extension fields, specify the extensions of the input file and output file respectively. Optionally, specify whether you want to save the processed data. See also: Using preprocessing to modify data on page On the Converter tab, perform these actions: Select the application you are using. Specify.zip for the extension of the file to be converted. Doing so is required because the conversion program (Morpheus.exe) can process only a single file. Folders must always be processed as a zip file. If you selected save every file on the Folder Import tab, the folders are nevertheless zipped for conversion. In the Target Format for Conversion field, select JCAMP or Properties. Select Properties to convert data to ASCII text files containing only the data, as key-value pairs. No curve data are included in ASCII files. Verify that the executable Morpheus.exe (by default located in c:\program Files\Waters\SDMSInstrumentAgents\Morpheus\) is selected. 12. On the Parsing tabs, specify how to extract metadata. See Data parsing on page 47. Configuring a Read files from file system agent (single file import) You set up this agent by selecting the Read files from file system option on the Result Reading tab, and the Import single files option on the Result Import tab of the agent configuration wizard (see To configure capture methods: on page 23). June 15, 2014, Rev. A 39

40 3 Setting up NuGenesis SDMS Instrument Agents software Set up this type of agent to capture all files in a folder, or a single type, based on a specified extension. You can also specify how long after the file is found the agent should wait before capturing it. Specify a time that ensures the agent does not try to capture a file while it is being written. Each file is a separate record in NuGenesis VISION. After the file is captured, it is deleted from the source location. To set up a Read files from file system agent configuration (single file import): 1. Configure the basic options, as described in To configure basic file capture agent options: on page On the Single Import tab, perform one of these actions: Capture all files in the folder by leaving the File extension field blank. Capture only files of a specified type by specifying the file extension. Optionally, select Delay, and then select a number of seconds to delay the capture after file is found. Configuring a Read files from file system agent (attach files to single file import) You set up this agent by selecting the Read files from file system option on the Result Reading tab, and the Attach files to single file option on the Result Import tab of the agent configuration wizard. See also: To configure capture methods: on page 23 Set up this type of agent to capture all files or a single file type, based on a specified extension. You can also specify a pattern for the file name. Files with the same name pattern are grouped together. You can also specify how long after the file is found the agent should wait before capturing it. Specify a time that ensures the agent does not try to capture a file while it is being written. Each file is a separate record in NuGenesis VISION. After the file is captured, it is deleted from the source location. To set up a Read files from file system agent configuration (attach files to single file import): 1. Configure the basic options, as described in To configure basic file capture agent options: on page June 15, 2014, Rev. A

41 Setting up agents to capture individual files 2. Click the Single Import Attached tab. 3. In the Define Trigger File pane, specify these settings: File extension of the files to capture. Leave the field blank to capture all files. Number of seconds to delay the capture after file is found. 4. In the Define Attached Files pane, choose one of the following: Attach all results found in location to put all files into a single group. Attach all with the same name until, and text to specify the point in the file name to scan to; for example, a period. Use regular expression, and then define a regular expression for the file pattern to search for. In the following examples, we have the set of files: abcde_1234_a1b3d abcde_1234_6_ggg.txt abcde_1234_6_a1b3d abcde_1234_6_a1b3d3.005 abcde_1234_6_243524bf.003 abcde_1234_5_3443xbc.001 abcde_1234_5_3443xbc.txt abcde_ If the file extension is txt and Attach all with the same name until is specified a period (.), the captured files will be grouped into two sets: abcde_1234_6_ggg.txt (no other files have the same name as the trigger txt file) abcde_1234_5_3443xbc.txt, abcde_1234_5_3443xbc.001 If the file extension is txt and you define a regular expression (.*?_.*?_.*?)_, the captured files will be grouped into two sets: abcde_1234_5_3443xbc.txt and abcde_1234_5_3443xbc.001 (expression evaluates to abcde_1234_5) June 15, 2014, Rev. A 41

42 3 Setting up NuGenesis SDMS Instrument Agents software abcde_1234_6_ggg.txt, abcde_1234_6_a1b3d , abcde_1234_6_a1b3d3.005, and abcde_1234_6_243524bf.003 (expression evaluates to abcde_1234_6) Configuring a Read files from file system agent (capture UNIFI data) You set up this agent by selecting the Store data to SDMS on the Record Creation tab, Read files from file system option on the Result Reading tab, Attach to Single File option on the Result Import tab, and Regular Expression Parsing on the Parsing tab of the agent configuration wizard. See also: To configure capture methods: on page 23 Set up this type of agent to capture UNIFI data files in a folder. You can also specify how long after the file is found the agent should wait before capturing it. Specify a time that ensures the agent does not try to capture a file while it is being written. Each file is a separate record in NuGenesis VISION. After the file is captured, it is deleted from the source location. To set up a Read files from file system agent configuration (capture UNIFI data): 1. Configure the basic options, as described in To configure basic folder capture agent options: on page On the Result Reading tab, specify these settings: Scan Directory Folder where the UNIFI files are stored. Save Directory If you select Use Save & Error Folder, the folder to which captured files are moved after they have been copied to the transfer folder. Error Directory If you select Use Save & Error Folder, the folder to which captured files are moved, if the copy to the transfer folder. Note: If you do not select Use Save & Error Folder, files are removed from the source location immediately after being copied to the transfer folder. If a problem occurs after this step, the files can be lost. To minimize the risk of losing files, use the save and error folders. You can manually delete the contents after verifying that the data have been successfully moved to the NuGenesis SDMS repository. Alternatively, you can specify data management settings on the General Preferences tab to automatically clear these folders. 42 June 15, 2014, Rev. A

43 Setting up agents to capture individual files Polling Definition Select 5 seconds to scan the directory for candidate folders every 5 seconds. The agent archives all available candidates, and then monitors the polling interval before scanning for new archive candidates. Rule: The polling interval specified on the General Preferences tab overrides the polling interval specified in this field. 3. On the Single Import Attached tab, specify these settings: File extension.xml Wait n seconds after file is found 4 Use Regular Expression Regular Expression (.*)uep 4. On the RegExp Parsing tab, define the RegExp to parse. Define these fields on the Define RegExp tab: Item Name File Encoding UTF-8 Text Extraction Filter No filter applied Filter result files Exclude Use RegExp <item id=.* name= [\s^\n]*(.+)\s* folder= RegExp Group 1 SDMS Target Label Item Name Folder Name File Encoding UTF-8 Text Extraction Filter No filter applied Predecessor folder= Value Type Predecessor All Successor SDMS Target Label Folder Name Item Category File Encoding UTF-8 Use RegExp Category displayname=.* attributetype=.* >[\s^\n]*(.+)\s*\<attribute> June 15, 2014, Rev. A 43

44 3 Setting up NuGenesis SDMS Instrument Agents software RegExp Group 1 Creation Date Time File Encoding UTF-8 Use RegExp ateddttm displayname=.* attributetype=.* >[\s^\n]*(.+)\s*\<attribute> RegExp Group 1 Note: The agent configuration file is located on the media under: <CD_ROOT>\SDMS_80_DA\SDMS_InstrumentAgents_DemoData\ SDMS_InstrumentAgent_DemoData\Unifi\Configuration\Unifi.xml Configuring Read modified files from file system identified by Archive Bit agent You set up this agent by selecting the Read modified files from file system identified by Archive Bit option on the Result Reading tab, and the Import single file or Attach files to single file option on the Result Import tab of the agent configuration wizard. See also: To configure capture methods: on page 23 Set up this type of agent to capture files based on their modified date attribute, but leave the files in the original location for further use. The files are captured whenever they become eligible, based on the time criteria you specify. To modify a Read modified files from file system identified by Archive Bit agent configuration: 1. Configure the agent as described in To set up a Read files from file system agent configuration (single file import): on page 40, or To set up a Read files from file system agent configuration (attach files to single file import): on page On the General Preferences tab, specify these additional settings: Select safe file deletion to enable the agent to securely delete files from the file system, after they have been successfully imported to NuGenesis SDMS. Cleanup (optional) Number of days after which you want to delete files from the Input directory (scan folder). If the folders are repeatedly modified, do not specify a period for the Input Directory. 44 June 15, 2014, Rev. A

45 Setting up agents to capture individual files Alternatively, you can specify a period long enough to ensure that the folders and files are no longer needed. 3. On the Result Reading File tab, specify these additional settings: Scan Folder Folder that the agent scans. If necessary, the number of subfolder levels to ignore. Note: The subfolder levels are counted down from the first subfolder of the scan folder. (The scan folder is excluded.) For example, if you set this value to 3, the scan folder s fourth subfolder is the highest-level folder scanned. In the example below, setting the ignore value to 2 instructs the agent to ignore the year and month subfolder levels of the scan folder Data. Only the Samplen.raw folders are scanned. This configuration eliminates the need to create a new agent each month, because the variable folder names (year and month) are ignored. The time criteria that a file must meet to cause its capture. Configuring Read modified files from file system identified by CRC agent You set up this agent by selecting the Read modified files from file system identified by CRC option on the Result Reading tab, and the Import single file or Attach files to single file option on the Result Import tab of the agent configuration wizard. See also: To configure capture methods: on page 23 The agent captures files of a specified type, whenever they change (based on their modified date attribute). The initial capture takes place when you start June 15, 2014, Rev. A 45

46 3 Setting up NuGenesis SDMS Instrument Agents software the agent. Then, when a file meeting the trigger criteria changes, the file is recaptured. The captured file remains in its original location. To modify a Read modified files from file system identified by CRC agent configuration: 1. Configure the agent as described in To set up a Read files from file system agent configuration (single file import): on page 40, or To set up a Read files from file system agent configuration (attach files to single file import): on page On the General Preferences tab, specify these additional settings: Select safe file deletion to enable the agent to securely delete files from the file system, after they have been successfully imported to NuGenesis SDMS. Cleanup (optional) Number of days after which you want to delete files from the Input directory (scan folder). If the folders are repeatedly modified, do not specify a period for the Input Directory. Alternatively, you can specify a period long enough to ensure that the folders and files are no longer needed. 3. On the Result Reading File - CRC tab, specify these additional settings: Scan Folder Folder that the agent scans. If necessary, the number of subfolder levels to ignore. Note: The subfolder levels are counted down from the first subfolder of the scan folder. (The scan folder is excluded.) For example, if you set this value to 3, the scan folder s fourth subfolder is the highest-level folder scanned. In the example below, setting the ignore value to 2 instructs the agent to ignore the year and month subfolder levels of the scan folder Data. Only the Samplen.raw folders are scanned. This configuration eliminates the need to create a new agent each month, because the variable folder names (year and month) are ignored. 46 June 15, 2014, Rev. A

47 Data parsing Data parsing The time criteria that a file must meet to cause its capture. NuGenesis SDMS Instrument Agents software provides several methods for parsing data files to extract metadata into SDMS fields: The File Parsing tab extracts metadata from file content. This parsing option is the easiest to use and builds the regular expression for text extraction for you. See also: To configure file parsing: on page 47 The Name Parsing tab extracts metadata from the file name. See also: To configure name parsing: on page 49 The RegExp Parsing tab extracts metadata from file content. This file parsing option is the most powerful, because an experienced user can enter their own regular expression. When you use this tab, the whole file is parsed, and the value extracted is the last matching occurrence in the file. See also: To configure regular expression parsing: on page 51 The Line Parsing tab extracts metadata from file content. This parsing option enables you to specify all or part of the file for parsing. See also: To test line parsing: on page 54 To configure file parsing: 1. On the File Parsing tab, specify any file extensions that should be included or excluded from parsing. June 15, 2014, Rev. A 47

48 3 Setting up NuGenesis SDMS Instrument Agents software Note: Typically, there are many file types, but you must set up tags based on a particular file type. Include the file extension that defines the type to be parsed for metadata. 2. Click Open, and in the Configuration box, select one of these options: To create a new configuration, click New. To edit the existing configuration, click Edit. To import a configuration from an XML file, click Import. Note: When you create a new configuration, the existing one is lost. If you want to save a current configuration, export it to an XML file. Adopt a naming or storage strategy (such as separate folders) so that you can distinguish between agent configuration XML files, file parsing configuration XML files, and name parsing configuration XML files. 3. In the Properties dialog box, specify a template name and the file encoding that applies to the file you want parsed. Restriction: Do not use spaces in template names. Note: The Source Program and Description fields are for information only. 4. Perform one of these actions: To add a tag value, click New. To modify an existing tag value, select the row in the Regular Expressions grid, and then click Edit. In the Regular Expression Definition dialog box, in the Predecessor Definition area, perform these actions: a. In the Predecessor field, specify unique text that precedes the text you want to extract for the tag value. Note: If the expression includes a line feed, select NewLine at the end of the field. b. In the Separator field, specify text that separates the predecessor text from the tag value text. Note: If the separator is a line feed, at the end of the field, select NewLine. 5. In the Successor Definition area, perform one of these actions: Select Successor, and in the successor field specify text that immediately follows the tag value text. 48 June 15, 2014, Rev. A

49 Data parsing Note: If a line feed immediately follows the text, at the end of the field, select NewLine. Select Value Length, and specify the number of characters you want to extract for the tag value. 6. In the label definition area, click the Keyword list, and select the tag marker for this tag value. Rule: Only tags specified in your view (selected on the SDMS Interface tab) are visible in the list. 7. Repeat this procedure for any additional tags, and then click OK. Note: The Expert mode can be used by Waters technical support personnel to customize tag extraction. To configure name parsing: 1. In the Name Parsing dialog box, click Open. 2. In the Configuration box, click one of these options: To create a new configuration, click New. To edit the existing configuration, click Edit. To import a configuration from an XML file, click Import. Note: When you create a new configuration, the existing one is lost. If you want to save a current configuration, export it to an XML file. Adopt a naming or storage strategy (such as separate folders) so that you can distinguish between agent configuration XML files, file parsing configuration XML files, and name parsing configuration XML files. 3. In the Properties dialog box, specify a template name. Restriction: Do not use spaces in template names. Note: The Source Program and Description fields are for information only. 4. Perform one of these actions: To add a tag value, click New. To modify an existing tag value, select the row in the Regular Expressions grid, and click Edit. 5. In the Predecessor field, perform one of these actions to specify where the tag text starts: June 15, 2014, Rev. A 49

50 3 Setting up NuGenesis SDMS Instrument Agents software Note: Read these examples carefully to understand the behavior that you can expect from the name parsing configurations. To start at the beginning of the file name, leave the Predecessor field blank. To skip to a variable section of the file name, click Add count value. In the Type area, specify what to count (characters, spaces, or tabs), and in the Count Values area, select X times. Example: If you select Characters and X times, and the file name is abc 123 defg hijkl.txt, all the characters up to the first non-alpha-numeric character (in this example the space between c and 1 ) are skipped. The parsing process starts at 1 and extends to the item you specify in the Value Type field as the end of the tag text (that is, a fixed number of items or a successor item). To skip a fixed number of file name items, click Add count value. In the Type area, specify what to count (characters, spaces, or tabs), and in the Count Values area, select time(s) and specify a value. The item you select must exist. Otherwise, no parsing occurs. Example: If you select Characters and 4 times, and the file name is abc 123 defg hijkl.txt, the extracted tag text starts at h and extends to the item you specify in the Value Type field as the end of the tag text (that is, a fixed number of items a successor item). The first character of the tag text is h because defg is the first string of four characters (without a space) that can be skipped. If you select Spaces and 2 times, no tag text is extracted because there are no instances within the file name of two contiguous spaces. Note: You can concatenate count values. If you know the file name always has the same format, and using the example name abc 123 defg hijkl.txt, you want to start extracting tag text at the position of the number 3, select Characters and 3. Then click Add count value again and select Spaces and 1. Finally, click Add count value again and select Characters and In the Value Type area, perform one of these actions to specify the tag text: Specify a fixed value Select Value Length, specify the value, and then select the type of character items to count from the list. Alternative: Click the Count Value button, and specify details, observing the instructions and examples in the previous step. 50 June 15, 2014, Rev. A

51 Data parsing Specify a successor item Select Successor and type the item into the left Value Type field. Example: If you specify. as the successor, the tag text terminates at the first period (which typically separates the name from the extension). Extract text from the starting point specified by the predecessor, to the end of the file name Select Successor, and then select End. Note: The Expert mode can be used by Waters technical support personnel to customize tag extraction. 7. In the label definition area, click the Keyword list, and select the NuGenesis SDMS project tag label. Note: Only tags specified in your view (selected on the SDMS Interface tab) appear in the list. To configure regular expression parsing: 1. On the RegExp Parsing tab, click the Define RegExp tab. 2. Click Add. 3. In the Predecessor Definition area, perform these actions: a. In the Predecessor field, specify unique text that precedes the text you want to extract for the tag value. Note: If the expression includes a line feed, select NewLine at the end of the field. b. In the Separator field, specify text that separates the predecessor text from the tag value text. Note: If the separator is a line feed, select NewLine at the end of the field. 4. In the Value Type Definition area, choose the value type for this field - one of All, Alphabetical, Alphanumerical, or Numerical. 5. In the Successor Definition area, perform one of these actions: Select Successor, and in the successor field specify text that immediately follows the tag value text. Note: If a line feed immediately follows the text, select NewLine at the end of the field. June 15, 2014, Rev. A 51

52 3 Setting up NuGenesis SDMS Instrument Agents software Select Value Length, and specify the number of characters you want to extract for the tag value. 6. In the Define Target Label area, click the list, and select the NuGenesis SDMS project tag label that will accept the parsed value. Rule: Only tags specified in your view (selected on the SDMS Interface tab) are visible in the list. 7. In the File Encoding area, choose the encoding type of the files that you are parsing. 8. If you want to apply a filter to cater for PDF or PostScript files, in the Text Extraction Filter area, choose the filter for the file type you are processing. 9. If you want to filter the result files that you are processing, in the Filter result files area, perform the following actions: Select Include or Exclude to include or exclude the listed file types from processing. For each file type you want to add to the list, click Add, and enter the file extension in the Files list. Result: The Target fields list is updated to show the target that you have just defined. 10. Repeat steps 2 through 9 for each NuGenesis SDMS project tag label for which you want to define line parsing, and then click Approve. Result: The Target fields list is updated to show the target that you have just defined. Note: The Expert mode can be used by Waters technical support personnel to customize tag extraction. To configure regular expression parsing for Japanese characters: 1. Perform steps 1 through 9 above. 2. In the Test RegExp tab, replace the string with.* and click Add Regular Expression to Configuration. 3. The string is added to the Expert Mode panel in the Define RegExp tab. To configure line parsing: 1. On the Line Parsing tab, click the Parsing Definition tab. 52 June 15, 2014, Rev. A

53 Data parsing 2. Click Add. 3. In the Value Start Definition area, perform one of these actions: Select Key, and in the Key value field specify unique text that identifies the start of the text you want to extract for the tag value. Select Position, and in the Position number of the first character field specify the position on the line of the character from which you want to start extracting the tag value. 4. In the Value End Definition area, perform one of these actions: Select Tag, and in the End tag field specify unique text that identifies the end of the text you want to extract for the tag value. Check the appropriate box to include CR and LF, LF, or just the text that you entered. Select Length, and in the Length of the value field specify the number of characters to be extracted. 5. If you want to restrict parsing to a section of the file, in the Define Range for Parsing area, check Enable Range Parsing, and enter the start and end line numbers. 6. If you want to parse a single line, in the Define specific line number area, perform the following actions: a. Check Only parse line number, and enter the required line number. b. If you want to parse by delimiter, overriding the other parsing definitions, check Parse by delimiters, and enter the number of delimiters before first extraction, and the delimiter character. 7. If you want to specify a default value for the SDMS target label, in the Define default value area, perform the following actions: a. Check Default value, and enter the value to be used if the parsing fails to find a match. b. If you want the specified default value to be used as the actual value for the target field, check Always Apply Default Value. Note: Choosing this option means that the lines are not parsed for this target. 8. In the Define Target Label area, click the list, and select the NuGenesis SDMS project tag label that will accept the parsed value. June 15, 2014, Rev. A 53

54 3 Setting up NuGenesis SDMS Instrument Agents software Rule: Only tags specified in your view (selected on the SDMS Interface tab) are visible in the list. 9. In the File Encoding area, choose the encoding type of the files that you are parsing. 10. If you want to filter the result files that you are processing, in the Filter result files area, perform the following actions: Select Include or Exclude to include or exclude the listed file types from processing. For each file type you want to add to the list, click Add, and enter the file extension in the Files list. 11. Repeat steps 2 through 10 for each NuGenesis SDMS project tag label for which you want to define line parsing, and then click Approve. Result: The Target fields list is updated to show the target that you have just defined. To test file parsing or name parsing: 1. Place a suitable file with known values for the tag text in the My Documents folder, or another convenient location. 2. In the Parsing Configuration dialog box, click New (file parsing) or Edit (name parsing), and then click Test. 3. For name parsing, click Browse to open the Windows Open dialog box. 4. In the Windows Open dialog box, double-click the file. Alternative: If you put the file somewhere other than in the My Documents folder, browse to the alternate location, and double-click the file. 5. In the Result dialog box, verify that the tag values are what you expect. To test line parsing: 1. Place a suitable file with known values for the tag text in the My Documents folder, or another convenient location. 2. On the Line Parsing tab, click the Test Complete Configuration tab. 3. Load the test file into the Parsing input area. 4. Click Start parsing. 54 June 15, 2014, Rev. A

55 Using preprocessing to modify data Result: The parsing result appears in the Parsing output area. 5. If the result is not what you expected, modify your parsing definition and test again. To test RegExp parsing: 1. Place a suitable file with known values for the tag text in the My Documents folder, or another convenient location. 2. On the RegExp Parsing tab, click the Test Complete Configuration tab. 3. Load the test file into the Parsing input area. 4. Click Start parsing. Result: The parsing result appears in the Parsing output area. 5. If the result is not what you expected, modify your parsing definition and test again. Tip: You can test the regular expression on the Test RegExp tab. Using preprocessing to modify data You use the Preprocessor feature to modify data before saving to the NuGenesis SDMS database. You can modify data to simplify the process of extracting tag information. This example shows how you can use GhostScript, a program that can create.pdf files from Adobe PostScript (.ps) files. The preprocessor in this example reads a single.ps file from the scan directory and sends it to the preprocessor, GhostScript, which creates a.pdf file. Then you specify the.pdf file in the parsing tab as the file to be parsed for tag metadata. The freshly created PDF is parsed and both files are then archived in the NuGenesis SDMS database using the metadata extracted from the PDF. June 15, 2014, Rev. A 55

56 3 Setting up NuGenesis SDMS Instrument Agents software Requirement: You must know how to use the preprocessor program s command line syntax to specify settings as well as the input and output files. 56 June 15, 2014, Rev. A