TIBCO Foresight Transaction Insight

Transcription

1 TIBCO Foresight Transaction Insight Using TI Purge Software Release February 2012 TIBCO provides the two-second advantage

2 Important Information SOME TIBCO SOFTWARE EMBEDS OR BUNDLES OTHER TIBCO SOFTWARE. USE OF SUCH EMBEDDED OR BUNDLED TIBCO SOFTWARE IS SOLELY TO ENABLE THE FUNCTIONALITY (OR PROVIDE LIMITED ADD-ON FUNCTIONALITY) OF THE LICENSED TIBCO SOFTWARE. THE EMBEDDED OR BUNDLED SOFTWARE IS NOT LICENSED TO BE USED OR ACCESSED BY ANY OTHER TIBCO SOFTWARE OR FOR ANY OTHER PURPOSE. USE OF TIBCO SOFTWARE AND THIS DOCUMENT IS SUBJECT TO THE TERMS AND CONDITIONS OF A LICENSE AGREEMENT FOUND IN EITHER A SEPARATELY EXECUTED SOFTWARE LICENSE AGREEMENT, OR, IF THERE IS NO SUCH SEPARATE AGREEMENT, THE CLICKWRAP END USER LICENSE AGREEMENT WHICH IS DISPLAYED DURING DOWNLOAD OR INSTALLATION OF THE SOFTWARE (AND WHICH IS DUPLICATED IN LICENSE.PDF) OR IF THERE IS NO SUCH SOFTWARE LICENSE AGREEMENT OR CLICKWRAP END USER LICENSE AGREEMENT, THE LICENSE(S) LOCATED IN THE LICENSE FILE(S) OF THE SOFTWARE. USE OF THIS DOCUMENT IS SUBJECT TO THOSE TERMS AND CONDITIONS, AND YOUR USE HEREOF SHALL CONSTITUTE ACCEPTANCE OF AND AN AGREEMENT TO BE BOUND BY THE SAME. This document contains confidential information that is subject to U.S. and international copyright laws and treaties. No part of this document may be reproduced in any form without the written authorization of TIBCO Software Inc. TIB, TIBCO, TIBCO Adapter, Predictive Business, Information Bus, The Power of Now, TIBCO ActiveMatrix, and TIBCO Foresight Transaction Insight are either registered trademarks or trademarks of TIBCO Software Inc. in the United States and/or other countries. EJB, Java EE, J2EE, and all Java-based trademarks and logos are trademarks or registered trademarks of Sun Microsystems, Inc. in the U.S. and other countries. All other product and company names and marks mentioned in this document are the property of their respective owners and are mentioned for identification purposes only. THIS SOFTWARE MAY BE AVAILABLE ON MULTIPLE OPERATING SYSTEMS. HOWEVER, NOT ALL OPERATING SYSTEM PLATFORMS FOR A SPECIFIC SOFTWARE VERSION ARE RELEASED AT THE SAME TIME. SEE THE README.TXT FILE FOR THE AVAILABILITY OF THIS SOFTWARE VERSION ON A SPECIFIC OPERATING SYSTEM PLATFORM. THIS DOCUMENT IS PROVIDED AS IS WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NON-INFRINGEMENT. THIS DOCUMENT COULD INCLUDE TECHNICAL INACCURACIES OR TYPOGRAPHICAL ERRORS. CHANGES ARE PERIODICALLY ADDED TO THE INFORMATION HEREIN; THESE CHANGES WILL BE INCORPORATED IN NEW EDITIONS OF THIS DOCUMENT. TIBCO SOFTWARE INC. MAY MAKE IMPROVEMENTS AND/OR CHANGES IN THE PRODUCT(S) AND/OR THE PROGRAM(S) DESCRIBED IN THIS DOCUMENT AT ANY TIME. THE CONTENTS OF THIS DOCUMENT MAY BE MODIFIED AND/OR QUALIFIED, DIRECTLY OR INDIRECTLY, BY OTHER DOCUMENTATION WHICH ACCOMPANIES THIS SOFTWARE, INCLUDING BUT NOT LIMITED TO ANY RELEASE NOTES AND "READ ME" FILES. Please see Licensing_Foresight_Products.pdf for licensing details. Copyright TIBCO Software Inc. ALL RIGHTS RESERVED. TIBCO Software Inc. Confidential Information

3 Contents USE WITH CARE!... 1 TI Purge Capabilities... 1 System Requirements... 1 Audience... 1 Comparison of TI Purge and Data Expiration... 2 Installing... 2 Checking your Database... 3 Setting up Logging... 4 Scheduling and Strategy... 6 Executing TI Purge... 6 Stopping a purge... 9 After a purge Program success or failure Using TI Purge Page i 2/2/2012

4 Using TI Purge Page ii 2/2/2012

5 USE WITH CARE! TI Purge deletes data from your TIBCO Foresight Transaction Insight database forever. Once purged, it cannot be restored. Do not confuse this with an archiving program. Therefore: Under no circumstances should you purge without backing up the database ahead of time. If you do so, TIBCO Foresight has no way to help you get any deleted data back. Send your proposed command line to TIBCO Foresight for feedback before executing it. For your first purge, use a test database and purge only one or two days worth of data. TI Purge Capabilities TI Purge is a Java program that lets you: Permanently delete Transaction Insight transaction data that has submission dates older than a specified number of days. This will include all transaction types; you cannot selectively purge one transaction type. Choose whether the database keeps monthly statistics for purged data. Administrative data (Tasks, Partners, etc) is not purged. Purging frees up space in your database but does not significantly improve performance. System Requirements Please see System_requirements.pdf. Audience This guide is for business analysts and database administrators who work with the Transaction Insight database. Using TI Purge Page 1 2/2/2012

6 Comparison of TI Purge and Data Expiration Data expiration TI Purge A TIUtilities function that should be set up in advance before you start importing into TI. Typically, each day data is deleted according to the selections under Settings Data Expirations. It is not suitable for lopping off large amounts of data at once. A Java tool that runs from a command line to remove data earlier than a specified date. Installing To install: 1. Check System_Requirements.pdf and be sure that you have the correct version of Java installed. 2. Unzip files into a separate directory in the location of your choice. This can be on any machine that can access the database. Example: Files LoggingConfig.txt RunTIPurge.cmd TIPurge.jar Logging configuration file Customize as described on page 4 Command line file Customize as described on page 6 TI Purge program Using TI Purge Page 2 2/2/2012

7 Checking your Database Before purging for the first time, be sure that your TI database is at version or later. To verify your database version, execute this query: select name, value from SystemSetting where name = 'Version'; This will return the database version. Example of value returned Meaning TI 3.3 patch TI 4.0 base If the version is not at least , please contact TIBCO Foresight technical support. Using TI Purge Page 3 2/2/2012

8 Setting up Logging The only TI Purge configuration file is the one for logging. 1. Determine where you want the log files to be written and create the directory. 2. Back up LoggingConfig.txt. 3. Customize it as explained below. Configuration file format LoggingConfig.txt is a typical log4j configuration file. An Internet search will give you plenty of information about this format. The file you are editing will resemble the one below. Notice the # that comments out lines. The annotation numbers are explained on page 5. Items that are frequently customized are shown in bold below. A B C D E F log4j.rootlogger=debug,filelog, console # log4j.rootlogger=[error WARN INFO DEBUG], appendername, appendername,... #a single logfile #log4j.appender.filelog=org.apache.log4j.fileappender #size based rolling log file #log4j.appender.filelog.maxbackupindex= 5 #log4j.appender.filelog.maxfilesize=2mb #log4j.appender.filelog=org.apache.log4j.rollingfileappender #log4j.appender.filelog.file=c:/foresight/tipurge/logs/etllog.txt #Date based rolling log file log4j.appender.filelog=org.apache.log4j.dailyrollingfileappender log4j.appender.filelog.file=c:/foresight/tipurge/logs/sql- TI310P3Purge.txt log4j.appender.filelog.datepattern='.'yyyy-mm-dd log4j.appender.filelog.layout=org.apache.log4j.patternlayout log4j.appender.filelog.layout.conversionpattern=[%d{mmm dd HH:mm:ss}] %-5p (%F:%L)[%t] - %m%n log4j.appender.console=org.apache.log4j.consoleappender log4j.appender.console.layout=org.apache.log4j.patternlayout log4j.appender.console.layout.conversionpattern=[%d{hh:mm:ss}] %-5p (%F:%L)[%t] - %m%n Using TI Purge Page 4 2/2/2012

9 Explanation of log file Your job is to pick the amount of logging (A above) and set the logging method (B, C, or D above). All three methods are shown in the file, with the first two commented out. A Set the amount of logging here. This is initially set to DEBUG but you can use ERROR, WARN, or INFO as shown in the commented out line below. Specify a path and filename for the log. The directory must already exist. B C This method, which is commented out in the example, would create one single log file that continues growing. This method, which is also commented out in the example, would create a log file that is has a maximum size. Once MaxFileSize is reached, its filename is appended with an underscore and number and a new log file is started. Specify how many old logs to keep. MaxBackupIndex=5 means only keep the 5 newest backups. Specify a path and filename for the log. The directory must already exist. D This method creates a new file on a schedule that you specify. This method is NOT commented out in the default LoggingConfig.txt. yyyy-mm-dd creates a new file at midnight each day, renaming the old with an appended date. yyyy-ww rolls over at midnight each Saturday. yyyy-mm-dd-hh rolls over at the top of each hour. There are several other formats in the log4j specifications. Specify the path and filename for your log file. E This specifies the format and contents of your log file. The one provided puts out the event s date, its property, filename, line number, thread, and message followed by a newline. F This specifies the format and contents of your console display. The one provided is puts out the same information as E except it omits the date. Using TI Purge Page 5 2/2/2012

10 Scheduling and Strategy Do not schedule a purge when TIUtilities might run. This could cause a deadlock in your database. Consider your organization s legal requirement for how long data must be kept. Consult your business analysts about how long they need access to transaction data and statistics. Be sure to notify all users well ahead of time about the dates that will be purged. You do not need to update TI statistics or optimize your database before purging. Purging is resource intensive and will slow database access, so consider scheduling it for times when TI is not busy. Users may continue to access the system, although they could experience timeouts. You will probably have a big initial purge. Consider breaking this into smaller time chunks and scheduling it over a weekend until you see how long it takes. During a big initial purge, you will need additional disk space for log files created by TI Purge, SQL Server, and Oracle. You can roughly multiply the number of days to estimate how long a purge will take. If purging 1 day s data takes 1 hour, then 7 days of data will take about 7 hours, etc. After the initial purge, consult with your DBA about optimizing the TI database. Thereafter, schedule frequent small purges. Since there is a sliding time window for purging, you can use the same command line each time. Executing TI Purge Things to know: TI Purge currently runs on all TI database platforms. Purging starts with the largest transmissions, distributing them among all threads. If a purge dies before it completes, just restart it. The transmission table record will remain as a reference when using most purge options. Purging should not affect Importer except for the demand on resources. TI Purge uses flags in the transaction table to mark items scheduled for purging, currently being purged, or having been purged. Using TI Purge Page 6 2/2/2012

11 Customize and execute the command file Please note: Send TIBCO Foresight Support your intended command line before you execute for the first time. You can run TI Purge by executing the command file that comes with it: Edit the file first, following this format: set path=path\jre1.6.0_02\bin;%path% java -jar path\tipurge.jar" -beyond n -purgemonthly n -maxthreads n -dbtype type -db "connectionstring" -logcfg "logconfig" Use double-quotes around values containing spaces or special characters. Parameter Explanation Required -dbtype type -db "connectionstring" -logcfg "logconfig" Optional -purgemonthly n -maxthreads n type is one of these: Oracle SqlServer connectionstring is the database connection string. logconfig is the path and filename of the logging configuration file (see page 4). n is one of these: 1 purge monthly statistics 0 do not purge monthly statistics When set to 1, purges monthly statistics of data older than the beyond date. The daily and weekly statistics for data older than the beyond date are always purged regardless of this parameter setting. Default is 0. Number of threads to use for purging. Suggestion: start at 5. If purging takes too long, keep incrementing the number by 5 until you get satisfactory results. Default is 1. Using TI Purge Page 7 2/2/2012

12 -monitor n -ndne -plan -purgebad -purgegood -purgeincomplete -scrub Report the progress of the purge to the log every n minutes. Value can be 0 to n, where 0 means do not report progress. Default is 30. If you have an implementation that has a do not erase option for transmissions, this overrides it so those documents can be purged if selected. Default is to leave data marked as do not erase. Logs how many rows would be purged from each table. Does not actually perform the purge. Purge only selected bad or good data. A good transmission is one in which there are no errors in any of its documents. All others are bad transmissions. Default is to purge all selected data. Remove all partial imports that are more than three days old. Default is to purge partial imports that are selected. Delete leftover purge-created information in the database. WARNING: Do not use -scrub while any other purge processes are running. Default is to ignore this information. Purge deletes this if it exits normally. -wipetrans Selection Criteria you must use one of these Deletes the purged rows from the Transmissions table for all selected data. -beyond n -start MM-DD-CCYY -end MM-DD-CCYY -purgeid ID -purgefilename filename Purge data with submission date older than n days. Use one or both to specify the start date and end date of the purge range in MM-DD-CCYY format. Documents with submission dates in this range will be purged. If you use only end, all documents with a submission date earlier than that day will be purged. If you use only start, all documents with a submission date later than that day will be purged. Purges data with a specific Transmission ID. This is an internal TI database ID that does not display on the TI portal. This option removes the row in the Transmissions table. Purges data with a specified filename. Wildcards will not work. If multiple files have the same name, they are all purged. This option removes the row in the Transmissions table. Using TI Purge Page 8 2/2/2012

13 Oracle example set path=c:\program Files\Java\jre1.6.0_02\bin;%PATH% java -jar "C:\Foresight\TIPurge\TIPurge.jar" -beyond 638 -purgemonthly 1 -maxthreads 15 -dbtype Oracle -db "jdbc:oracle:thin:ti330- TEST/TI410-TESTpw@//DBserver1:1521/or10.DBserver1" -logcfg "C:\Foresight\TIPurge\ORA-LoggingConfig.txt" Where: -beyond 638 Data older than 638 days will be purged -purgemonthly 1 Monthly statistics in the beyond range are purged also -maxthreads 15 TI330-TEST TI330-TESTpw DBserver1 Use 15 threads for purging Username Schema password Server hosting the database 1521 Connection port (1521 is default) or10.dbserver1 SQL Server example Oracle instance name set path=c:\program Files\Java\jre1.6.0_02\bin;%PATH% java -jar "C:\Foresight\TIPurge\TIPurge.jar" -beyond 638 -purgemonthly 1 -maxthreads 15 -dbtype SqlServer -db "jdbc:sqlserver://dbserver1:1433; databasename=ti410-test;user=tiuser;password=qawse246;" -logcfg " C:\Foresight\TIPurge\SQL-LoggingConfig.txt" Where: -beyond 638 Data older than 638 days will be purged -purgemonthly 1 Monthly statistics in the beyond range are purged also -maxthreads 15 DBSERVER Port TI330-TEST TIuser QaWs246 -logcfg Use 15 threads for purging Server hosting the database Database schema (user) name Schema user Schema password Logging configuration file is C:\Foresight\TIPurge\SQL-LoggingConfig.txt Stopping a purge TI Purge exits when it finishes a requested purge. You can stop it in mid-purge in the normal ways to kill a command line process (Ctrl- C, kill command, Task Manager, etc.). When restarted, it will resume where it left off. Using TI Purge Page 9 2/2/2012

14 After a purge Statistics pages like Transmissions or Documents will show monthly summary data for items that have been purged unless purgemonthly was set to 1. Purged documents are not included in statistics by day or week. There is no unpurge. If your database administrator optimizes the TI database after the purge, it will become smaller and more efficient. If it is not optimized after a purge, the database will be the same size. New data coming in will be placed in the newly-emptied gaps, thus causing the database to stay the same size or grow more slowly. Program success or failure The log and sysout will show one of these: Normal exit Abnormal exit Using TI Purge Page 10 2/2/2012

15 TIBCO Software Inc., Foresight Group 655 Metro Place South Suite 900 Dublin OH Phone: (614) Fax: (614) Web: