Jyotheswar Kuricheti 1
Agenda: 1. Performance Tuning Overview 2. Identify Bottlenecks 3. Optimizing at different levels : Target Source Mapping Session System 2
3
Performance Tuning Overview: 4
What is Performance Tuning? Goal of Performance Tuning How do you measure performance? Throughput Why is it Critical? Load time is very critical to meet SLA needs of the data availability in the reports. How do you improve performance? Identify Bottlenecks Eliminate Bottlenecks Test Load option to see if any improvement in the performance Add partitions Change one variable at a time 5
Reasons for Session Performance Issues: CPU : CPU intensive operations like string manipulation inside Expression transformation Memory/Disk access : File system read/write issues Paging (lookup cache etc.) due to non availability of RAM Non availability of buffer blocks Network : Database and PowerCenter servers connected by WAN Input/output Operations Poor/Complex Design Incorrect Load Strategies 6
The optimization can be done on different levels of Informatica: 1) Target level 2) Source level 3) Mapping level 4) Transformation level 5) Session level 6) Grid Level 7) Component level 8) System level 7
Identify Bottlenecks : 8
WRT_8165 : TIMEOUT BASED COMMIT POINT This is a common message that can be seen in the session log where there are session performance issues Signifies that there aren t enough rows available in memory to insert and issue a commit This message means that there is a bottle neck either in source, target or any of the transformations and the bottleneck needs to be identified and removed to improve session performance. 9
Methods to identify performance bottlenecks: 1. Run test sessions 2. Analyze performance details 3. Analyze thread statistics 4. Monitor system performance 1.Run Test Sessions: Running a test load of few records to read data from a flat file or to write to a flat file target to identify source and target bottlenecks. Precisely depending upon the throughput performance will be measured 2.Analyze thread statistics: Analyze thread statistics to determine the optimal number of partition points. 10
3.Analyze performance details: Like performance counters, to determine where session performance decreases. Use Collect Performance Data in Session properties Areas to check when repository performance is a concern User would like to see statistics from the monitor 4.Monitor system performance: You can use system monitoring tools to view the percentage of CPU use, I/O waits, and paging to identify system bottlenecks. Use the Workflow Monitor to view system resource usage. 11
2.Using Thread statistics: This is the way where we get statistics from a session log file. Before going we need to know few points about Thread. DTM (Data Transformation manager) create a master thread to run our sessions. For each target load order group in a mapping, the master thread can create several threads. The types of threads depend on the session properties and the transformations in the mapping. The number of threads depends on the partitioning information for each target load order group in the mapping. 1. Mapping Threads 2. Pre- and Post-Session Threads 3. Reader Threads 4. Transformation Threads 5. Writer Threads Thread analysis is to decide the mapping performance depending upon the statistics of threads. we can use these statistics to identifying the source, target, or transformation bottlenecks. From session log file we will have 4 entries which give details about performance. 12
1. Run Time : total time taken by a thread 2. Idle Time: time period where thread is idle. 3. Busy: (run time - idle time) / run time X 100 4. Thread work time: time taken by each transformation in a thread Example : MANAGER> PETL_24018 Thread [READER_1_1_1] created for the read stage of partition point [SQ_XXXX] has completed: Total Run Time = [576.620988] secs, Total Idle Time = [535.601729] secs, Busy Percentage = [7.113730]. MANAGER> PETL_24019 Thread [TRANSF_1_1_1_1] created for the transformation stage of partition point [SQ_XXXX] has completed: Total Run Time = [577.301318] secs, Total Idle Time = [0.000000] secs, Busy Percentage = [99.000000]. LKP_ADDRESS: 20.000000 percent AGG_ADDRESS: 79.000000 percent MANAGER> PETL_24022 Thread [WRITER_1_1_1] created for the write stage of partition point(s) [TGT_XXXX] has completed: Total Run Time = [577.363934] secs, Total Idle Time = [492.602374] secs, Busy Percentage = [14.680785]. The thread with the highest busy percentage identifies the bottleneck in the session. Blindly we can't add a partition point also to enable few thread to the process because the CPU may be busy with other task and just we are putting more pressure on CPU. Even we can ignore high busy percentage if total run time is less than 60 seconds 13
Optimizing at different levels : 14
(1).Target Level optimization: We are having two different target types. 1. Flat File 2. Database. If you are facing issue with flat file target then problem may not be with the flat file target, but problem lies with the storage space or with storage drive. Database: While loading into database we need to consider the following points. 1. Drop indexes and key constraints 2. Increase checkpoint intervals 3. Use bulk loading 4. Use external loading 5. Minimize deadlocks 6. Increase database network packet size 7. Optimize Oracle target databases 15
Drop indexes and key constraints : The loading of data will be slow on indexes or key constraints defined tables. Use pre-session commands to drop indexes before session loading. After loading the data the constraints or indexes need to be built again using postsession commands. Increase checkpoint intervals: The performance of loading depends on how many less check points do we have. To do so increase the checkpoint interval in the database Use bulk loading: Integration Service bypasses the database log, which speeds performance. Recovery is not possible. Use external loaders: With almost all the databases we have self built loading mechanism. Like for oracle SQL loader, for Teradata we can use Teradata external loader, to increase the performance we can load separate pipelines for separate partitions. Minimize deadlocks: We need to avoid attacking on same target from multiple sources systems, i.e. using multiple ways we shouldn't try to populate data at a single target. Use different target connection groups. 16
Increase database network packet size: If you realized that problem is with database consult your DBA and try to increase network packet size in listener.ora and tnsnames.ora Optimize Oracle target databases: With help of your DBA you can increase storage segments size or any database level changes can be created or added. Tune the Oracle redo log in the init.ora file 17
(2).Source Level Optimization: 1. Optimize the query. 2. Use conditional filters 3. Increase database network packet size Optimize the query: Join multiple sources in one SQ with hints, Indexes on group by and order by classes. Configure the source database to run parallel queries. Use conditional Filters: Apply filter on source data, but we need to do complete analysis before applying filters on source data. Connect ports from SQ only if they are needed in target. Increase database network packet size: If you realized that problem is with database consult your DBA and try to increase network packet size in listener.ora and tnsnames.ora 18
(3). Mapping Level Optimization: Mapping level optimization is a time taking process. Eliminate unwanted transformations, unwanted fields and links. The mapping optimization has to be done after source and target level optimization. 1. Optimize the flat file sources 2. Configure single-pass reading 3. Optimize Simple Pass Through mappings 4. Optimize filters 5. Optimize data type conversions 6. Optimize expressions Optimize the flat file sources: by avoiding double or single quotes and escape characters we can optimize flat file sources for delimiter files and managing the sequential buffer length for fixed files. Configure single-pass reading: Consider using single-pass reading if you have multiple sessions that use the same sources. It allows you to populate multiple targets with one source qualifier. It avoids using of a joiner for RDBMS source tables. 19
20
Optimize Simple Pass Through mappings : If we are passing data from source to target, connect directly from source qualifier to target. If use wizard to create Simple Pass Through mappings it will add an expression in between target and source qualifier. Optimize filters: If your source is a relational table use filter at source qualifier. It restricts some of the rows which are not valid for mapping process. For flat file sources, use filter transformation after the source qualifier. Avoid complex conditions at filter, go for integer or true/false conditions. Optimize datatype conversions: eliminate unnecessary datatype conversions. Use integer values in conditions of Lookup and Filter transformations. Before doing data conversion be aware of source and target data types. Optimize expressions: Factoring Out Common Logic. Minimizing Aggregate Function Calls. Ex: Use SUM(COLUMN_A + COLUMN_B) instead of SUM(COLUMN_A) + SUM(COLUMN_B) Call lookups conditionally. Use local variables in expression transformation. Use operators instead of Functions. 21
(4).Transformation level Optimization: The Transformation level optimization we can consider as a part of mapping optimization. Here we will get more information how to handle transformations more effectively. Optimizing Aggregator Transformations: They often slow performance because they must group data before processing it. 1. Group on Numeric : Group on numeric columns instead of string and date columns. 2. Group on Indexed Columns: 3. Using Sorted input: It reduces the amount of data cached which improves performance. 4. Reduce complex logic in aggregator expressions 5. Using Incremental Aggregation 5. Filter Data Before You Aggregate. 6. Limiting Port Connections 22
Optimizing Joiner Transformations : Joiner joins data of different sources into a single pipeline. 1.Designate the master source as the source with fewer duplicate key values. 2.Designate the master source as the source with fewer rows as it compares each row of the detail source against the master source 3.Perform joins in a database. Use SQ to perform join for relational tables 4.Use Sorted Data to join Optimizing Lookup Transformations : Caching lookup tables: Use the appropriate cache type: Static, Shared and Persistent caches Enable concurrent caches: number of additional concurrent pipelines is set to one or more Optimize Lookup policy on multiple matching: use any matching value, performance can improve because the transformation does not index on all ports but it still returns first value that matches lkp condition. Reduce the number of cached rows. Override the ORDER BY statement. 23
Optimizing the Lookup Condition: =,<,>,<=,>=,!= Filtering Lookup Rows Indexing the conditional columns in Lookup Table Optimizing Multiple Lookups Optimizing Sequence Generator Transformations: create a reusable Sequence Generator and use it in multiple mappings simultaneously. By configuring the Number of Cached Values property for sequence number we get some good results. Optimizing Sorter Transformations: If the Integration Service cannot allocate enough memory to sort data, it fails the session. For best performance, configure Sorter cache size with a value less than or equal to the amount of available physical RAM on the Integration Service machine. Default size is 16 MB Use the following formula to determine the size of incoming data: # input rows ([Sum(column size)] + 16 24
Optimizing Source Qualifier Transformations: Select distinct, filter, tune the query Optimizing SQL Transformations: Use query mode instead of script mode Do not use transaction statements like commit, rollback in an SQL transformation query. In query mode construct a static query by using parameter binding instead of string substitution in the SQL Editor Choose static connection instead of dynamic connection 25
(5).Session Level optimization: 1.Grid 2.Pushdown Optimization 3.Concurrent Sessions and Workflows 4.Buffer Memory 5.Caches 6.Target-Based Commit 7.Real-time Processing 8.Staging Areas 9.Log Files 10.Error Tracing 11.Post-Session Emails 26
1.Grid: A Load Balancer distributes tasks to nodes without overloading any node. A grid can improve performance when you have a performance bottleneck in the extract and load steps of a session, when memory or temporary storage is a performance bottleneck Ex : Sorter, Aggregator, Joiner (stores intermediate results) 2.Pushdown Optimization: Integration Service executes SQL against the source or target database instead of processing the transformation logic within the Integration Service. 3.Concurrent Sessions and Workflows : 4.Buffer Memory: Adjust DTM Buffer Size & Default Buffer Block Size 27
5.Caches: Limit the Number of Connected Ports With a 64-bit platform, the Integration Service is not limited to the 2 GB cache limit of a 32-bit platform. If the allocated cache is not large enough to store the data, the Integration Service stores the data in a temporary disk file, a cache file. Performance slows each time the Integration Service pages to a temporary file. The Transformation_readfromdisk or Transformation_writetodisk counters for any Aggregator, Rank, or Joiner transformation indicate the number of times the Integration Service pages to disk to process the transformation. 6.Target-Based Commit : If the commit interval is too high, the Integration Service may fill the database log file and cause the session to fail. 28
7.Real-time Processing: Increase the flush latency to improve throughput Source-based commit interval determines how often the Integration Service commits real-time data to the target. To obtain the fastest latency, set the source-based commit to 1. 8.Staging Areas: The Integration Service can read multiple sources with a single pass, which can reduce the need for staging areas. 9.Log Files: Workflows and sessions always create binary logs which can be accessed in the Administrator tool. 10.Error Tracing: Set the tracing level appropriately. To debug use Verbose. Use Terse when you do not want to log error messages for reject data. 29
11.Post-Session Emails: configure the session to write to log file when you configure post-session email to attach a session log. Enable flat file logging (6).Optimizing Grid Deployments: Add nodes to the grid. Increase storage capacity and bandwidth. Use shared file systems. Use a high-throughput network (7).Optimizing PowerCenter Repository Performance: Ensure the Repository Service process runs on the same machine where the repository database resides Order conditions in object queries Use a single-node tablespace for the PowerCenter repository if you install it on a DB2 db. Optimize the database schema for the PowerCenter repository if you install it on a DB2 or Microsoft SQL Server database by 30
enabling the Optimize Database Schema option for the Repository Service in the Administration Console. Optimizing Integration Service Performance: Use native drivers instead of ODBC drivers for the Integration Service. Run the Integration Service in ASCII data movement mode if character data is 7-bit ASCII or EBCDIC. ASCII mode take 1 byte to store each character where as UNICODE takes 2 bytes. Cache PowerCenter metadata for the Repository Service. Run Integration Service with high availability : Integration Service recovers workflows and sessions that may fail because of temporary network or machine failures. To recover from a workflow or session, the Integration Service writes the states of each workflow and session to temporary files in a shared directory which may decrease performance 31
(8).Optimizing the System: Improve network speed : Minimize the number of network hops between the source and target databases and the IS A local disk can move data 5 to 20 times faster than a network. Store flat files as source or target in IS machine Move Target DB to a Server System if possible Ask Network Engineer to provide enough Bandwidth Use multiple CPUs to run multiple sessions in parallel Reduce paging Use processor binding Using Pipeline Partitions: After you tune the application, databases, and system for maximum singlepartition performance, you may find that the system is under-utilized. At this point, you can configure the session to have two or more partitions To improve performance, ensure the number of pipeline partitions equals the number of database partitions. 32
Use the database partitioning partition type for source and target databases. Enable parallel queries/inserts SQ : pass-through partition Filter : round-robin partition Sorter : hash auto-keys partitioning. Delete default partition at Aggregator Performance Counters : All transformations have counters. The Integration Service tracks the number of input rows, output rows, and error rows for each transformation. Some transformations have performance counters: right-click the session in the Workflow Monitor and choose Properties. Click the Properties tab in the details dialog box. Errorrows Readfromcache and Writetocache Readfromdisk and Writetodisk Rowsinlookupcache 33
If these counters display any number other than zero, you can increase the cache sizes to improve session performance. 34
2014 by Author (Jyotheswar Kuricheti). All rights reserved. No part of this document may be reproduced or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without prior written permission of Author. 35