Performance Tuning

Informatica PowerCenter
(Version 8.6.1)
Jishnu Pramanik

1.Performance Tuning Overview

1.1 Overview
The goal of performance tuning is to optimize session performance by eliminating performance bottlenecks. To tune
session performance, first identify a performance bottleneck, eliminate it, and then identify the next performance
bottleneck until you are satisfied with the session performance. You can use the test load option to run sessions
when you tune session performance.
If you tune all the bottlenecks, you can further optimize session performance by increasing the number of pipeline
partitions in the session. Adding partitions can improve performance by utilizing more of the system hardware while
processing the session.

1.2 Need for Performance Tuning

Performance is not just one job loading maximum data in a particular time frame. Performance can be more
accurately defined as a combination of several small jobs which affect the overall performance of a system.

Informatica is an ETL tool with high performance capability. We need to make maximum utilization of its features
to increase its performance. With the ever increasing user requirements and exploding data volumes, we need to
achieve more in less time. The goal of performance tuning is optimize session performance. This document lists all
the techniques available to tune Informatica performance.

2.Identifying Bottlenecks
2.1 Overview
Performance of Informatica is dependent on the performance of its several components like database, network,
transformations, mappings, sessions etc. To tune the performance of Informatica, we have to identify the bottleneck
first. Bottleneck may be present in source, target, transformations, mapping, session, database or network. It is best
to identify performance issue in components in the order source, target, transformations, mapping and session. After
identifying the bottleneck, apply the tuning mechanisms in whichever way they are applicable to the project.

2.2 Identify bottleneck in Source

If source is a relational table, put a filter transformation in the mapping, just after source qualifier; make the
condition of filter to FALSE. So all records will be filtered off and none will proceed to other parts of the mapping.
In original case, without the test filter, total time taken is as follows:-

Total Time = time taken by (source + transformations + target load)

Now because of filter, Total Time = time taken by source

So if source was fine, then in the latter case, session should take less time. Still if the session takes near equal time
as former case, then there is a source bottleneck.

2.3 Identify bottleneck in Target

The most common performance bottleneck occurs when the Integration Service writes to a target database. To
identify a target bottleneck, configure a copy of the session to write to a flat file target. If the session
performance increases significantly when you write to a flat file, you have a target bottleneck. If a session
already writes to a flat file target, you probably do not have a target bottleneck.

2.4 Identify bottleneck in Transformation

Remove the transformation from the mapping and run it. Note the time taken. Then put the transformation back and
run the mapping again. If the time taken now is significantly more than previous time, then the transformation is the
bottleneck.

But removal of transformation for testing can be a pain for the developer since that might require further changes for
the session to get into the ‘working mode’. So we can put filter with the FALSE condition just after the
transformation and run the session. If the session run takes equal time with and without this test filter, then
transformation is the bottleneck.
2.5 Identify bottleneck in sessions
We can use the session log to identify whether the source, target or transformations are the performance bottleneck.
Session logs contain thread summary records like the following:-

MASTER> PETL_24018 Thread [READER_1_1_1] created for the read stage of

partition point [SQ_test_all_text_data] has completed: Total Run Time =
[11.703201] secs, Total Idle Time = [9.560945] secs, Busy Percentage =
[18.304876].
MASTER> PETL_24019 Thread [TRANSF_1_1_1_1] created for the transformation
stage of partition point [SQ_test_all_text_data] has completed: Total Run
Time = [11.764368] secs, Total Idle Time = [0.000000] secs, Busy Percentage
= [100.000000].

If busy percentage is 100, then that part is the bottleneck.

Basically we have to rely on thread statistics to identify the cause of performance issues. Once the ‘Collect
Performance Data’ option (In session- ‘Properties’ tab) is enabled, all the performance related information would
appear in the log created by the session.

2.6 Identifying System Bottlenecks on Windows

On Windows, you can view the Performance and Processes tab in the Task Manager. To access the Task Manager,
press use Ctrl+Alt+Del, and click Task Manager. The Performance tab in the Task Manager provides an overview of
CPU usage and total memory used. You can view more detailed performance information by using the Performance
Monitor on Windows. To access the Performance Monitor, click Start > Programs > Administrative Tools, and
choose Performance Monitor.
Use the Windows Performance Monitor to create a chart that provides the following information:
♦Percent processor time : If you have more than one CPU, monitor each CPU for percent processor time. If the
processors are utilized at more than 80%, you may consider adding more processors.
♦Pages/second : If pages/second is greater than five, you may have excessive memory pressure (thrashing). You
may consider adding more physical memory.
♦Physical disks percent time : The percent of time that the physical disk is busy performing read or write requests.
If the percent of time is high, tune the cache for PowerCenter to use in-memory cache instead of writing to disk. If
you tune the cache, requests are still in queue, and the disk busy percentage is at least 50%, add another disk device
or upgrade to a faster disk device: You can also use a separate disk for each partition in the session.
♦Physical disks queue length : The number of users waiting for access to the same disk device. If physical disk
queue length is greater than two, you may consider adding another disk device or upgrading the disk device. You
also can use separate disks for the reader, writer, and transformation threads.
♦Server total bytes per second : This is the number of bytes the server has sent to and received from the network.
You can use this information to improve network bandwidth.

3.Optimizing the Target

3.1 Overview
You can optimize the following types of targets:
♦Flat file
♦Relational

3.2 Flat File Target

If you use a shared storage directory for flat file targets, you can optimize session performance by ensuring that the
shared storage directory is on a machine that is dedicated to storing and managing files, instead of performing other
tasks.
If the Integration Service runs on a single node and the session writes to a flat file target, you can optimize session
performance by writing to a flat file target that is local to the Integration Service process node.

3.3 Relational Target

If the session writes to a relational target, you can perform the following tasks to increase performance:
♦Drop indexes and key constraints : When you define key constraints or indexes in target tables, you slow the
loading of data to those tables. To improve performance, drop indexes and key constraints before running the
session. You can rebuild those indexes and key constraints after the session completes.
If you decide to drop and rebuild indexes and key constraints on a regular basis, you can use the following methods
to perform these operations each time you run the session:
-Use pre-load and post-load stored procedures.
-Use pre-session and post-session SQL commands.

♦Increase checkpoint intervals : The Integration Service performance slows each time it waits for the database to
perform a checkpoint. To increase performance, consider increasing the database checkpoint interval. When you
increase the database checkpoint interval, you increase the likelihood that the database performs checkpoints as
necessary, when the size of the database log file reaches its limit.

♦Use bulk loading : You can use bulk loading to improve the performance of a session that inserts a large amount
of data into a DB2, Sybase ASE, Oracle, or Microsoft SQL Server database. Configure bulk loading in the session
properties.
When bulk loading, the Integration Service bypasses the database log, which speeds performance. Without writing
to the database log, however, the target database cannot perform rollback. As a result, you may not be able to
perform recovery. When you use bulk loading, weigh the importance of improved session
performance against the ability to recover an incomplete session.
When bulk loading to Microsoft SQL Server or Oracle targets, define a large commit interval to increase
performance. Microsoft SQL Server and Oracle start a new bulk load transaction after each commit. Increasing the
commit interval reduces the number of bulk load transactions, which increases performance
♦Use external loading : You can use an external loader to increase session performance. If you have a DB2 EE or
DB2 EEE target database, you can use the DB2 EE or DB2 EEE external loaders to bulk load target files. The DB2
EE external loader uses the Integration Service db2load utility to load data. The DB2 EEE external loader uses the
DB2 Autoloader utility.
If you have a Teradata target database, you can use the Teradata external loader utility to bulk load target files. To
use the Teradata external loader utility, set up the attributes, such as Error Limit, Tenacity, MaxSessions, and Sleep,
to optimize performance.
If the target database runs on Oracle, you can use the Oracle SQL*Loader utility to bulk load target files. When you
load data to an Oracle database using a pipeline with multiple partitions, you can increase performance if you create
the Oracle target table with the same number of partitions you use for the pipeline.
If the target database runs on Sybase IQ, you can use the Sybase IQ external loader utility to bulk load target files. If
the Sybase IQ database is local to the Integration Service process on the UNIX system, you can increase
performance by loading data to target tables directly from named pipes. If you run the Integration Service on a grid,
configure the Integration Service to check resources, make Sybase IQ a resource, make the resource available on all
nodes of the grid, and then, in the Workflow Manager, assign the Sybase IQ resource to the applicable sessions.

♦Minimize deadlocks : If the Integration Service encounters a deadlock when it tries to write to a target, the
deadlock only affects targets in the same target connection group. The Integration Service still writes to targets in
other target connection groups.
Encountering deadlocks can slow session performance. To improve session performance, you can increase the
number of target connection groups the Integration Service uses to write to the targets in a session. To use a different
target connection group for each target in a session, use a different database connection name for each target
instance. You can specify the same connection information for each connection name.

♦Increase database network packet size : If you write to Oracle, Sybase ASE or, Microsoft SQL Server targets,
you can improve the performance by increasing the network packet size. Increase the network packet size to allow
larger packets of data to cross the network at one time. Increase the network packet size based on the database you
write to:
-Oracle. You can increase the database server network packet size in listener.ora and tnsnames.ora. Consult your
database documentation for additional information about increasing the packet size, if necessary.
-Sybase ASE and Microsoft SQL. Consult your database documentation for information about how to increase the
packet size.
For Sybase ASE or Microsoft SQL Server, you must also change the packet size in the relational connection object
in the Workflow Manager to reflect the database server packet size.

♦Optimize Oracle target databases : If the target database is Oracle, you can optimize the target database by
checking the storage clause, space allocation, and rollback or undo segments.
When you write to an Oracle database, check the storage clause for database objects. Make sure that tables are using
large initial and next values. The database should also store table and index data in separate tablespaces, preferably
on different disks.
When you write to Oracle databases, the databases uses rollback or undo segments during loads. Ask the Oracle
database administrator to ensure that the database stores rollback or undo segments in appropriate tablespaces,
preferably on different disks. The rollback or undo segments should also have appropriate storage clauses.
You can optimize the Oracle database by tuning the Oracle redo log. The Oracle database uses the redo log to log
loading operations. Make sure the redo log size and buffer size are optimal. You can view redo log properties in the
init.ora file.
If the Integration Service runs on a single node and the Oracle instance is local to the Integration Service process
node, you can optimize performance by using IPC protocol to connect to the Oracle database. You can set up Oracle
database connection in listener.ora and tnsnames.ora.

3.4 Tips and Tricks

• If the target is a flat file, ensure that the flat file is local to the Informatica server. If target is a relational
table, then try not to use synonyms or aliases.

• Use bulk load whenever possible.

• Increase the commit level.

• Drop constraints and indexes of the table before loading.

4.Optimizing the Source

4.1 Overview
If the session reads from a relational source, review the following suggestions for improving performance:
♦Optimize the query.
♦Use conditional filters.
♦Increase database network packet size
♦Connect to Oracle databases using IPC protocol.
♦Use the FastExport utility to extract Teradata data.
♦Create tempdb to join Sybase ASE or Microsoft SQL Server tables.

4.2 Optimizing the Query

If a session joins multiple source tables in one Source Qualifier, you might be able to improve performance by
optimizing the query with optimizing hints. Also, single table select statements with an ORDER BY or GROUP BY
clause may benefit from optimization such as adding indexes.
Usually, the database optimizer determines the most efficient way to process the source data. However, you might
know properties about the source tables that the database optimizer does not. The database administrator can create
optimizer hints to tell the database how to execute the query for a particular set of source tables.
The query that the Integration Service uses to read data appears in the session log. You can also find the query in the
Source Qualifier transformation. Have the database administrator analyze the query, and then create optimizer hints
and indexes for the source tables.
Use optimizing hints if there is a long delay between when the query begins executing and when PowerCenter
receives the first row of data. Configure optimizer hints to begin returning rows as quickly as possible, rather than
returning all rows at once. This allows the Integration Service to process rows parallel with the query execution.
Queries that contain ORDER BY or GROUP BY clauses may benefit from creating an index on the ORDER BY or
GROUP BY columns. Once you optimize the query, use the SQL override option to take full advantage of these
modifications.
You can also configure the source database to run parallel queries to improve performance. For more information
about configuring parallel queries, see the database documentation.

4.3 Using Conditional Filters

A simple source filter on the source database can sometimes negatively impact performance because of the lack of
indexes. You can use the PowerCenter conditional filter in the Source Qualifier to improve performance.
Whether you should use the PowerCenter conditional filter to improve performance depends on the session. For
example, if multiple sessions read from the same source simultaneously, the PowerCenter conditional filter may
improve performance.
However, some sessions may perform faster if you filter the source data on the source database. You can test the
session with both the database filter and the PowerCenter filter to determine which method improves performance.

4.4 Increasing Database Network Packet Size

If you read from Oracle, Sybase ASE or, Microsoft SQL Server sources, you can improve the performance by
increasing the network packet size. Increase the network packet size to allow larger packets of data to cross the
network at one time. Increase the network packet size based on the database you read from:
-Oracle. You can increase the database server network packet size in listener.ora and tnsnames.ora. Consult your
database documentation for additional information about increasing the packet size, if necessary.
-Sybase ASE and Microsoft SQL. Consult your database documentation for information about how to increase the
packet size.
For Sybase ASE or Microsoft SQL Server, you must also change the packet size in the relational connection object
in the Workflow Manager to reflect the database server packet size.

4.5 Connecting to Oracle Database Sources

If you are running the Integration Service on a single node and the Oracle instance is local to the Integration Service
process node, you can optimize performance by using IPC protocol to connect to the Oracle database. You can set
up an Oracle database connection in listener.ora and tnsnames.ora.

4.6 Using Teradata FastExport

FastExport is a utility that uses multiple Teradata sessions to quickly export large amounts of data from a Teradata
database. You can create a PowerCenter session that uses FastExport to read Teradata sources quickly. To use
FastExport, create a mapping with a Teradata source database. In the session, use FastExport reader instead of
Relational reader. Use a FastExport connection to the Teradata tables that you want to export in a session.

4.7 Using tempdb to Join Sybase or Microsoft SQL Server

Tables
When you join large tables on a Sybase or Microsoft SQL Server database, it is possible to improve performance by
creating the tempdb as an in-memory database to allocate sufficient memory. For more information, see the Sybase
or Microsoft SQL Server documentation.

4.8 Tips and Tricks

• If the source is a flat file, ensure that the flat file is local to the Informatica server. If source is a relational
table, then try not to use synonyms or aliases.

• If the source is a flat file, reduce the number of bytes (By default it is 1024 bytes per line) the Informatica
reads per line. If we do this, we can decrease the Line Sequential Buffer Length setting of the session
properties.

• If possible, give a conditional query in the source qualifier so that the records are filtered off as soon as
possible in the process.

• In the source qualifier, if the query has ORDER BY or GROUP BY, then create an index on the source
table and order by the index field of the source table.

5.Optimizing Mappings
5.1 Overview
Mapping-level optimization may take time to implement, but it can significantly boost session performance. Focus
on mapping-level optimization after you optimize the targets and sources.
Generally, you reduce the number of transformations in the mapping and delete unnecessary links between
transformations to optimize the mapping. Configure the mapping with the least number of transformations and
expressions to do the most amount of work possible. Delete unnecessary links between transformations to minimize
the amount of data moved.
You can also perform the following tasks to optimize the mapping:
♦Optimize the flat file sources.
♦Configure single-pass reading.
♦Optimize Simple Pass Through mappings.
♦Optimize filters.
♦Optimize datatype conversions
♦Optimize expressions
♦Optimize external procedures.

5.2 Optimizing Flat File Sources

Complete the following tasks to optimize flat file sources:
♦Optimize the line sequential buffer length.
♦Optimize delimited flat file sources.
♦Optimizing XML and flat file sources.

-Optimizing the Line Sequential Buffer Length

If the session reads from a flat file source, you can improve session performance by setting the number of bytes the
Integration Service reads per line. By default, the Integration Service reads 1024 bytes per line. If each line in the
source file is less than the default setting, you can decrease the line sequential buffer length in the session properties.
-Optimizing Delimited Flat File Sources
If a source is a delimited flat file, you must specify the delimiter character to separate columns of data in the source
file. You must also specify the escape character. The Integration Service reads the delimiter character as a regular
character if you include the escape character before the delimiter character. You can improve session performance if
the source flat file does not contain quotes or escape characters.

-Optimizing XML and Flat File Sources

XML files are usually larger than flat files because of the tag information. The size of an XML file depends on the
level of tagging in the XML file. More tags result in a larger file size. As a result, the Integration Service may take
longer to read and cache XML sources.

5.3 Configuring Single-Pass Reading

Single-pass reading allows you to populate multiple targets with one source qualifier. Consider using single-pass
reading if you have multiple sessions that use the same sources. You can combine the transformation logic for each
mapping in one mapping and use one source qualifier for each source. The Integration Service reads each source
once and then sends the data into separate pipelines. A particular row can be used by all the pipelines, by any
combination of pipelines, or by no pipelines.
For example, you have the Purchasing source table, and you use that source daily to perform an aggregation and a
ranking. If you place the Aggregator and Rank transformations in separate mappings and sessions, you force the
Integration Service to read the same source table twice. However, if you include the aggregation and ranking logic
in one mapping with one source qualifier, the Integration Service reads the Purchasing source table once, and then
sends the appropriate data to the two separate pipelines.
When changing mappings to take advantage of single-pass reading, you can optimize this feature by factoring out
common functions from mappings. For example, if you need to subtract a percentage from the Price ports for both
the Aggregator and Rank transformations, you can minimize work by subtracting the percentage before splitting the
pipeline. You can use an Expression transformation to subtract the percentage, and then split the mapping after the
transformation.

5.4 Optimizing Simple Pass through Mappings

You can optimize performance for Simple Pass Through mappings. To pass directly from source to target without
any other transformations, connect the Source Qualifier transformation directly to the target. If you use a mapping
wizard to create a Simple Pass Through mapping, the wizard creates an Expression transformation between the
Source Qualifier transformation and the target.

5.5 Optimizing Filters

Use one of the following methods to filter data:
♦Use a Source Qualifier transformation : The Source Qualifier transformation filters rows from relational
sources.
♦Use a Filter transformation : The Filter transformation filters data within a mapping. The Filter transformation
filters rows from any type of source.
If you filter rows from the mapping, you can improve efficiency by filtering early in the data flow. Use a filter in the
Source Qualifier transformation to remove the rows at the source. The Source Qualifier transformation limits the
row set extracted from a relational source.
If you cannot use a filter in the Source Qualifier transformation, use a Filter transformation and move it as close to
the Source Qualifier transformation as possible to remove unnecessary data early in the data flow. The Filter
transformation limits the row set sent to a target.
Avoid using complex expressions in filter conditions. You can optimize Filter transformations by using simple
integer or true/false expressions in the filter condition.
Note: You can also use a Filter or Router transformation to drop rejected rows from an Update Strategy
transformation if you do not need to keep rejected rows.
5.6 Optimizing Datatype Conversions
You can increase performance by eliminating unnecessary datatype conversions. For example, if a mapping moves
data from an Integer column to a Decimal column, then back to an Integer column, the unnecessary datatype
conversion slows performance. Where possible, eliminate unnecessary datatype conversions from mappings.
Use the following datatype conversions to improve system performance:
♦Use integer values in place of other datatypes when performing comparisons using Lookup and Filter
transformations. For example, many databases store U.S. zip code information as a Char or Varchar datatype. If you
convert the zip code data to an Integer datatype, the lookup database stores the zip code 94303-1234 as 943031234.
This helps increase the speed of the lookup comparisons based on zip code.
♦Convert the source dates to strings through port-to-port conversions to increase session performance. You
can either leave the ports in targets as strings or change the ports to Date/Time ports.

5.7 Optimizing Expressions

You can also optimize the expressions used in the transformations. When possible, isolate slow expressions and
simplify them.
Complete the following tasks to isolate the slow expressions:
1. Remove the expressions one-by-one from the mapping.
2. Run the mapping to determine the time it takes to run the mapping without the transformation.
If there is a significant difference in session run time, look for ways to optimize the slow expression.

Factoring Out Common Logic

If the mapping performs the same task in multiple places, reduce the number of times the mapping performs the task
by moving the task earlier in the mapping. For example, you have a mapping with five target tables. Each target
requires a Social Security number lookup. Instead of performing the lookup five times, place the Lookup
transformation in the mapping before the data flow splits. Next, pass the lookup results to all five targets.

Minimizing Aggregate Function Calls

When writing expressions, factor out as many aggregate function calls as possible. Each time you use an aggregate
function call, the Integration Service must search and group the data. For example, in the following expression, the
Integration Service reads COLUMN_A, finds the sum, then reads COLUMN_B, finds the sum, and finally finds the
sum of the two sums:
SUM(COLUMN_A) + SUM(COLUMN_B)

If you factor out the aggregate function call, as below, the Integration Service adds COLUMN_A to COLUMN_B,
then finds the sum of both.
SUM(COLUMN_A + COLUMN_B)

Replacing Common Expressions with Local Variables

If you use the same expression multiple times in one transformation, you can make that expression a local variable.
You can use a local variable only within the transformation. However, by calculating the variable only once, you
speed performance.
Choosing Numeric Versus String Operations
The Integration Service processes numeric operations faster than string operations. For example, if you look up large
amounts of data on two columns, EMPLOYEE_NAME and EMPLOYEE_ID, configuring the lookup around
EMPLOYEE_ID improves performance.

Optimizing Char-Char and Char-Varchar Comparisons

When the Integration Service performs comparisons between CHAR and VARCHAR columns, it slows each time it
finds trailing blank spaces in the row. You can use the Treat CHAR as CHAR On Read option when you configure
the Integration Service in the Administration Console so that the Integration Service does not trim trailing spaces
from the end of Char source fields

Choosing DECODE Versus LOOKUP

When you use a LOOKUP function, the Integration Service must look up a table in a database. When you use a
DECODE function, you incorporate the lookup values into the expression so the Integration Service does not have
to look up a separate table. Therefore, when you want to look up a small set of unchanging values, using DECODE
may improve performance.

Using Operators Instead of Functions

The Integration Service reads expressions written with operators faster than expressions with functions. Where
possible, use operators to write expressions. For example, you have the following expression that contains nested
CONCAT functions:

CONCAT( CONCAT( CUSTOMERS.FIRST_NAME, ‘ ’) CUSTOMERS.LAST_NAME)

You can rewrite that expression with the || operator as follows:

CUSTOMERS.FIRST_NAME || ‘ ’ || CUSTOMERS.LAST_NAME

Optimizing IIF Expressions

IIF expressions can return a value and an action, which allows for more compact expressions. For example, you
have a source with three Y/N flags: FLG_A, FLG_B, FLG_C. You want to return values based on the values of each
flag.

You use the following expression:

IIF( FLG_A = 'Y' and FLG_B = 'Y' AND FLG_C = 'Y',

VAL_A + VAL_B + VAL_C,

IIF( FLG_A = 'Y' and FLG_B = 'Y' AND FLG_C = 'N',

VAL_A + VAL_B,

IIF( FLG_A = 'Y' and FLG_B = 'N' AND FLG_C = 'Y',

VAL_A + VAL_C,

IIF( FLG_A = 'Y' and FLG_B = 'N' AND FLG_C = 'N',

VAL_A ,

IIF( FLG_A = 'N' and FLG_B = 'Y' AND FLG_C = 'Y',

VAL_B + VAL_C,

IIF( FLG_A = 'N' and FLG_B = 'Y' AND FLG_C = 'N',

VAL_B ,

IIF( FLG_A = 'N' and FLG_B = 'N' AND FLG_C = 'Y',

VAL_C,

IIF( FLG_A = 'N' and FLG_B = 'N' AND FLG_C = 'N', 0.0,
))))))))

This expression requires 8 IIFs, 16 ANDs, and at least 24 comparisons.

If you take advantage of the IIF function, you can rewrite that expression as:

IIF(FLG_A='Y', VAL_A, 0.0)+ IIF(FLG_B='Y', VAL_B, 0.0)+ IIF(FLG_C='Y', VAL_C, 0.0)

This results in three IIFs, two comparisons, two additions, and a faster session.

Evaluating Expressions
If you are not sure which expressions slow performance, evaluate the expression performance to isolate the problem.

Complete the following steps to evaluate expression performance:

1. Time the session with the original expressions.

2. Copy the mapping and replace half of the complex expressions with a constant.

3. Run and time the edited session.

4. Make another copy of the mapping and replace the other half of the complex expressions with a constant.

5. Run and time the edited session.

5.8 Optimizing External Procedures

You might want to block input data if the external procedure needs to alternate reading from input groups. Without
the blocking functionality, you would need to write the procedure code to buffer incoming data. You can block input
data instead of buffering it which usually increases session performance.

For example, you need to create an external procedure with two input groups. The external procedure reads a row
from the first input group and then reads a row from the second input group. If you use blocking, you can write the
external procedure code to block the flow of data from one input group while it processes the data from the other
input group. When you write the external procedure code to block data, you increase performance because the
procedure does not need to copy the source data to a buffer. However, you could write the external procedure to
allocate a buffer and copy the data from one input group to the buffer until it is ready to process the data. Copying
source data to a buffer decreases performance.
5.9 Tips and Tricks
• Avoid executing major sql queries from mapplets or mappings.
• Use optimized queries when we are using them.
• Reduce the number of transformations in the mapping. Active transformations like rank, joiner, filter,
aggregator etc should be used as less as possible.
• Remove all the unnecessary links between the transformations from mapping.
• If a single mapping contains many targets, then dividing them into separate mappings can improve
performance.
• If we need to use a single source more than once in a mapping, then keep only one source and source
qualifier in the mapping. Then create different data flows as required into different targets or same target.
• If a session joins many source tables in one source qualifier, then an optimizing query will improve
performance.
• In the sql query that Informatica generates, ORDERBY will be present. Remove the ORDER BY clause if
not needed or at least reduce the number of column names in that list. For better performance it is best to
order by the index field of that table.
• Combine the mappings that use same set of source data.
• On a mapping, field with the same information should be given the same type and length throughout the
mapping. Otherwise time will be spent on field conversions.
• Instead of doing complex calculation in query, use an expression transformer and do the calculation in the
mapping.
• If data is passing through multiple staging areas, removing the staging area will increase performance.
• Stored procedures reduce performance. Try to keep the stored procedures simple in the mappings.
• Unnecessary data type conversions should be avoided since the data type conversions impact performance.
• Transformation errors result in performance degradation. Try running the mapping after removing all
transformations. If it is taking significantly less time than with the transformations, then we have to fine-
tune the transformation.
• Keep database interactions as less as possible.

6. Optimizing Transformations
6.1 Overview
You can further optimize mappings by optimizing the transformations contained in the mappings.
You can optimize the following transformations in a mapping:
♦Aggregator transformations.
♦Custom transformations.
♦Joiner transformations.
♦Lookup transformations.
♦Sequence Generator transformations.
♦Sorter transformations.
♦Source Qualifier transformations.
♦SQL transformations
♦Update transformation
♦Filter transformation
♦Expression transformation

6.2 Optimizing Aggregator Transformations

Aggregator transformations often slow performance because they must group data before processing it. Aggregator
transformations need additional memory to hold intermediate group results.
You can use the following guidelines to optimize the performance of an Aggregator transformation:
♦Group by simple columns.
♦Use sorted input.
♦Use incremental aggregation.
♦Filter data before you aggregate it.
♦Limit port connections.

Group By Simple Columns

You can optimize Aggregator transformations when you group by simple columns. When possible, use numbers
instead of string and dates in the columns used for the GROUP BY. Avoid complex expressions in the Aggregator
expressions.

Use Sorted Input

You can increase session performance by sorting data for the Aggregator transformation. Use the Sorted Input
option to sort data.
The Sorted Input option decreases the use of aggregate caches. When you use the Sorted Input option, the
Integration Service assumes all data is sorted by group. As the Integration Service reads rows for a group, it
performs aggregate calculations. When necessary, it stores group information in memory.
The Sorted Input option reduces the amount of data cached during the session and improves performance. Use this
option with the Source Qualifier Number of Sorted Ports option or a Sorter transformation to pass sorted data to the
Aggregator transformation.
You can benefit from better performance when you use the Sorted Input option in sessions with multiple partitions.

Use Incremental Aggregation

If you can capture changes from the source that affect less than half the target, you can use Incremental Aggregation
to optimize the performance of Aggregator transformations.

When using incremental aggregation, you apply captured changes in the source to aggregate calculations in a
session. The Integration Service updates the target incrementally, rather than processing the entire source and
recalculates the same calculations every time you run the session.
You can increase the index and data cache sizes to hold all data in memory without paging to disk.

Filter Data before You Aggregate

Filter the data before you aggregate it. If you use a Filter transformation in the mapping, place the transformation
before the Aggregator transformation to reduce unnecessary aggregation.

Limit Port Connections

Limit the number of connected input/output or output ports to reduce the amount of data the Aggregator
transformation stores in the data cache.

Tips and Tricks

Aggregator transformation helps in performing aggregate calculations like SUM, AVERAGE etc.
1. Aggregator, rank and joiner transformation will decrease performance since they group data before
processing. So to improve performance here, use sorted ports.
2. In aggregator transformation, in the GROUP BY clause, use numbers instead of strings if possible.
3. Avoid complex expressions in aggregator conditions.
4. Limit the number of connected input or output ports. This reduces the cache size.

6.3 Optimizing Custom Transformations

The Integration Service can pass a single row to a Custom transformation procedure or a block of rows in an array.
You can write the procedure code to specify whether the procedure receives one row or a block of rows. You can
increase performance when the procedure receives a block of rows:
♦You can decrease the number of function calls the Integration Service and procedure make. The Integration Service
calls the input row notification function fewer times, and the procedure calls the output notification function fewer
times.
♦You can increase the locality of memory access space for the data.
♦You can write the procedure code to perform an algorithm on a block of data instead of each row of data.

6.4 Optimizing Joiner Transformations

Joiner transformations can slow performance because they need additional space at run time to hold intermediary
results. You can view Joiner performance counter information to determine whether you need to optimize the Joiner
transformations.
Use the following tips to improve session performance with the Joiner transformation:
♦Designate the master source as the source with fewer duplicate key values. When the Integration Service
processes a sorted Joiner transformation, it caches rows for one hundred unique keys at a time. If the master source
contains many rows with the same key value, the Integration Service must cache more rows, and performance can
be slowed.
♦Designate the master source as the source with the fewer rows. During a session, the Joiner transformation
compares each row of the detail source against the master source. The fewer rows in the master, the fewer iterations
of the join comparison occur, which speeds the join process.
♦Perform joins in a database when possible. Performing a join in a database is faster than performing a join in the
session. The type of database join you use can affect performance. Normal joins are faster than outer joins and result
in fewer rows. In some cases, you cannot perform the join in the database, such as joining tables from two different
databases or flat file systems.
To perform a join in a database, use the following options:
♦Create a pre-session stored procedure to join the tables in a database.
♦Use the Source Qualifier transformation to perform the join.
♦Join sorted data when possible. You can improve session performance by configuring the Joiner transformation
to use sorted input. When you configure the Joiner transformation to use sorted data, the Integration Service
improves performance by minimizing disk input and output. You see the greatest performance improvement when
you work with large data sets. For an unsorted Joiner transformation, designate the source with fewer rows as the
master source.

Tips and Tricks

Joiner transformation helps to perform joins of two source tables.

1. Sort the data before joining.

2. In joiner transformations, normal joins are faster than outer joins.

3. Instead of joiner transformation, perform joins in database.

4. In joiner transformation, the source with lesser number of records should be the master source.

5. Join on fewer columns as possible.

6. Use source qualifier to perform joins instead of joiner transformation wherever possible.

6.5 Optimizing Lookup Transformations

If the lookup table is on the same database as the source table in your mapping and caching is not feasible, join the
tables in the source database rather than using a Lookup transformation.
If you use a Lookup transformation, perform the following tasks to increase performance:
♦Use the optimal database driver.
♦Cache lookup tables.
♦Optimize the lookup condition.
♦Index the lookup table.
♦Optimize multiple lookups.

Using Optimal Database Drivers

The Integration Service can connect to a lookup table using a native database driver or an ODBC driver. Native
database drivers provide better session performance than ODBC drivers.

Caching Lookup Tables

If a mapping contains Lookup transformations, you might want to enable lookup caching. When you enable caching,
the Integration Service caches the lookup table and queries the lookup cache during the session. When this option is
not enabled, the Integration Service queries the lookup table on a row-by-row basis.
The result of the Lookup query and processing is the same, whether or not you cache the lookup table. However,
using a lookup cache can increase session performance for smaller lookup tables. In general, you want to cache
lookup tables that need less than 300 MB.
Complete the following tasks to further enhance performance for Lookup transformations:
♦Use the appropriate cache type.
♦Enable concurrent caches
♦Optimize Lookup condition matching
♦Reduce the number of cached rows.
♦Override the ORDER BY statement.
♦Use a machine with more memory.

Types of Caches
Use the following types of caches to increase performance:
♦Shared cache. You can share the lookup cache between multiple transformations. You can share an unnamed
cache between transformations in the same mapping. You can share a named cache between transformations in the
same or different mappings.
♦Persistent cache. If you want to save and reuse the cache files, you can configure the transformation to use a
persistent cache. Use this feature when you know the lookup table does not change between session runs. Using a
persistent cache can improve performance because the Integration Service builds the memory cache from the cache
files instead of from the database.

OPTIMUM CACHE SIZE IN LOOKUPS

-Calculating Lookup Index Cache

The lookup index cache holds data for the columns used in the lookup condition. For best session performance,
specify the maximum lookup index cache size. Use the following information to calculate the minimum and
maximum lookup index cache for both connected and unconnected Lookup transformations: -

To calculate the minimum lookup index cache size, use the formula:-

Columns in lookup cache = 200 * [<Column size> + 16]

To calculate the maximum lookup index cache size, use the formula:-

Columns in lookup cache = <Number of rows in lookup table>* [<column size> + 16] * 2

Example:-

Suppose the lookup table has lookup values based in the field ITEM_ID. It uses the lookup condition, ITEM_ID =
IN_ITEM_ID1.

This ITEM_ID has data type as ‘integer’ and size as ‘16’.

Therefore the total column size is 16. The table contains 60000 rows.

Minimum lookup index cache size = 200 * [16 + 16] = 6400

Maximum lookup index cache size = 60000 * [16+16] * 2 = 3,840,000

So this lookup transformation needs an index cache size between 6400 and 3,840,000.

For best session performance, this lookup transformation needs an index cache size of 3,840,000 bytes.

-Calculating Lookup Data Cache

In a connected transformation, the data cache contains data for the connected output ports, not including ports used
in the lookup condition. In an unconnected transformation, the data cache contains data from the return port.

To calculate the minimum lookup data cache size, use the formula:-

Columns in lookup cache = <Number of rows in lookup table> * [<Column size of connected output ports not
in lookup condition > + 8]

Example:-

Suppose the lookup table has column names as PROMOTION_ID and DISCOUNT which are connected output
ports not in lookup condition

Column size of each is 16. Therefore total column size is 32.The table contains 60000 rows.

Minimum lookup data cache size = 60000 * [32 + 8] = 2,400,000

So this lookup transformation needs a data cache size of 2, 40,000 bytes.

Enable Concurrent Caches

When the Integration Service processes sessions that contain Lookup transformations, the Integration Service builds
a cache in memory when it processes the first row of data in a cached Lookup transformation. If there are multiple
Lookup transformations in a mapping, the Integration Service creates the caches sequentially when the first row of
data is processed by the Lookup transformation. This slows Lookup transformation processing.
You can enable concurrent caches to improve performance. When the number of additional concurrent pipelines is
set to one or more, the Integration Service builds caches concurrently rather than sequentially. Performance
improves greatly when the sessions contain a number of active transformations that may take time to complete, such
as Aggregator, Joiner, or Sorter transformations. When you enable multiple concurrent pipelines, the Integration
Service no longer waits for active sessions to complete before it builds the cache. Other Lookup transformations in
the pipeline also build caches concurrently.

Optimize Lookup Condition Matching

When the Lookup transformation matches lookup cache data with the lookup condition, it sorts and orders the data
to determine the first matching value and the last matching value. You can configure the transformation to return
any value that matches the lookup condition. When you configure the Lookup transformation to return any matching
value, the transformation returns the first value that matches the lookup condition. It does not index all ports as it
does when you configure the transformation to return the first matching value or the last matching value. When you
use any matching value, performance can improve because the transformation does not index on all ports, which can
slow performance.

Reducing the Number of Cached Rows

You can reduce the number of rows included in the cache to increase performance. Use the Lookup SQL Override
option to add a WHERE clause to the default SQL statement.

Overriding the ORDER BY Statement

By default, the Integration Service generates an ORDER BY statement for a cached lookup. The ORDER BY
statement contains all lookup ports. To increase performance, you can suppress the default ORDER BY statement
and enter an override ORDER BY with fewer columns.
The Integration Service always generates an ORDER BY statement, even if you enter one in the override. Place two
dashes ‘--’ after the ORDER BY override to suppress the generated ORDER BY statement.
For example, a Lookup transformation uses the following lookup condition:
ITEM_ID = IN_ITEM_ID
PRICE <= IN_PRICE

The Lookup transformation includes three lookup ports used in the mapping, ITEM_ID, ITEM_NAME, and PRICE.
When you enter the ORDER BY statement, enter the columns in the same order as the ports in the lookup condition.
You must also enclose all database reserved words in quotes.
Enter the following lookup query in the lookup SQL override:
SELECT ITEMS_DIM.ITEM_NAME, ITEMS_DIM.PRICE, ITEMS_DIM.ITEM_ID FROM
ITEMS_DIM ORDER BY ITEMS_DIM.ITEM_ID, ITEMS_DIM.PRICE --

Using a Machine with More Memory

You can also increase session performance by running the session on an Integration Service machine with a large
amount of memory. Increase the index and data cache sizes as high as you can without straining the machine. If the
Integration Service machine has enough memory, increase the cache so it can hold all data in memory without
paging to disk.

Optimizing the Lookup Condition

If you include more than one lookup condition, place the conditions with an equal sign first to optimize lookup
performance.

Indexing the Lookup Table

The Integration Service needs to query, sort, and compare values in the lookup condition columns. The index needs
to include every column used in a lookup condition.
You can improve performance for the following types of lookups:

♦Cached lookups. To improve performance, index the columns in the lookup ORDER BY statement. The session
log contains the ORDER BY statement.
♦Uncached lookups. To improve performance, index the columns in the lookup condition. The Integration Service
issues a SELECT statement for each row that passes into the Lookup transformation.

Optimizing Multiple Lookups

If a mapping contains multiple lookups, even with caching enabled and enough heap memory, the lookups can slow
performance. Tune the Lookup transformations that query the largest amounts of data to improve overall
performance.
To determine which Lookup transformations process the most data, examine the Lookup_rowsinlookupcache
counters for each Lookup transformation. The Lookup transformations that have a large number in this counter
might benefit from tuning their lookup expressions. If those expressions can be optimized, session performance
improves.

Tips and Tricks

Lookup transformations are used to lookup a set of values in another table. Lookups slows down the performance.

1. To improve performance, cache the lookup tables. Informatica can cache all the lookup and reference
tables; this makes operations run very fast. (Meaning of cache is given in point 2 of this section and the
procedure for determining the optimum cache size is given at the end of this document.)

2. Even after caching, the performance can be further improved by minimizing the size of the lookup cache.
Reduce the number of cached rows by using a sql override with a restriction.
Cache: Cache stores data in memory so that Informatica does not have to read the table each time it is
referenced. This reduces the time taken by the process to a large extent. Cache is automatically generated
by Informatica depending on the marked lookup ports or by a user defined sql query.

Example for caching by a user defined query: -

Suppose we need to lookup records where employee_id=eno.

‘employee_id’ is from the lookup table, EMPLOYEE_TABLE and ‘eno’ is the input that comes from the
from the source table, SUPPORT_TABLE.

We put the following sql query override in Lookup Transform

‘select employee_id from EMPLOYEE_TABLE’

If there are 50,000 employee_id, then size of the lookup cache will be 50,000.

Instead of the above query, we put the following:-

‘select emp employee_id from EMPLOYEE_TABLE e, SUPPORT_TABLE s where e.

employee_id=s.eno’

If there are 1000 eno, then the size of the lookup cache will be only 1000.
 But here the performance gain will happen only if the number of records in SUPPORT_TABLE is not
huge. Our concern is to make the size of the cache as less as possible.

3. In lookup tables, delete all unused columns and keep only the fields that are used in the mapping.
4. If possible, replace lookups by joiner transformation or single source qualifier. Joiner transformation takes
more time than source qualifier transformation.
5. If lookup transformation specifies several conditions, then place conditions that use equality operator ‘=’
first in the conditions that appear in the conditions tab.
6. In the sql override query of the lookup table, there will be an ORDER BY clause. Remove it if not needed
or put fewer column names in the ORDER BY list.
7. Do not use caching in the following cases: -
-Source is small and lookup table is large.

-If lookup is done on the primary key of the lookup table.

8. Cache the lookup table columns definitely in the following case: -

-If lookup table is small and source is large.

9. If lookup data is static, use persistent cache. Persistent caches help to save and reuse cache files. If several
sessions in the same job use the same lookup table, then using persistent cache will help the sessions to
reuse cache files. In case of static lookups, cache files will be built from memory cache instead of from the
database, which will improve the performance.
10. If source is huge and lookup table is also huge, then also use persistent cache.
11. If target table is the lookup table, then use dynamic cache. The Informatica server updates the lookup cache
as it passes rows to the target.
12. Use only the lookups you want in the mapping. Too many lookups inside a mapping will slow down the
session.
13. If lookup table has a lot of data, then it will take too long to cache or fit in memory. So move those fields to
source qualifier and then join with the main table.
14. If there are several lookups with the same data set, then share the caches.
15. If we are going to return only 1 row, then use unconnected lookup.
16. All data are read into cache in the order the fields are listed in lookup ports. If we have an index that is even
partially in this order, the loading of these lookups can be speeded up.
17. If the table that we use for look up has an index (or if we have privilege to add index to the table in the
database, do so), then the performance would increase both for cached and uncached lookups.

6.6 Optimizing Sequence Generator Transformations

You can optimize Sequence Generator transformations by creating a reusable Sequence Generator and using it in
multiple mappings simultaneously. You can also optimize Sequence Generator transformations by configuring the
Number of Cached Values property.
The Number of Cached Values property determines the number of values the Integration Service caches at one time.
Make sure that the Number of Cached Value is not too small. You may consider configuring the Number of Cached
Values to a value greater than 1,000.
If you do not have to cache values, set the Number of Cache Values to 0. Sequence Generator transformations that
do not use cache are faster than those that require cache.

Tips and Tricks

A sequence generator transformation is used to generate primary keys in the Informatica.

1. If we need the sequence generator more than once in a job, then make it reusable & use multiple times in
the folder.
 2. To generate primary keys, use Sequence generator transformation instead of using a stored procedure for
generating sequence numbers.
3. We can also opt for sequencing in the source qualifier by adding a dummy field in the source definition and
source qualifier, and then giving a sql query like
‘select seq_name.nextval, <other column names>... from <source table name> where <condition if
any>’.

Seq_name is the sequence that generates primary key for our source table. <Sequence name>. Nextval is a
sequence generator object in Oracle.

This method of primary key generation is faster than using sequence generator transformation.

6.7 Optimizing Sorter Transformations

Complete the following tasks to optimize a Sorter transformation:
♦Allocate enough memory to sort the data.
♦Specify a different work directory for each partition in the Sorter transformation.
Allocating Memory
If the Integration Service cannot allocate enough memory to sort data, it fails the session. For best performance,
configure Sorter cache size with a value less than or equal to the amount of available physical RAM on the
Integration Service machine. Informatica recommends allocating at least 8 MB (8,388,608 bytes) of physical
memory to sort data using the Sorter transformation. Sorter cache size is set to 8,388,608 bytes by default.
If the amount of incoming data is greater than the amount of Sorter cache size, the Integration Service temporarily
stores data in the Sorter transformation work directory. The Integration Service requires disk space of at least twice
the amount of incoming data when storing data in the work directory. If the amount of incoming data is significantly
greater than the Sorter cache size, the Integration Service may require much more than twice the amount of disk
space available to the work directory.
Use the following formula to determine the size of incoming data:
# input rows ([Sum (column size)] + 16)

Work Directories for Partitions

The Integration Service creates temporary files when it sorts data. It stores them in a work directory. You can
specify any directory on the Integration Service machine to use as a work directory. By default, the Integration
Service uses the value specified for the $PMTempDir server variable.
When you partition a session with a Sorter transformation, you can specify a different work directory for each
partition in the pipeline. To increase session performance, specify work directories on physically separate disks on
the Integration Service nodes.

Tips and Tricks

Sorter transformation is used to sort the input data.

1. While using the sorter transformation, configure sorter cache size to be larger than the input data size.
2. Configure the sorter cache size setting to be larger than the input data size while using sorter
transformation.
3. At the sorter transformation, use hash auto keys partitioning or hash user keys partitioning.
6.8 Optimizing Source Qualifier Transformations
Use the Select Distinct option for the Source Qualifier transformation if you want the Integration Service to select
unique values from a source. Use Select Distinct option to filter unnecessary data earlier in the data flow. This can
improve performance.

6.9 Optimizing SQL Transformations

When you create an SQL transformation, you configure the transformation to use external SQL queries or queries
that you define in the transformation. When you configure an SQL transformation to run in script mode, the
Integration Service processes an external SQL script for each input row. When the transformation runs in query
mode, the Integration Service processes an SQL query that you define in the transformation.
Each time the Integration Service processes a new query in a session, it calls a function called SQLPrepare to create
an SQL procedure and pass it to the database. When the query changes for each input row, it has a performance
impact.
When the transformation runs in query mode, you can improve performance by constructing a static query in the
transformation. A static query statement does not change, although the data in the query clause changes. To create a
static query, use parameter binding instead of string substitution in the SQL Editor. When you use parameter binding
you set parameters in the query clause to values in the transformation input ports.
When an SQL query contains commit and rollback query statements, the Integration Service must recreate the SQL
procedure after each commit or rollback. To optimize performance, do not use transaction statements in an SQL
transformation query.
When you create the SQL transformation, you configure how the transformation connects to the database. You can
choose a static connection or you can pass connection information to the transformation at run time.
When you configure the transformation to use a static connection, you choose a connection from the Workflow
Manager connections. The SQL transformation connects to the database once during the session. When you pass
dynamic connection information, the SQL transformation connects to the database each time the transformation
processes an input row.

6.10 Optimizing Update Transformation

Sorter transformation is used to sort the input data.

1. While using the sorter transformation, configure sorter cache size to be larger than the input data size.
2. Configure the sorter cache size setting to be larger than the input data size while using sorter
transformation.
3. At the sorter transformation, use hash auto keys partitioning or hash user keys partitioning.

6.11 Optimizing Filter Transformation

Filter transformation is used to filter off unwanted fields based on conditions we specify.

1. Use filter transformation as close to source as possible so that unwanted data gets eliminated sooner.
2. If elimination of unwanted data can be done by source qualifier instead of filter, then eliminate them using
former.
3. Use conditional filters and keep the filter condition simple, involving TRUE/FALSE or 1/0.
6.12 Optimizing Expression Transformation
Expression transformation is used to perform simple calculations and also to do source lookups.

1. Use operators instead of functions.

2. Minimize the usage of string functions.
3. If we use a complex expression multiple times in the expression transformer, then make that expression as
a variable. Then we need to use only this variable for all computations.

7.Optimizing Sessions
7.1 Overview
Once you optimize the source database, target database, and mapping, you can focus on optimizing the session. You
can perform the following tasks to improve overall performance:
♦Use a grid. You can increase performance by using a grid to balance the Integration Service workload.
♦Use pushdown optimization. You can increase session performance by pushing transformation logic to the source
or target database.
♦Run sessions and workflows concurrently. You can run independent sessions and workflows concurrently to
improve session and workflow performance.
♦Allocate buffer memory. You can increase the buffer memory allocation for sources and targets that require
additional memory blocks. If the Integration Service cannot allocate enough memory blocks to hold the data, it fails
the session.
♦Optimize caches. You can improve session performance by setting the optimal location and size for the caches.
♦Increase the commit interval. Each time the Integration Service commits changes to the target, performance
slows. You can increase session performance by increasing the interval at which the Integration Service commits
changes.
♦Disable high precision. Performance slows when the Integration Service reads and manipulates data with the high
precision datatype. You can disable high precision to improve session performance.
♦Reduce errors tracing. To improve performance, you can reduce the error tracing level, which reduces the
number of log events generated by the Integration Service..
♦Remove staging areas. When you use a staging area, the Integration Service performs multiple passes on the data.
You can eliminate staging areas to improve session performance.

7.2 Using a Grid

You can use a grid to increase session and workflow performance. A grid is an alias assigned to a group of nodes
that allows you to automate the distribution of workflows and sessions across nodes.
When you use a grid, the Integration Service distributes workflow tasks and session threads across multiple nodes.
Running workflows and sessions on the nodes of a grid provides the following performance gains:
♦Balances the Integration Service workload.
♦Processes concurrent sessions faster.
♦Processes partitions faster.

7.3 Using Pushdown Optimization

You can increase session performance by pushing transformation logic to the source or target database. Based on the
mapping and session configuration, the Integration Service executes SQL against the source or target database
instead of processing the transformation logic within the Integration Service.

7.4 Run Concurrent Sessions and Workflows

If possible, run sessions and workflows concurrently to improve performance. For example, if you load data into an
analytic schema, where you have dimension and fact tables, load the dimensions concurrently.

7.5 Allocating Buffer Memory

When the Integration Service initializes a session, it allocates blocks of memory to hold source and target data. The
Integration Service allocates at least two blocks for each source and target partition. Sessions that use a large number
of sources and targets might require additional memory blocks. If the Integration Service cannot allocate enough
memory blocks to hold the data, it fails the session.
You can configure the amount of buffer memory, or you can configure the Integration Service to automatically
calculate buffer settings at run time.

You can increase the number of available memory blocks by adjusting the following session parameters:

♦DTM Buffer Size. Increase the DTM buffer size on the Properties tab in the session properties.
♦Default Buffer Block Size. Decrease the buffer block size on the Config Object tab in the session properties.
To configure these settings, first determine the number of memory blocks the Integration Service requires to
initialize the session. Then, based on default settings, calculate the buffer size and/or the buffer block size to create
the required number of session blocks.
If you have XML sources or targets in a mapping, use the number of groups in the XML source or target in the
calculation for the total number of sources and targets.
For example, you create a session that contains a single partition using a mapping that contains 50 sources and 50
targets. Then you make the following calculations:
1.You determine that the session requires a minimum of 200 memory blocks:

[(total number of sources + total number of targets)* 2] = (session buffer blocks)

100 * 2 = 200

2.Based on default settings, you determine that you can change the DTM Buffer Size to 15,000,000, or you can
change the Default Buffer Block Size to 54,000:
(session Buffer Blocks) = (.9) * (DTM Buffer Size) / (Default Buffer Block Size) * (number of partitions)
200 = .9 * 14222222 / 64000 * 1

or
200 = .9 * 12000000 / 54000 * 1

Note: For a session that contains n partitions, set the DTM Buffer Size to at least n times the value for the session
with one partition. The Log Manager writes a warning message in the session log if the number of memory blocks is
so small that it causes performance degradation. The Log Manager writes this warning message even if the number
of memory blocks is enough for the session to run successfully. The warning message also gives a suggestion for the
proper value.
If you modify the DTM Buffer Size, increase the property by multiples of the buffer block size.

Increasing DTM Buffer Size

The DTM Buffer Size setting specifies the amount of memory the Integration Service uses as DTM buffer memory.
The Integration Service uses DTM buffer memory to create the internal data structures and buffer blocks used to
bring data into and out of the Integration Service. When you increase the DTM buffer memory, the Integration
Service creates more buffer blocks, which improves performance during momentary slowdowns.
Increasing DTM buffer memory allocation generally causes performance to improve initially and then level off.
When you increase the DTM buffer memory allocation, consider the total memory available on the Integration
Service process system.
If you do not see a significant increase in performance, DTM buffer memory allocation is not a factor in session
performance.
Note: Reducing the DTM buffer allocation can cause the session to fail early in the process because the Integration
Service is unable to allocate memory to the required processes.
To increase the DTM buffer size, open the session properties and click the Properties tab. Edit the DTM Buffer Size
property in the Performance settings.
Increase the property by multiples of the buffer block size, and then run and time the session after each increase.

Optimizing the Buffer Block Size

Depending on the session source data, you might need to increase or decrease the buffer block size.
If the machine has limited physical memory and the mapping in the session contains a large number of sources,
targets, or partitions, you might need to decrease the buffer block size.
If you are manipulating unusually large rows of data, you can increase the buffer block size to improve performance.
If you do not know the approximate size of the rows, you can determine the configured row size by completing the
following steps.
To evaluate needed buffer block size:
1. In the Mapping Designer, open the mapping for the session.
2. Open the target instance.
3. Click the Ports tab.
4. Add the precision for all columns in the target.
5. If you have more than one target in the mapping, repeat steps 2 to 4 for each additional target to calculate the
precision for each target.
6. Repeat steps 2 to 5 for each source definition in the mapping.
7. Choose the largest precision of all the source and target precisions for the total precision in the buffer block size
calculation.
The total precision represents the total bytes needed to move the largest row of data. For example, if the total
precision equals 33,000, then the Integration Service requires 33,000 bytes in the buffers to move that row. If the
buffer block size is 64,000 bytes, the Integration Service can move only one row at a time.
Ideally, a buffer accommodates at least 100 rows at a time. So if the total precision is greater than 32,000, increase
the size of the buffers to improve performance.
To increase the buffer block size, open the session properties and click the Config Object tab. Edit the Default
Buffer Block Size property in the Advanced settings.
Increase the DTM buffer block setting in relation to the size of the rows. As with DTM buffer memory allocation,
increasing buffer block size should improve performance. If you do not see an increase, buffer block size is not a
factor in session performance.

7.6 Optimizing Caches

The Integration Service uses the index and data caches for XML targets and Aggregator, Rank, Lookup, and Joiner
transformations. The Integration Service stores transformed data in the data cache before returning it to the pipeline.
It stores group information in the index cache. Also, the Integration Service uses a cache to store data for Sorter
transformations.
You can configure the amount of cache memory using the cache calculator or by specifying the cache size. You can
also configure the Integration Service to automatically calculate cache memory settings at run time.
If the allocated cache is not large enough to store the data, the Integration Service stores the data in a temporary disk
file as it processes the session data. Performance slows each time the Integration Service pages to a temporary file.
Examine the performance details to determine how often the Integration Service pages to a file.
Perform the following tasks to optimize caches:
♦Limit the number of connected input/output and output only ports.
♦Select the optimal cache directory location.
♦Increase the cache sizes.
♦Use the 64-bit version of PowerCenter to run large cache sessions.

Limiting the Number of Connected Ports

For transformations that use data cache, limit the number of connected input/output and output only ports. Limiting
the number of connected input/output or output ports reduces the amount of data the transformations store in the
data cache.

Cache Directory Location

If you run the Integration Service on a grid and only some Integration Service nodes have fast access to the shared
cache file directory, configure each session with a large cache to run on the nodes with fast access to the directory.
To configure a session to run on a node with fast access to the directory, complete the following steps:
1.Create a PowerCenter resource.
2. Make the resource available to the nodes with fast access to the directory.
3. Assign the resource to the session.
If all Integration Service processes in a grid have slow access to the cache files, set up a separate, local cache file
directory for each Integration Service process. An Integration Service process may have faster access to the cache
files if it runs on the same machine that contains the cache directory.

Note: You may encounter performance degradation when you cache large quantities of data on a mapped or
mounted drive.

Increasing the Cache Sizes

If the allocated cache is not large enough to store the data, the Integration Service stores the data in a temporary disk
file as it processes the session data. Each time the Integration Service pages to the temporary file, performance
slows.
You can examine the performance details to determine when the Integration Service pages to the temporary file. The
Transformation_readfromdisk or Transformation_writetodisk counters for any Aggregator, Rank, Lookup, or Joiner
transformation indicate the number of times the Integration Service must page to disk to process the transformation.
Since the data cache is typically larger than the index cache, increase the data cache more than the index cache.
If the session contains a transformation that uses a cache and you run the session on a machine with ample memory,
increase the cache sizes so all data can fit in memory.

Using the 64-bit version of PowerCenter

If you process large volumes of data or perform memory-intensive transformations, you can use the 64-bit
PowerCenter version to increase session performance. The 64-bit version provides a larger memory space that can
significantly reduce or eliminate disk input/output.
This can improve session performance in the following areas:
♦Caching. With a 64-bit platform, the Integration Service is not limited to the 2 GB cache limit of a 32-bit platform.
♦Data throughput. With a larger available memory space, the reader, writer, and DTM threads can process larger
blocks of data.

7.7 Increasing the Commit Interval

The commit interval setting determines the point at which the Integration Service commits data to the targets. Each
time the Integration Service commits, performance slows. Therefore, the smaller the commit interval, the more often
the Integration Service writes to the target database and the slower the overall performance.
If you increase the commit interval, the number of times the Integration Service commits decreases and performance
improves.
When you increase the commit interval, consider the log file limits in the target database. If the commit interval is
too high, the Integration Service may fill the database log file and cause the session to fail.
Therefore, weigh the benefit of increasing the commit interval against the additional time you would spend
recovering a failed session.
Click the General Options settings in the session properties to review and adjust the commit interval.

7.8 Disabling High Precision

If a session runs with high precision enabled, disabling high precision might improve session performance.
The Decimal datatype is a numeric datatype with a maximum precision of 28. To use a high precision Decimal
datatype in a session, configure the Integration Service to recognize this datatype by selecting Enable High Precision
in the session properties. However, since reading and manipulating the high precision datatype slows the Integration
Service, you can improve session performance by disabling high precision.
When you disable high precision, the Integration Service converts data to a double. The Integration Service reads the
Decimal row 3900058411382035317455530282 as 390005841138203 x 1013.
Click the Performance settings in the session properties to enable high precision.

7.9 Reducing Error Tracing

To improve performance, you can reduce the number of log events generated by the Integration Service when it runs
the session. If a session contains a large number of transformation errors, and you do not need to correct them, set
the session tracing level to Terse. At this tracing level, the Integration Service does not write error messages or row-
level information for reject data.
If you need to debug the mapping and you set the tracing level to Verbose, you may experience significant
performance degradation when you run the session. Do not use Verbose tracing when you tune performance.
The session tracing level overrides any transformation-specific tracing levels within the mapping. This is not
recommended as a long-term response to high levels of transformation errors.

7.10 Removing Staging Areas

When you use a staging area, the Integration Service performs multiple passes on the data. When possible, remove
staging areas to improve performance. The Integration Service can read multiple sources with a single pass, which
may alleviate the need for staging areas.
7.11 Tips and Tricks
A session specifies the location from where the data is to be taken, where the transformations are done and where
the data is to be loaded. It has various properties that help us to schedule and run the job in the way we want.

1. Partition the session: This creates many connections to the source and target, and loads data in parallel
pipelines. Each pipeline will be independent of the other. But the performance of the session will not
improve if the number of records is less. Also the performance will not improve if it does updates and
deletes. So session partitioning should be used only if the volume of data is huge and the job is mainly
insertion of data.

2. Run the sessions in parallel rather than serial to gain time, if they are independent of each other.

3. Drop constraints and indexes before we run session. Rebuild them after the session run completes.
Dropping can be done in pre session script and Rebuilding in post session script. But if data is too much,
dropping indexes and then rebuilding them etc. will be not possible. In such cases, stage all data, pre-create
the index, use a transportable table space and then load into database.

4. Use bulk loading, external loading etc. Bulk loading can be used only if the table does not have an index.

5. In a session we have options to ‘Treat rows as ‘Data Driven, Insert, Update and Delete’. If update strategies
are used, then we have to keep it as ‘Data Driven’. But when the session does only insertion of rows into
target table, it has to be kept as ‘Insert’ to improve performance.

6. Increase the database commit level (The point at which the Informatica server is set to commit data to the
target table. For e.g. commit level can be set at every every 50,000 records)

7. By avoiding built in functions as much as possible, we can improve the performance. E.g. For
concatenation, the operator ‘||’ is faster than the function CONCAT (). So use operators instead of
functions, where possible. The functions like IS_SPACES (), IS_NUMBER (), IFF (), DECODE () etc.
reduce the performance to a big extent in this order. Preference should be in the opposite order.

8. String functions like substring, ltrim, and rtrim reduce the performance. In the sources, use delimited
strings in case the source flat files or use varchar data type.

9. Manipulating high precision data types will slow down Informatica server. So disable ‘high precision’.

10. Localize all source and target tables, stored procedures, views, sequences etc. Try not to connect across
synonyms. Synonyms and aliases slow down the performance.
8.Optimizing the System
8.1 Overview
Often performance slows because the session relies on inefficient connections or an overloaded Integration Service
process system. System delays can also be caused by routers, switches, network protocols, and usage by many users.
Slow disk access on source and target databases, source and target file systems, and nodes in the domain can slow
session performance. Have the system administrator evaluate the hard disks on the machines.
After you determine from the system monitoring tools that you have a system bottleneck, make the following global
changes to improve the performance of all sessions:
♦Improve network speed. Slow network connections can slow session performance. Have the system administrator
determine if the network runs at an optimal speed. Decrease the number of network hops between the Integration
Service process and databases.
♦Use multiple CPUs. You can use multiple CPUs to run multiple sessions in parallel and run multiple pipeline
partitions in parallel.
♦Reduce paging. When an operating system runs out of physical memory, it starts paging to disk to free physical
memory. Configure the physical memory for the Integration Service process machine to minimize paging to disk.
♦Use processor binding. In a multi-processor UNIX environment, the Integration Service may use a large amount
of system resources. Use processor binding to control processor usage by the Integration Service process. Also, if
the source and target database are on the same machine, use processor binding to limit the resources used by the
database.

8.2 Improving Network Speed

The performance of the Integration Service is related to network connections. A local disk can move data 5 to 20
times faster than a network. Consider the following options to minimize network activity and to improve Integration
Service performance.
If you use flat file as a source or target in a session and the Integration Service runs on a single node, store the files
on the same machine as the Integration Service to improve performance. When you store flat files on a machine
other than the Integration Service, session performance becomes dependent on the performance of the network
connections. Moving the files onto the Integration Service process system and adding disk space might improve
performance.
If you use relational source or target databases, try to minimize the number of network hops between the source and
target databases and the Integration Service process. Moving the target database onto a server system might improve
Integration Service performance.
When you run sessions that contain multiple partitions, have the network administrator analyze the network and
make sure it has enough bandwidth to handle the data moving across the network from all partitions.

8.3 Using Multiple CPUs

Configure the system to use more CPUs to improve performance. Multiple CPUs allow the system to run multiple
sessions in parallel as well as multiple pipeline partitions in parallel.
However, additional CPUs might cause disk bottlenecks. To prevent disk bottlenecks, minimize the number of
processes accessing the disk. Processes that access the disk include database functions and operating system
functions. Parallel sessions or pipeline partitions also require disk access.

8.4 Reducing Paging

Paging occurs when the Integration Service process operating system runs out of memory for a particular operation
and uses the local disk for memory. You can free up more memory or increase physical memory to reduce paging
and the slow performance that results from paging. Monitor paging activity using system tools.
You might want to increase system memory in the following circumstances:
♦You run a session that uses large cached lookups.
♦You run a session with many partitions.
If you cannot free up memory, you might want to add memory to the system.

8.5 Using Processor Binding

In a multi-processor UNIX environment, the Integration Service may use a large amount of system resources if you
run a large number of sessions. As a result, other applications on the machine may not have enough system
resources available. You can use processor binding to control processor usage by the Integration Service process
node. Also, if the source and target database are on the same machine, use processor binding to limit the resources
used by the database.
In a Sun Solaris environment, the system administrator can create and manage a processor set using the psrset
command. The system administrator can then use the pbind command to bind the Integration Service to a processor
set so the processor set only runs the Integration Service. The Sun Solaris environment also provides the psrinfo
command to display details about each configured processor and the psradm command to change the operational
status of processors. For more information, see the system administrator and Sun Solaris documentation.
In an HP-UX environment, the system administrator can use the Process Resource Manager utility to control CPU
usage in the system. The Process Resource Manager allocates minimum system resources and uses a maximum cap
of resources.
In an AIX environment, system administrators can use the Workload Manager in AIX 5L to manage system
resources during peak demands. The Workload Manager can allocate resources and manage CPU, memory, and disk
I/O bandwidth.

9.Optimizing Database
9.1 Tips and Tricks
To gain the best Informatica performance, the database tables, stored procedures and queries used in Informatica
should be tuned well.

1. If the source and target are flat files, then they should be present in the system in which the Informatica
server is present.

2. Increase the network packet size.

3. The performance of the Informatica server is related to network connections.

Data generally moves across a network at less than 1 MB per second, whereas a local disk moves data five
to twenty times faster. Thus network connections often affect on session performance. So avoid network
connections.

4. Optimize target databases.

10.Optimizing the PowerCenter Components
10.1 Overview
You can optimize performance of the following PowerCenter components:
♦PowerCenter repository
♦Integration Service
If you run PowerCenter on multiple machines, run the Repository Service and Integration Service on different
machines. To load large amounts of data, run the Integration Service on the higher processing machine. Also, run
the Repository Service on the machine hosting the PowerCenter repository.

10.2 Optimizing PowerCenter Repository Performance

Complete the following tasks to improve PowerCenter repository performance:
♦Ensure the PowerCenter repository is on the same machine as the Repository Service process.
♦Order conditions in object queries.
♦Use a single-node tablespace for the PowerCenter repository if you install it on a DB2 database.

Location of the Repository Service Process and Repository

You can optimize the performance of a Repository Service that you configured without the high availability option.
To optimize performance, ensure that the Repository Service process runs on the same machine where the repository
database resides.

Ordering Conditions in Object Queries

When the Repository Service processes a parameter with multiple conditions, it processes them in the order you
enter them. To receive expected results and improve performance, enter parameters in the order you want them to
run.

Using a Single-Node DB2 Database Tablespace

You can optimize repository performance on IBM DB2 EEE databases when you store a PowerCenter repository in
a single-node tablespace. When setting up an IBM DB2 EEE database, the database administrator can define the
database on a single node.
When the tablespace contains one node, the PowerCenter Client and Integration Service access the repository faster
than if the repository tables exist on different database nodes.
If you do not specify the tablespace name when you create, copy, or restore a repository, the DB2 system specifies
the default tablespace for each repository table. The DB2 system may or may not specify a single-node tablespace.

10.3 Optimizing Integration Service Performance

Complete the following tasks to improve Integration Service performance:
♦Use native drivers instead of ODBC drivers for the Integration Service.
♦Run the Integration Service in ASCII data movement mode if character data is 7-bit ASCII or EBCDIC.
♦Cache PowerCenter metadata for the Repository Service.
♦Run Integration Service with high availability.
Note: When you configure the Integration Service with high availability, the Integration Service recovers workflows
and sessions that may fail because of temporary network or machine failures. To recover from a workflow or
session, the Integration Service writes the states of each workflow and session to temporary files in a shared
directory. This may decrease performance.

Using Native and ODBC Drivers

The Integration Service can use ODBC or native drivers to connect to databases. Use native drivers to improve
performance.

Running the Integration Service in ASCII Data Movement Mode

When all character data processed by the Integration Service is 7-bit ASCII or EBCDIC, configure the Integration
Service to run in the ASCII data movement mode. In ASCII mode, the Integration Service uses one byte to store
each character. When you run the Integration Service in Unicode mode, it uses two bytes for each character, which
can slow session performance.

Caching PowerCenter Metadata for the Repository Service

You can use repository agent caching to improve DTM process performance. When you enable repository agent
caching, the Repository Service caches metadata requested by the Integration Service. When you cache metadata,
the Integration Service reads the cache for subsequent runs of the task rather than fetching the metadata from the
repository. Only metadata requested by the Integration Service is cached.
For example, you run a workflow with 1,000 sessions. The first time you run a workflow with caching enabled, the
Integration Service fetches the session metadata from the repository. During subsequent runs of the workflow, the
Repository Service fetches the session metadata from the cache. This increases DTM process performance.