Best Practices for Performance for Concurrent Managers in E-Business Suite

This Document contains 5 topics

1. Generic Tips

2. Transaction Manager (TM).

3. Parallel Concurrent Processing (PCP) Environment.

4. Tuning Output Post Processor (OPP).

5. Concurrent Processing Server Tuning.

Generic Tips

1) Sleep Seconds - is the number of seconds your Concurrent manager waits between
checking the list of pending concurrent requests (concurrent requests waiting to be
started). A manager only sleeps if there are no runnable jobs in the queue.
Tip: During peak time, when the number of requests submitted is expected to be
high, Set the sleep time to a reasonable wait time(e.g. 30 seconds) dependent on
the average run time and to prevent backlog. Otherwise set the sleep time to a high
number (e.g. 2 minutes). This avoids constant polls to check for new requests.

2) Increase the cache size (number of requests cached) to at least twice the number
of target processes.
For example, if a manager's work shift has 1 target process and a cache value of 3,
it will read three requests, and try to run those three requests before reading
any new requests

Tip: Enter a value of 1 when defining a manager that runs long, time-consuming
jobs, and a value of 3 or 4 for managers that run small, quick jobs.
This is only guidance and a balance needs to struck in tuning the cache, so with
fast jobs you need to cache to get enough work for a few minutes. With slow jobs, a
small queue helps should you need to reprioritize requests.
4) To maximize throughput consider reducing the sleep time of the Conflict
Resolution Manager (CRM). The default value is 60 seconds. You can consider
setting to 5 or 10 seconds.

5) Avoid enabling an excessive number of standard or specialized managers. It can

degrade the performance due polling on queue tables (FND_CONCURRENT_REQUESTS...).
You need to create specialized managers only if there is a real need.

6) Set the system profile option "Concurrent: Force Local Output File Mode" to
"Yes" if required . You need to apply Patch 7530490 for R12 (or) Patch 7834670 for
11i to get this profile.

Refer to Note 822368.1 - 'Purge Concurrent Request FNDCPPUR Does Not Delete Files
From File System or Slow performance'.

Note:- The profile option "Concurrent: Force Local Output File Mode" is set to "No"
by default. After applying the patch, set the profile option to YES will cause
FNDCPPUR to always access files on the local file system, hence FNDCPPUR will
remove the OS files faster.To enable this feature, All Concurrent Manager nodes
must be able to access the output file location via the local filesystem.

7) Truncate the reports.log file in log directory. Refer to Note 844976.1 for
more details.
Truncation of file "reports.log" is a regular maintenance work of Application DBA.
Make sure that reports log file size should not increase to its maximum limit of 2
GB. There is no purge program to truncate file "reports.log". This maintenance
needs to be done manually and regularly depending on number of concurrent program
which uses "reports.log". You can safely truncate "reports.log".

The "reports.log" file can be located under $APPLCSF/$APPLLOG.

8) Ensure "Purge Concurrent Request and/or Manager Data, FNDCPPUR," is run at

regular intervals with "Entity" parameter as "ALL". A high number of records in
FND_CONCURRENT tables can degrade the performance.

Additionally, the following are very good methods to follow for optimizing the
process:
Run the job in hours with low workload. Doing this after hours will lessen the
contention on the tables from running against your daily processing.
To get the requests under control, run the FNDCPPUR program with Age=20 or Age=18
would be a good method. That means, all requests older than 18 or 20 days will be
purged.
Once the requests are under control, run the FNDCPPUR program with Age=7 to
maintain an efficient process. This would solely depend on the level of processing
that is performed at your site
9) Ensure that the log/out files are removed from the locations shown below as you
run "Purge Concurrent Request and/or Manager Data program".
$APPLCSF/$APPLLOG
$APPLCSF/$APPLOUT
In the event that it does not remove the log/out files, over a period of time it
will slow down the performance. Please refer to the following note which suggests
the patch which fixes it.

Note 822368.1 - 'Purge Concurrent Request FNDCPPUR Does Not Delete Files From File
System or Slow performance'.
Note 1616827.1 Managing Concurrent Manager Log and Out Directories
10) Defragment the tables periodically to reclaim unused space / improve
performance.
FND_CONCURRENT_REQUESTS
FND_CONCURRENT_PROCESSES
FND_CRM_HISTORY
FND_ENV_CONTEXT
FND_TEMP_FILES
FND_CONFLICTS_DOMAIN

HOW TO DEFRAGMENT

10.1) alter table <owner>.<table_name> move;

10.2)Note that, some indexes might become unusable after table is moved, check the
index status from dba_indexes for the table moved and rebuild them too as explained
in next bullet.
select owner, index_name, status from dba_indexes
where table_owner = upper('&OWNER') and
table_name = upper('&SEGMENT_NAME');

10.3) alter index <owner>.<index_name> rebuild online;

Note:
Ensure Concurrent Manager is down before you defragment.

Ensure the tablespace in which the object currently exists has got sufficient space
before you move/defragment.

Always take backup of the tables before moving the data. It is recommended to
perform ths action on Test instance initially then test it thoroughly before
performing it on Production instance.

10.4) You will need to re-collect the statistics for the tables.

For example:
exec fnd_stats.gather_table_stats
('APPLSYS','FND_CONCURRENT_REQUESTS',PERCENT=>99);
11) Ensure you upgraded to the latest code to avoid below known Performance ,
Availablity & Deadlock related issues issues.
Performance

Note 1615981.1 - Java Concurrent Managers Are Running Slower In Newer Servers Due
To Lack Of Entropy In The System
Note 1075684.1 - 'Concurrent Managers are consuming high CPU and memory'
Note 1492893.1 - 'R12: Performance Issue When Standard Managers Waiting for "enq:
TX - row lock contention" Held By ICM'
Note 1360118.1 - 'Performance: Concurrent Requests Hang in Pending Status For Long
Time'
Note 1541526.1 - 'Performance: Concurrent Requests Hang in Pending Standby Status
For Long Time'
Patch 10065439 When multiple JVMs run as in Concurrent Program can affect the
performance
Dead Lock

Note 1060736.1 - 'Deadlock Error During Concurrent Request Termination'

Note 866298.1 - 'Concurrent Processing - ORA-00060: Deadlock Detected - UPDATE
FND_CONCURRENT_QUEUES'
Availablity

Note 1604300.1 Concurrent Manager FNDCRM Down / Crashed All Scheduled Concurrent
Programs Are Stuck Pending/Scheduled Pending/Standby
Note 1577982.1 Concurrent Manager FNDSM intermittently Crashes / Shutting Down
Abnormally When a Concurrent Request is Cancelled
Note 1567057.1 Request Submitted By Custom Responsibility Associated With Custom
Data Group Causes FNDLIBR To Coredump
Note 1506643.1 Concurrent Manager Crashes FNDLIBR or INVLIBR Core dumps
Continuously and get Terminated
Note 1601174.1 Concurrent Manager Standard Manager Crashes / Terminated ; FNDLIBR
Segfault In Operating System Log
Note 1457414.1 Faulting application FNDLIBR.exe, version 0.0.0.0, faulting module
oranls10.dll
Note 1542216.1 Concurrent Requests fail Due to FNDSM Log File Size Grows 2 GB ;
SQL*Loader-101: Invalid Argument for username/password
Note 1413393.1 12.1.3 PO Document Approval Manager (POXCON), Receiving Transaction
Manager (RCVOLTM) and INV Remote Procedure Manager (INCTM) Do Not Start / Die After
Restart
Latest Performance Patches (RDBMS)

RDBMS Patches Patch 20355502 (OR) Patch 22521733 should be applied based on the
RDBMS version to resolve issues with fnd_concurrent_worker_requests view and/or
screen FNDCPVCM Administer Concurrent Manager
Reference Note 2106106.1 - 12.2 E-Business Suite Conflict Resolution Concurrent
Manager Not Picking Up Requests Due To FND_CONCURRENT_CRM_REQUESTS View Performance
Issue (Doc ID 2106106.1)
Transaction Manager(TM)
12 ) Profile Concurrent:Wait for Available TM - Total time to wait for a TM
before switchover to next available TM. Consider setting this to 1 (second).

13) Ensure enough TMs exist to service the incoming request load.

14) When the load is high, set the following profile to optimum values to achieve
better results.

PO: Approval Timeout Value - Total time for workflow call (When initiated from
Forms) to time out.

15) Set the sleep time on the Transaction Manager to a high number (e.g. 10
minutes), this avoids constant polls to check for shutdown requests.

Parallel Concurrent Processing (PCP) Environment

16) If the failover of managers is taking too long refer to Note 551895.1 -
'Failover Of Concurrent Manager Processes Takes More than 30 Minutes'.

17) To avoid known issues, apply Patch 15900099 (11i), Patch 15981173 (12.0 ),
Patch 15981176 (12.1.3) along with pre-requisites , when you are in the process of
implementing PCP. Also refer to Note 1389261.1.

18) Set profile option 'Concurrent: PCP Instance Check' to 'OFF' if instance-
sensitive failover is not required. Setting it to 'ON' means that concurrent
managers will fail over to a secondary application tier node if the database
instance to which it is connected goes down.

19)Transaction Manager uses DBMS_PIPE to communicate with application session prior

to 11i.ATG_PF.H RUP3. DBMS_PIPE in turn uses OS Pipe.We might use Advance Queue(AQ)
with 11i.ATG_PF.H RUP3 by setting System Profile Concurrent: TM Transport Type to
QUEUE.

Note Pipes are more efficient but require a Transaction Manager to be running on
each DB Instance (RAC). So you might want to use "Queue" for easy maintenance.

20) Add these parameters depends on your Database version

+ _lm_global_posts=TRUE
+ _immediate_commit_propagation=TRUE (11g RAC)
+ max_commit_propagation_delay=0 (9i RAC)

21) To speed up the PCP Failover ,Tune the below parameters.

Kernel parameters (Find the analogous parameter for your platform)
tcp_keepalive_intvl
tcp_keepalive_probes
tcp_keepalive_time ( Do not set this value to low; since it will then use up your
network resources with unnecessary traffic)

DCD (Dead connection detection); To setup update sqlnet.ora from the Database Tier.
sqlnet.expire_time

Set the environment Variable at Concurrent Manager Tier.

FDCPTRW

Check PMON Cycle & Sleep Intervals for ICM (internal Concurrent Manager) setup.
Navigation OAM -> SiteMap -> Monitoring -> Internal Concurrent Manager Link(Under
Availability) -> "View Status" -> "Edit ICM Runtime Parameters"

Tune Failover Processes.

In the case of node failover, the maximum number of processes that the work shift
can run simultaneously.

Nodes can become overloaded when a middle-tier node fails and service instances on
that node failover to their secondary nodes. The Failover Processes value should be
smaller than the normal Processes value, to lessen the impact on the existing
resources allocated on a secondary node. When failover occurs, the ICM uses the
Failover Processes value in place of the normal running processes value as it
iterates through service instances to perform queue sizing.

Navigation System Administration > Concurrent Managers > Standard Manager > Edit >
Failover Processes

System Administration > Concurrent Managers > Output Post

Processor > Edit > Failover Processes

Enable Reviver.
What is FNDREVIVER and How Is It Set? Note 466752.1

Ensure Internal Monitor is up and running on all the PCP Nodes. Ensure it has a
valid workshift.
Concurrent > Manager > Define > Query "Internal Monitor" > workshift

Note:- The sole job of an Internal Monitor Process is to monitor the

Internal Concurrent Manager and to restart that manager should it fail. The first
Internal Monitor Process to detect that the Internal Concurrent Manager has failed
restarts that manager on its own node.

PCP references:
NOTE: Multiple active managers are capable of processing the same jobs and can be
defined to run at the same time.
How To Run a Concurrent Program Against a Specific RAC Instance with PCP/RAC Setup?
(Note 1129203.1)
How to Activate Parallel Concurrent Processing - Background Facts and Setup Steps
(Note 602899.1)

Tuning Output Post Processor (OPP)

In order to tune the OPP to improve performance refer the below Note. It discusses
How to monitor the workload of OPP and Suggests you how to tune Output Post
Processor (OPP) to improve performance and avoid java.lang.OutOfMemoryError
exception.

Note 1399454.1 - 'Tuning Output Post Processor (OPP) to Improve Performance'.

Concurrent Processing Server Tuning

1. Any Concurrent Processing (CP) server tuning or load balancing needs are to be
addressed by Oracle Consulting. There are way too many site specific factors that
needs to be considered for optimum CP throughput: from machine hardware, to user
request volume, to required Work Shifts, to programs run time characteristics (long
/ short running)--not to mention also testing and benchmarking. Such a tasks, is
beyond the scope of ATG Support.

ATG support would be glad to investigate a failing manager or program issue;

however, CP performance issues due to increased concurrent request volume or due to
a new installation needs to be addressed by Oracle Consulting.

2. The "Tuning Concurrent Processing" chapter of the Technical Brief "A Holistic
Approach To Performance Tuning Oracle Applications Systems Release 11 and 11i" Note
69565.1 may provide some basic insight. Also reference the "Defining Concurrent
Managers" and the "Setting Up and Starting Concurrent Managers" chapters of the
"Oracle Applications System Administrator's Guide - Configuration".

3. As per Note 69565.1 "A Holistic Approach to Performance Tuning Oracle

Applications Systems", "50% of concurrent processing performance tuning is in the
business!"

4. Visit the Concurrent Processing Product Information Center (PIC) Note 1304305.1
for additional performance and setup documentation.