ADDM Report for Task 'TASK_36562'

---------------------------------

Analysis Period
---------------
AWR snapshot range from 36558 to 36559.
Time period starts at 26-JAN-17 11.00.30 AM
Time period ends at 26-JAN-17 12.00.40 PM

Analysis Target
---------------
Database 'MBLPROD' with DB ID 359300053.
Database version 11.2.0.3.0.
ADDM performed an analysis of instance mblprod, numbered 1 and hosted at
MBLDRDB.

Activity During the Analysis Period

-----------------------------------
Total database time was 45578 seconds.
The average number of active sessions was 12.63.

Summary of Findings
-------------------
Description Active Sessions Recommendation
s
Percent of Activity
---------------------------------------- ------------------- --------------
-
1 Virtual Memory Paging 12.63 | 100 1
2 Top SQL Statements 4.93 | 39.05 5
3 "User I/O" wait Class 4.2 | 33.23 0
4 Commits and Rollbacks 3.04 | 24.07 2
5 Top Segments by "User I/O" and "Cluster" 2.27 | 17.96 4
6 High Watermark Waits .68 | 5.37 0
7 PL/SQL Execution .35 | 2.77 2

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Findings and Recommendations

----------------------------

Finding 1: Virtual Memory Paging

Impact is 12.62 active sessions, 100% of total activity.
--------------------------------------------------------
Significant virtual memory paging was detected on the host operating system.

Recommendation 1: Host Configuration

Estimated benefit is 12.63 active sessions, 100% of total activity.
-------------------------------------------------------------------
Action
Host operating system was experiencing significant paging but no
particular root cause could be detected. Investigate processes that do
not belong to this instance running on the host that are consuming
significant amount of virtual memory. Also consider adding more physical
memory to the host.
Finding 2: Top SQL Statements
Impact is 4.93 active sessions, 39.05% of total activity.
---------------------------------------------------------
SQL statements consuming significant database time were found. These
statements offer a good opportunity for performance improvement.

Recommendation 1: SQL Tuning

Estimated benefit is 1.84 active sessions, 14.55% of total activity.
--------------------------------------------------------------------
Action
Run SQL Tuning Advisor on the UPSERT statement with SQL_ID
"998df80db92bv".
Related Object
SQL statement with SQL_ID 998df80db92bv.
MERGE INTO FBNK_LIMIT_REP_KEYS_CLL USING DUAL ON (RECID = :RECID)
WHEN MATCHED THEN UPDATE SET XMLRECORD=:XMLRECORD
WHEN NOT MATCHED THEN INSERT VALUES(:RECID, :XMLRECORD)
Rationale
The SQL spent 100% of its database time on CPU, I/O and Cluster waits.
This part of database time may be improved by the SQL Tuning Advisor.
Rationale
Database time for this SQL was divided as follows: 100% for SQL
execution, 0% for parsing, 0% for PL/SQL execution and 0% for Java
execution.
Rationale
SQL statement with SQL_ID "998df80db92bv" was executed 1261379 times and
had an average elapsed time of 0.0046 seconds.
Rationale
I/O and Cluster wait for INDEX "MBLLIVE.PK_FBNK_LIMIT_REP_KEYS_CLL" with
object ID 1387725 consumed 66% of the database time spent on this SQL
statement.
Rationale
I/O and Cluster wait for TABLE "MBLLIVE.FBNK_LIMIT_REP_KEYS_CLL" with
object ID 1356952 consumed 30% of the database time spent on this SQL
statement.

Recommendation 2: SQL Tuning

Estimated benefit is 1.52 active sessions, 12.07% of total activity.
--------------------------------------------------------------------
Action
Run SQL Tuning Advisor on the SELECT statement with SQL_ID
"1zh2ms1aymubn".
Related Object
SQL statement with SQL_ID 1zh2ms1aymubn.
SELECT t.XMLRECORD.getClobVal() FROM FBNK_ACCOUNT t WHERE RECID
=:RECID
Rationale
The SQL spent 97% of its database time on CPU, I/O and Cluster waits.
This part of database time may be improved by the SQL Tuning Advisor.
Rationale
Database time for this SQL was divided as follows: 89% for SQL
execution, 0% for parsing, 11% for PL/SQL execution and 0% for Java
execution.
Rationale
SQL statement with SQL_ID "1zh2ms1aymubn" was executed 4685287 times and
had an average elapsed time of 0.001 seconds.

Recommendation 3: SQL Tuning

Estimated benefit is .58 active sessions, 4.62% of total activity.
------------------------------------------------------------------
Action
Investigate the UPSERT statement with SQL_ID "bfha7fmd9d5s6" for
possible performance improvements. You can supplement the information
given here with an ASH report for this SQL_ID.
Related Object
SQL statement with SQL_ID bfha7fmd9d5s6.
MERGE INTO FBNK_PD_BALANCES USING DUAL ON (RECID = :RECID)
WHEN MATCHED THEN UPDATE SET XMLRECORD=XMLTYPE(:XMLRECORD, NULL, 1,
1)
WHEN NOT MATCHED THEN INSERT VALUES(:RECID, XMLTYPE(:XMLRECORD, NULL,
1, 1))
Rationale
The SQL spent only 8% of its database time on CPU, I/O and Cluster
waits. Therefore, the SQL Tuning Advisor is not applicable in this case.
Look at performance data for the SQL to find potential improvements.
Rationale
Database time for this SQL was divided as follows: 99% for SQL
execution, 0% for parsing, 1% for PL/SQL execution and 0% for Java
execution.
Rationale
SQL statement with SQL_ID "bfha7fmd9d5s6" was executed 25316 times and
had an average elapsed time of 0.088 seconds.
Rationale
Waiting for event "enq: HW - contention" in wait class "Configuration"
accounted for 91% of the database time spent in processing the SQL
statement with SQL_ID "bfha7fmd9d5s6".

Recommendation 4: SQL Tuning

Estimated benefit is .52 active sessions, 4.14% of total activity.
------------------------------------------------------------------
Action
Run SQL Tuning Advisor on the SELECT statement with SQL_ID
"73rqybqwja8k7".
Related Object
SQL statement with SQL_ID 73rqybqwja8k7.
SELECT t.XMLRECORD.getClobVal() FROM FBNK_ACCOUNT_STATEMENT t WHERE
RECID =:RECID
Rationale
The SQL spent 94% of its database time on CPU, I/O and Cluster waits.
This part of database time may be improved by the SQL Tuning Advisor.
Rationale
Database time for this SQL was divided as follows: 86% for SQL
execution, 0% for parsing, 14% for PL/SQL execution and 0% for Java
execution.
Rationale
SQL statement with SQL_ID "73rqybqwja8k7" was executed 3131080 times and
had an average elapsed time of 0.00059 seconds.

Recommendation 5: SQL Tuning

Estimated benefit is .4 active sessions, 3.15% of total activity.
-----------------------------------------------------------------
Action
Run SQL Tuning Advisor on the SELECT statement with SQL_ID
"25bqrc9pvapz9".
Related Object
SQL statement with SQL_ID 25bqrc9pvapz9.
SELECT t.XMLRECORD.getClobVal() FROM FBNK_EB_C004 t WHERE RECID
 =:RECID
Rationale
The SQL spent 100% of its database time on CPU, I/O and Cluster waits.
This part of database time may be improved by the SQL Tuning Advisor.
Rationale
Database time for this SQL was divided as follows: 100% for SQL
execution, 0% for parsing, 0% for PL/SQL execution and 0% for Java
execution.
Rationale
SQL statement with SQL_ID "25bqrc9pvapz9" was executed 240565 times and
had an average elapsed time of 0.0059 seconds.
Rationale
I/O and Cluster wait for TABLE "MBLLIVE.FBNK_EB_C004" with object ID
1355245 consumed 96% of the database time spent on this SQL statement.

Finding 3: "User I/O" wait Class

Impact is 4.19 active sessions, 33.23% of total activity.
---------------------------------------------------------
Wait class "User I/O" was consuming significant database time.
The throughput of the I/O subsystem was not significantly lower than expected.
The Oracle instance memory (SGA and PGA) was adequately sized.

No recommendations are available.

Finding 4: Commits and Rollbacks

Impact is 3.04 active sessions, 24.07% of total activity.
---------------------------------------------------------
Waits on event "log file sync" while performing COMMIT and ROLLBACK operations
were consuming significant database time.

Recommendation 1: Application Analysis

Estimated benefit is 3.04 active sessions, 24.07% of total activity.
--------------------------------------------------------------------
Action
Investigate application logic for possible reduction in the number of
COMMIT operations by increasing the size of transactions.
Rationale
The application was performing 55501 transactions per minute with an
average redo size of 5107 bytes per transaction.

Recommendation 2: Host Configuration

Estimated benefit is 3.04 active sessions, 24.07% of total activity.
--------------------------------------------------------------------
Action
Investigate the possibility of improving the performance of I/O to the
online redo log files.
Rationale
The average size of writes to the online redo log files was 27 K and the
average time per write was 1 milliseconds.
Rationale
The total I/O throughput on redo log files was 4.7 M per second for
reads and 9.4 M per second for writes.
Rationale
The redo log I/O throughput was divided as follows: 0% by RMAN and
recovery, 66% by Log Writer, 0% by Archiver, 0% by Streams AQ and 33% by
all other activity.
 Symptoms That Led to the Finding:
---------------------------------
Wait class "Commit" was consuming significant database time.
Impact is 3.04 active sessions, 24.07% of total activity.

Finding 5: Top Segments by "User I/O" and "Cluster"

Impact is 2.27 active sessions, 17.96% of total activity.
---------------------------------------------------------
Individual database segments responsible for significant "User I/O" and
"Cluster" waits were found.

Recommendation 1: Segment Tuning

Estimated benefit is 1.09 active sessions, 8.59% of total activity.
-------------------------------------------------------------------
Action
Investigate application logic involving I/O on INDEX
"MBLLIVE.PK_FBNK_LIMIT_REP_KEYS_CLL" with object ID 1387725.
Related Object
Database object with ID 1387725.
Action
Look at the "Top SQL Statements" finding for SQL statements consuming
significant I/O on this segment. For example, the UPSERT statement with
SQL_ID "998df80db92bv" is responsible for 100% of "User I/O" and
"Cluster" waits for this segment.
Rationale
The I/O usage statistics for the object are: 0 full object scans, 721278
physical reads, 35287 physical writes and 0 direct reads.

Recommendation 2: Segment Tuning

Estimated benefit is .5 active sessions, 3.96% of total activity.
-----------------------------------------------------------------
Action
Investigate application logic involving I/O on TABLE
"MBLLIVE.FBNK_LIMIT_REP_KEYS_CLL" with object ID 1356952.
Related Object
Database object with ID 1356952.
Action
Look at the "Top SQL Statements" finding for SQL statements consuming
significant I/O on this segment. For example, the UPSERT statement with
SQL_ID "998df80db92bv" is responsible for 100% of "User I/O" and
"Cluster" waits for this segment.
Rationale
The I/O usage statistics for the object are: 0 full object scans, 247656
physical reads, 253359 physical writes and 0 direct reads.

Recommendation 3: Segment Tuning

Estimated benefit is .35 active sessions, 2.79% of total activity.
------------------------------------------------------------------
Action
Investigate application logic involving I/O on TABLE
"MBLLIVE.FBNK_EB_C004" with object ID 1355245.
Related Object
Database object with ID 1355245.
Action
Look at the "Top SQL Statements" finding for SQL statements consuming
significant I/O on this segment. For example, the SELECT statement with
SQL_ID "25bqrc9pvapz9" is responsible for 100% of "User I/O" and
"Cluster" waits for this segment.
 Rationale
The I/O usage statistics for the object are: 0 full object scans, 224329
physical reads, 32312 physical writes and 0 direct reads.

Recommendation 4: Segment Tuning

Estimated benefit is .33 active sessions, 2.62% of total activity.
------------------------------------------------------------------
Action
Investigate application logic involving I/O on LOB
"MBLLIVE.SYS_LOB0001355245C00003$$" with object ID 1355246.
Related Object
Database object with ID 1355246.
Rationale
The I/O usage statistics for the object are: 0 full object scans, 187008
physical reads, 31718 physical writes and 0 direct reads.

Symptoms That Led to the Finding:

---------------------------------
Wait class "User I/O" was consuming significant database time.
Impact is 4.2 active sessions, 33.23% of total activity.

Finding 6: High Watermark Waits

Impact is .68 active sessions, 5.37% of total activity.
-------------------------------------------------------
Contention on the high watermark (HW) enqueue was consuming significant
database time.

No recommendations are available.

Symptoms That Led to the Finding:

---------------------------------
Wait class "Configuration" was consuming significant database time.
Impact is .69 active sessions, 5.44% of total activity.

Finding 7: PL/SQL Execution

Impact is .35 active sessions, 2.77% of total activity.
-------------------------------------------------------
PL/SQL execution consumed significant database time.

Recommendation 1: SQL Tuning

Estimated benefit is .32 active sessions, 2.52% of total activity.
------------------------------------------------------------------
Action
Tune the entry point PL/SQL "SYS.XMLTYPE.GETCLOBVAL#1" of type "TYPE"
and ID 5029. Refer to the PL/SQL documentation for addition information.
Rationale
1150 seconds spent in executing PL/SQL "SYS.XMLTYPE.GETCLOBVAL#1" of
type "TYPE" and ID 5029.

Recommendation 2: SQL Tuning

Estimated benefit is .03 active sessions, .24% of total activity.
-----------------------------------------------------------------
Action
Tune the entry point PL/SQL "SYS.XMLTYPE.XMLTYPE#4" of type "TYPE" and
ID 5029. Refer to the PL/SQL documentation for addition information.
Rationale
109 seconds spent in executing PL/SQL "SYS.XMLTYPE.XMLTYPE#4" of type
 "TYPE" and ID 5029.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Additional Information
----------------------

Miscellaneous Information
-------------------------
Wait class "Application" was not consuming significant database time.
Wait class "Concurrency" was not consuming significant database time.
CPU was not a bottleneck for the instance.
Wait class "Network" was not consuming significant database time.
Session connect and disconnect calls were not consuming significant database
time.
Hard parsing of SQL statements was not consuming significant database time.