Using AWR for IO Subsystem Analysis

Using AWR For IO Subsystem Analysis Mike Ault, Oracle Guru April, 2011 The World’s Fastest Storage ® for over thirty years!

Michael R. Ault, Oracle Guru Nuclear Navy 6 years Nuclear Chemist/Programmer 10 years Kennedy Western University Graduate Bachelors Degree Computer Science Certified in all Oracle Versions Since 6 Oracle DBA, author, since 1990 A Texas Memory Systems Presentation

Free Statspack/AWR Analysis Sponsored by Texas Memory Systems Looks for IO bottlenecks and other configuration issues. Straightforward tuning advice StatspackAnalyzer.com

Know Your Load Type! Online Transaction Processing Few reads Many writes Many small transactions Look for redo/undo and sequential read issues Decision Support/Data Warehouse Many reads Few writes (other then possible temp) Few transactions Look for sort/workarea and scattered read issues Mixed or Hybrid

Load Profile 1 Load Profile ~~~~~~~~~~~~ Per Second Per Transaction --------------- --------------- Redo size: 32,222,638.27 107,697,649.70 Logical reads: 368,069.30 1,230,197.16 Block changes: 177,853.38 594,438.94 Physical reads: 398.81 1,332.94 Physical writes: 1,404.91 4,695.64 User calls: 5.06 16.92 Parses: 28.29 94.55 Hard parses: 0.00 0.00 Sorts: 3.18 10.62 Logons: 0.24 0.80 Executes: 28.58 95.52 Transactions: 0.30

Load Profile 2 Load Profile Per Second Per Transaction Per Exec Per Call ~~~~~~~~~~~~ --------------- --------------- ---------- ---------- DB Time(s): 2.3 7.1 0.63 1.05 DB CPU(s): 0.3 0.9 0.07 0.13 Redo size: 800.5 2,461.8 Logical reads: 6,307.6 19,396.7 Block changes: 3.6 10.9 Physical reads: 2,704.9 8,317.8 Physical writes: 86.9 267.3 User calls: 2.2 6.8 Parses: 2.0 6.1 Hard parses: 0.0 0.1 W/A MB processed: 932,965.4 2,868,990.9 Logons: 0.1 0.2 Executes: 3.7 11.3 Rollbacks: 0.1 0.3 Transactions: 0.3

Top 5 Waits Section Critical to look closely at this section Use highest wait times to guide investigation DB FILE type waits – physical IO BUFFER type waits – Logical IO LOG type waits – Redo related PX – Parallel Query GC – Global Cache (RAC related) Undo – Undo or rollback segment related

Top 5 Waits Section Top 5 Timed Foreground Events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Avg wait % DB Event Waits Time(s) (ms) time Wait Class ------------------------------ ------------ ----------- ------ ------ ---------- db file sequential read 465,020 3,969 9 47.4 User I/O DB CPU 995 11.9 db file parallel read 2,251 322 143 3.8 User I/O db file scattered read 15,268 153 10 1.8 User I/O gc current block 2-way 108,739 116 1 1.4 Cluster

DB File Type Waits DB File Sequential Reads – memory starvation, non-selective indexes DB File Scattered Reads – full table scans, insufficient indexing Direct Path Writes – Appends, data loads Direct Path Reads – Parallel slaves used to retrieve data DB File Parallel Writes – Backup and partition use DB File Parallel Reads – Partition use DB File Single Write – File header writes, excessive data files Direct path read temp – Temp file activity (sorts, hashes, temp tables, bitmaps) Direct path write temp – Temp file activity (sorts, hashes, temp tables, bitmaps)

Log Type Waits log file parallel write – Look for log file contention log buffer space – Look at increasing log buffer size log file switch (checkpoint incomplete) – May indicate excessive db files or slow IO subsystem log file switch (archiving needed) – Indicates archive files are written too slowly log file switch completion – May need more log files per thread log file sync – Could indicate excessive commits

Undo Events undo segment extension – If excessive, tune undo latch: In memory undo latch – If excessive could be bug, check for your version, may have to turn off in memory undo wait for a undo record – Usually only during recovery of large transactions, look at turning off parallel undo recovery.

What Next? Determine wait events of concern Drill down to specific sections of report for deeper analysis Use custom scripts, ADDM and Ash to investigate issues

Drilling Down to IO Issues Operating System Statistics – Look at comparison between Busy, Idle and IOWait Events – See what events other than top five are IO related SQL Area – Look at SQL statements with high physical IO Tablespace Area – Review IO characteristics of tablespaces Datafile Area – Review IO characteristics of datafiles

Operating System Statistics What you get depends on OS Some may not give IO timings

Operating System Statistics Operating System Statistics DB/Inst: AULTDB/aultdb1 Snaps: 91-92 -> *TIME statistic values are diffed. All others display actual values. End Value is displayed if different -> ordered by statistic type (CPU Use, Virtual Memory, Hardware Config), Name Statistic Value End Value ------------------------- ---------------------- ---------------- BUSY_TIME 126,029 IDLE_TIME 597,505 IOWAIT_TIME 327,861 NICE_TIME 766 SYS_TIME 48,452 USER_TIME 76,784 LOAD 0 3 PHYSICAL_MEMORY_BYTES 3,189,190,656 NUM_CPUS 2 NUM_CPU_CORES 1 NUM_CPU_SOCKETS 1 GLOBAL_RECEIVE_SIZE_MAX 4,194,304 GLOBAL_SEND_SIZE_MAX 262,144 TCP_RECEIVE_SIZE_DEFAULT 87,380 TCP_RECEIVE_SIZE_MAX 1,048,576 TCP_RECEIVE_SIZE_MIN 4,096 TCP_SEND_SIZE_DEFAULT 65,536 TCP_SEND_SIZE_MAX 1,048,576 TCP_SEND_SIZE_MIN 4,096

Foreground Wait events Foreground=user processes Usually most important Usual source for top 5 wait events 2 sections – classes and events Classes are the rolled up sums for waits

Foreground Wait Classes Foreground Wait Class DB/Inst: AULTDB/aultdb1 Snaps: 91-92 -> s - second, ms - millisecond - 1000th of a second -> ordered by wait time desc, waits desc -> %Timeouts: value of 0 indicates value was < .5%. Value of null is truly 0 -> Captured Time accounts for 68.9% of Total DB time 8,371.33 (s) -> Total FG Wait Time: 4,770.85 (s) DB CPU time: 995.13 (s) Avg %Time Total Wait wait Wait Class Waits -outs Time (s) (ms) %DB time -------------------- ---------------- ----- ---------------- -------- --------- User I/O 518,267 0 4,449 9 53.1 DB CPU 995 11.9 Cluster 188,753 9 173 1 2.1 Other 3,806,446 100 146 0 1.7 Concurrency 1,854 2 2 1 0.0 Commit 15 0 1 39 0.0 Application 740 0 0 0 0.0 System I/O 40 0 0 3 0.0 Network 6,970 0 0 0 0.0 Configuration 0 0 0.0 -------------------------------------------------------------

Foreground Wait Events Foreground Wait Events DB/Inst: AULTDB/aultdb1 Snaps: 91-92 -> s - second, ms - millisecond - 1000th of a second -> Only events with Total Wait Time (s) >= .001 are shown -> ordered by wait time desc, waits desc (idle events last) -> %Timeouts: value of 0 indicates value was < .5%. Value of null is truly 0 Avg %Time Total Wait wait Waits % DB Event Waits -outs Time (s) (ms) /txn time -------------------------- ------------ ----- ---------- ------- -------- ------ db file sequential read 465,020 0 3,969 9 395.8 47.4 db file parallel read 2,251 0 322 143 1.9 3.8 db file scattered read 15,268 0 153 10 13.0 1.8 gc current block 2-way 108,739 11 116 1 92.5 1.4 PX Deq: reap credit 3,247,703 100 107 0 2,764.0 1.3 gc cr grant 2-way 57,265 7 28 0 48.7 .3 gc cr multi block request 22,451 6 23 1 19.1 .3 enq: BF - allocation conte 14 93 14 983 0.0 .2 PX qref latch 555,843 100 9 0 473.1 .1 IPC send completion sync 1,070 52 8 8 0.9 .1 gc remaster 22 0 5 221 0.0 .1

Background Wait Events SMON, PMON, DBWR, LMON, LMS, etc process waits Usually not a big contributor Types of waits should correspond to foreground waits

Background Wait Events Background Wait Events DB/Inst: AULTDB/aultdb1 Snaps: 91-92 -> ordered by wait time desc, waits desc (idle events last) -> Only events with Total Wait Time (s) >= .001 are shown -> %Timeouts: value of 0 indicates value was < .5%. Value of null is truly 0 Avg %Time Total Wait wait Waits % bg Event Waits -outs Time (s) (ms) /txn time -------------------------- ------------ ----- ---------- ------- -------- ------ control file sequential re 8,336 0 72 9 7.1 33.5 control file parallel writ 1,287 0 31 24 1.1 14.5 db file parallel write 792 0 11 14 0.7 5.3 log file parallel write 701 0 11 15 0.6 4.9 events in waitclass Other 44,191 98 5 0 37.6 2.5 library cache pin 449 0 2 4 0.4 .8 db file sequential read 221 0 2 7 0.2 .8 gc cr multi block request 1,915 0 2 1 1.6 .7 os thread startup 19 0 1 56 0.0 .5 gc cr block 2-way 246 0 0 1 0.2 .2 db file scattered read 18 0 0 12 0.0 .1 db file parallel read 3 0 0 59 0.0 .1 gc current grant 2-way 98 0 0 1 0.1 .1

Wait event Histograms Allows you to see wait time distributions Not available in earlier versions of Statspack Shows distribution of waits by time buckets

Wait Event Histograms Wait Event Histogram DB/Inst: AULTDB/aultdb1 Snaps: 91-92 -> Units for Total Waits column: K is 1000, M is 1000000, G is 1000000000 -> % of Waits: value of .0 indicates value was <.05%. Value of null is truly 0 -> % of Waits: column heading of <=1s is truly <1024ms, >1s is truly >=1024ms -> Ordered by Event (idle events last) % of Waits ----------------------------------------------- Total Event Waits <1ms <2ms <4ms <8ms <16ms <32ms <=1s >1s -------------------------- ----- ----- ----- ----- ----- ----- ----- ----- ----- control file parallel writ 1287 59.0 24.1 16.9 control file sequential re 9147 23.4 21.3 23.3 22.3 6.8 2.9 .0 db file parallel read 2256 .3 1.0 7.4 32.6 56.8 1.9 db file parallel write 792 .5 .8 4.2 28.7 50.0 8.8 7.1 db file scattered read 15K .4 2.7 31.5 59.2 5.8 .5 db file sequential read 465K .0 .6 2.2 49.5 45.0 2.3 .4 gc cr grant 2-way 50K 87.2 11.1 1.3 .3 .2 .0 gc cr multi block request 24K 59.0 36.8 3.0 .5 .6 .0 gc current block 2-way 84K 6.5 87.7 5.2 .3 .2 .0 library cache lock 488 82.8 10.9 4.9 1.0 .2 .2 library cache pin 4371 77.6 11.1 7.4 3.1 .6 .0 gcs remote message 274K 28.5 15.4 9.9 11.6 7.5 5.8 21.4 ges remote message 53K 11.4 3.3 2.7 1.9 1.8 2.1 76.8 -------------------------------------------------------------

Service Related Statistics New in 10g and above A service is a grouping of processes Users may be grouped in SYS$USER Application logins (single user) may be grouped with that user name Used to confirm IO issues, show what users are dominating IO

Service Statistics Service Statistics DB/Inst: AULTDB/aultdb1 Snaps: 91-92 -> ordered by DB Time Physical Logical Service Name DB Time (s) DB CPU (s) Reads (K) Reads (K) ---------------------------- ------------ ------------ ------------ ------------ aultdb 8,344 981 9,769 22,715 SYS$USERS 23 12 1 56 SYS$BACKGROUND 1 0 1 17 aultdbXDB 0 0 0 0 -------------------------------------------------------------

Service Wait Class Statistics Service Wait Class Stats DB/Inst: AULTDB/aultdb1 Snaps: 91-92 -> Wait Class info for services in the Service Statistics section. -> Total Waits and Time Waited displayed for the following wait classes: User I/O, Concurrency, Administrative, Network -> Time Waited (Wt Time) in seconds Service Name ---------------------------------------------------------------- User I/O User I/O Concurcy Concurcy Admin Admin Network Network Total Wts Wt Time Total Wts Wt Time Total Wts Wt Time Total Wts Wt Time --------- --------- --------- --------- --------- --------- --------- --------- aultdb 517710 4446 234 1 0 0 5828 0 SYS$USERS 555 3 1615 1 0 0 1140 0 SYS$BACKGROUND 350 3 3486 4 0 0 0 0 -------------------------------------------------------------

Operating System Statistics Shows breakdown by AWR period Only shown if the span of the report brackets other AWR collections Can be used to pinpoint peak activity times for further analysis

Operating System Statistics ------------------------------------------------------------- Operating System Statistics - Detail DB/Inst: AULTDB/aultdb1 Snaps: 91-92 Snap Time Load %busy %user %sys %idle %iowait --------------- -------- -------- -------- -------- -------- -------- 04-Aug 12:00:15 0.4 N/A N/A N/A N/A N/A 04-Aug 13:00:28 3.0 17.4 10.6 6.7 45.3 82.6 -------------------------------------------------------------

The SQL Sections Total Elapsed Time – IO waits Total CPU Time – Sorting, hashing Total Buffer Gets – High logical IO Total Disk Reads – High physical IO Total Executions – May indicate loop issues Total Parse Calls – Memory issues Total Sharable Memory - Informational Total Version Count – May indicate unsafe bind variables Total Cluster Wait Time – Indicates physical issues (RPB, block size)

AWR SQL Disk Reads Section SQL ordered by Reads DB/Inst: TEST/test Snaps: 29415-29416 -> Total Disk Reads: 1,736,422 -> Captured SQL account for 93.6% of Total Reads CPU Elapsed Physical Reads Executions per Exec %Total Time (s) Time (s) SQL Id -------------- ----------- ------------- ------ -------- --------- ------------- 268,145 5 53,629.0 15.4 16.42 1214.21 7k0dau5j6m6mw select * from ( SELECT c.TRANSACTION_NUMBER trx , c.CARD_BRANCH_CODE CARD_B RANCH_CODE , c.TERMINAL_ID_ACTUAL TERMINAL_ID_ACTUAL , c.BANK_CODE BANK_CODE , c.MERCHANT_ID MERCHANT_ID , c.Product_Type Product_Type , c.AUX_SERIAL_NUMB ER AUX_SERIAL_NUMBER , c.STATUS_CODE STATUS_CODE , c.TRANSACTION_TYPE TRANSACT 246,073 4 61,518.3 14.2 14.56 798.11 adf3kr30mg56w select count(*) aa from ISWITCH_TRANSACTIONS c Where c.Added_Date between : 1 and :2 and c.status_code ='1' AND c.TERMINAL_ID_ACTUAL IN (select INSTANCE_ ID from ISWITCH_CHANNEL_INSTANCE i where i.BRANCH_CODE = '012' ) and c.TERMINAL _ID_ACTUAL ='A0000012' and c.Transaction_Type ='102' and c.Product_Type ='ATM'

Instance Statistics Instance Activity Stats DB/Inst: ORCL/orcl Snaps: 29415-29416 Statistic Total per Second per Trans -------------------------------- -------- ----------- ------------- DBWR checkpoint buffers written 2,764 0.8 0.7 DBWR checkpoints 36 0.0 0.0 DBWR object drop buffers written 0 0.0 0.0 DBWR parallel query checkpoint b 118 0.0 0.0 DBWR revisited being-written buf 0 0.0 0.0 DBWR tablespace checkpoint buffe 0 0.0 0.0 DBWR thread checkpoint buffers w 0 0.0 0.0 DBWR transaction table writes 84 0.0 0.0 DBWR undo block writes 612 0.2 0.2 summed dirty queue length 1,747 0.5 0.4

Instance Statistics Statistic Total per Second per Trans -------------------------------- -------- ----------- ------------- change write time 68 0.0 0.0 cluster key scan block gets 10,664 3.0 2.6 cluster key scans 2,363 0.7 0.6 index fast full scans (direct re 0 0.0 0.0 index fast full scans (full) 787 0.2 0.2 index fast full scans (rowid ran 0 0.0 0.0 index fetch by key 93,700 26.0 22.4 index scans kdiixs1 71,615 19.9 17.2 leaf node 90-10 splits 20 0.0 0.0 leaf node splits 67 0.0 0.0 lob reads 49 0.0 0.0 lob writes 157 0.0 0.0 lob writes unaligned 157 0.0 0.0

Instance Statistics Statistic Total per Second per Trans ------------------------------- -------------- ----------- ----------- physical read IO requests 1,306,076 362.7 312.7 physical read bytes 14,224,769,024 3,950,124.3 3,405,498.9 physical read total IO requests 1,318,016 366.0 315.5 physical read total bytes 14,446,929,920 4,011,816.9 3,458,685.6 physical read total multi block 66,857 18.6 16.0 physical reads 1,736,422 482.2 415.7 physical reads cache 1,634,630 453.9 391.3 physical reads cache prefetch 344,579 95.7 82.5 physical reads direct 101,792 28.3 24.4 physical reads direct (lob) 0 0.0 0.0 physical reads direct temporary 0 0.0 0.0 physical reads prefetch warmup 0 0.0 0.0

Instance Statistics Statistic Total per Second per Trans -------------------------------- ----------- ---------- -------------physical write IO requests 4,208 1.2 1.0 physical write bytes 42,860,544 11,902.1 10,261.1 physical write total IO requests 26,028 7.2 6.2 physical write total bytes 238,408,192 66,204.4 57,076.4 physical write total multi block 12,240 3.4 2.9 physical writes 5,232 1.5 1.3 physical writes direct 247 0.1 0.1 physical writes direct (lob) 0 0.0 0.0 physical writes direct temporary 0 0.0 0.0 physical writes from cache 4,985 1.4 1.2 physical writes non checkpoint 3,862 1.1 0.9

Instance statistics Statistic Total per Second per Trans -------------------------------- -------- ----------- ------------- redo synch time 739 0.2 0.2 redo synch writes 9,341 2.6 2.2 redo write time 803 0.2 0.2 redo writes 6,159 1.7 1.5

Instance Statistics Statistic Total per Second per Trans -------------------------------- ----------- ----------- ------------- table fetch by rowid 4,701,301 1,305.5 1,125.5 table fetch continued row 71 0.0 0.0 table scan blocks gotten 3,411,330 947.3 816.7 table scan rows gotten 108,225,337 30,053.5 25,909.8 table scans (cache partitions) 0 0.0 0.0 table scans (direct read) 3,055 0.9 0.7 table scans (long tables) 3,087 0.9 0.7 table scans (rowid ranges) 3,055 0.9 0.7 table scans (short tables) 30,677 8.5 7.3 user I/O wait time 585,938 162.7 140.3 sorts (disk) 0 0.0 0.0 workarea executions – multipass 0 0.0 0.0 workarea executions - onepass 0 0.0 0.0 ------------------------------------------------------------

Instance Thread Activity Shows how often your logs are switching Rule of thumb says 4 per hour. Whatever works for your application and doesn’t result in waits is good.

Instance Thread Activity Instance Activity Stats - Thread ActivityDB/Inst: TEST/test Snaps: 29415-2941 -> Statistics identified by '(derived)' come from sources other than SYSSTAT Statistic Total per Hour -------------------------------- ------------------ --------- log switches (derived) 6 6.00 -------------------------------------------------------------

Tablespace IO Shows reads, reads/sec, read latency, avg blocks read, writes, writes/sec, (no write latency), bufer busy waits, avg wait for BBW Sum of reads+writes divided by clock time for report should come close to total physical reads+total physical writes on a per second basis, however, I have seen it differ! Tells you what tablesapces are doing the most IO Datafile IO (not shown) breaks it out by data file

Tablespace IO -> ordered by IOs (Reads + Writes) desc Tablespace ------------------------------ Av Av Av Av Buffer Av Buf Reads Reads/s Rd(ms) Blks/Rd Writes Writes/s Waits Wt(ms) -------------- ------- ------ ------- ------------ -------- ---------- ------ DEVDATA 1,173,883 326 3.7 1.3 2,734 1 323,173 2.8 IBKDATA 98,513 27 5.7 1.4 264 0 1 10.0 DEVINDEX 15,225 4 0.5 1.0 125 0 0 0.0 RETAILDATA 9,218 3 8.7 1.0 202 0 0 0.0 SYSTEM 5,594 2 3.3 1.5 144 0 0 0.0 SYSAUX 919 0 5.0 1.1 344 0 0 0.0 UNDOTBS1 86 0 4.9 1.0 221 0 0 0.0 TEMP_NEW 16 0 0.6 1.0 0 0 0 0.0

Buffer Space advisory Shows how much phsyical IO will be reduced by increasing )or decreasing) buffer size A majority of this IO will be single block reads Use to determine if increasing cache will reduce physical IO However, if buffer busy waits are predominately disk blocks and majority waits are sequential read, usually memory starvation is occurring regardless of what this says

Buffer Space Advisory Est Phys Size for Size Buffers for Read Estimated P Est (M) Factor Estimate Factor Physical Reads --- -------- ------ ---------------- ------ ------------------ D 192 .3 23,796 1.1 438,875,348 D 256 .4 31,728 1.1 431,906,549 D 320 .5 39,660 1.0 426,776,467 D 384 .5 47,592 1.0 422,102,165 D 448 .6 55,524 1.0 419,074,429 D 512 .7 63,456 1.0 416,407,215 D 576 .8 71,388 1.0 414,312,661 D 640 .9 79,320 1.0 412,509,975 D 704 1.0 87,252 1.0 410,854,050 D 768 1.1 95,184 1.0 408,796,940 D 832 1.2 103,116 1.0 407,142,199 D 896 1.3 111,048 1.0 405,649,002 D 960 1.4 118,980 1.0 404,148,251 D 1,024 1.5 126,912 1.0 402,745,613 D 1,088 1.5 134,844 1.0 401,506,706 D 1,152 1.6 142,776 1.0 400,276,499 D 1,216 1.7 150,708 1.0 399,110,953 D 1,280 1.8 158,640 0.9 389,800,510

PGA Aggregate Target Controls temp usage Sorts, hashes, GT tables, bitmap work Excessive temp IO shows PGA_AGGREGATE_TARGET issues sorts(disk), single and multipass operations show problems Note: some sort operations (shared server connections, backup up, etc) still use sort area size and hash_area_size

PGA_AGGREGATE_TARGET > PGA cache hit % - percentage of W/A (WorkArea) data processed only in-memory PGA Cache Hit % W/A MB Processed Extra W/A MB Read/Written --------------- ------------------ -------------------------- 99.8 845,324 1,607 -------------------------------------------------------------

PGA_AGGREGATE_TARGET PGA Aggr Target Histogram DB/Inst: TEST/test Snaps: 24080-24107 -> Optimal Executions are purely in-memory operations Low High Optimal Optimal Total Execs Optimal Execs 1-Pass Execs M-Pass Execs ------- ------- -------------- -------------- ------------ ------------ 2K 4K 8,446,124 8,446,124 0 0 64K 128K 28,112 28,112 0 0 128K 256K 43,917 43,917 0 0 256K 512K 173,347 173,347 0 0 512K 1024K 643,107 643,107 0 0 1M 2M 24,595 24,595 0 0 2M 4M 15,934 15,912 22 0 4M 8M 20,446 20,436 10 0 8M 16M 3,848 3,846 2 0 16M 32M 1,671 1,668 3 0 32M 64M 1,173 1,172 1 0 64M 128M 76 74 2 0 128M 256M 378 378 0 0 512M 1024M 2 0 2 0 -------------------------------------------------------------

PGA_AGGREGATE_TARGET PGA Target Size W/A MB W/A MB Read/ Cache Overallo Estd Est (MB) Factr Processed Written to Disk Hit % Count Time ---------- ------- ---------------- ---------------- ------ -------- ------- 7,872 0.1 35,936,407.5 32,945.6 100.0 0 5.2E+10 15,744 0.3 35,936,407.5 31,937.3 100.0 0 5.2E+10 31,488 0.5 35,936,407.5 31,937.3 100.0 0 5.2E+10 47,232 0.8 35,936,407.5 31,937.3 100.0 0 5.2E+10 62,976 1.0 35,936,407.5 31,937.3 100.0 0 5.2E+10 75,571 1.2 35,936,407.5 5,941.2 100.0 0 5.2E+10 88,166 1.4 35,936,407.5 5,941.2 100.0 0 5.2E+10 100,762 1.6 35,936,407.5 5,941.2 100.0 0 5.2E+10 113,357 1.8 35,936,407.5 5,941.2 100.0 0 5.2E+10 125,952 2.0 35,936,407.5 5,941.2 100.0 0 5.2E+10 188,928 3.0 35,936,407.5 5,941.2 100.0 0 5.2E+10 251,904 4.0 35,936,407.5 5,941.2 100.0 0 5.2E+10 -------------------------------------------------------------

Streams Pool If you get spills to disk increase streams pool Streams Pool Advisory DB/Inst: TMSP/tmsp Snap: 24107 Size for Size Est Spill Est Spill Est Unspill Est Unspill Est (MB) Factor Count Time (s) Count Time (s) ---------- --------- ----------- ----------- ----------- ----------- 1,536 0.8 0 0 0 0 1,792 0.9 0 0 0 0 2,048 1.0 0 0 0 0 2,304 1.1 0 0 0 0 2,560 1.3 0 0 0 0 2,816 1.4 0 0 0 0 3,072 1.5 0 0 0 0 3,328 1.6 0 0 0 0 3,584 1.8 0 0 0 0 3,840 1.9 0 0 0 0 4,096 2.0 0 0 0 0 4,352 2.1 0 0 0 0 4,608 2.3 0 0 0 0 4,864 2.4 0 0 0 0 5,120 2.5 0 0 0 0 -------------------------------------------------------------

Buffer Wait Statistics Tells you exactly where buffer busy waits are coming from Usually will be from disk blocks Excessive disk block waits + db file sequential reads means memory starvation By themselves could indicate insufficient or congested db writer processes

Buffer Wait Statistics Buffer Wait Statistics DB/Inst: TEST/test Snaps: 24080-24107 -> ordered by wait time desc, waits desc Class Waits Total Wait Time (s) Avg Time (ms) ------------------ ----------- ------------------- -------------- data block 2,859,144 29,615 10 segment header 1,173 25 21 1st level bmb 1,204 20 16 2nd level bmb 992 18 19 undo header 15,332 14 1 undo block 19,509 6 0 file header block 2 0 30 -------------------------------------------------------------

UNDO (rollback) Segments Self adjusting Starts with10 whose size is based on the size of tablespace When ratio of transactions to transactions per rollback segment is > number of current undo segments, more are created They are never dropped

UNDO (Rollback) segments Undo Num Undo Number of Max Qry Max Tx Min/Max STO/ uS/uR/uU/ TS# Blocks (K) Transactions Len (s) Concurcy TR (mins) OOS eS/eR/eU ---- ---------- --------------- -------- -------- --------- ----- -------------- 127 690.1 2,408,751 9,232 133 420/420 0/0 0/0/0/0/0/0 ------------------------------------------------------------- Undo Segment Stats DB/Inst: TEST/test Snaps: 24080-24107 -> Most recent 35 Undostat rows, ordered by Time desc Num Undo Number of Max Qry Max Tx Tun Ret STO/ uS/uR/uU/ End Time Blocks Transactions Len (s) Concy (mins) OOS eS/eR/eU ------------ ----------- ------------ ------- ------- ------- ----- ------------ 27-Jan 23:55 1,057 2,119 2,033 44 420 0/0 0/0/0/0/0/0 27-Jan 23:45 1,652 5,535 1,431 80 420 0/0 0/0/0/0/0/0 27-Jan 23:35 965 3,234 830 66 420 0/0 0/0/0/0/0/0 27-Jan 23:25 1,021 4,856 945 65 420 0/0 0/0/0/0/0/0 27-Jan 23:15 566 3,781 2,267 55 420 0/0 0/0/0/0/0/0 27-Jan 23:05 6,914 30,730 1,665 70 420 0/0 0/0/0/0/0/0 27-Jan 22:55 1,561 4,536 1,064 53 420 0/0 0/0/0/0/0/0 27-Jan 22:45 543 2,085 940 48 420 0/0 0/0/0/0/0/0 27-Jan 22:35 1,111 6,020 338 71 420 0/0 0/0/0/0/0/0 27-Jan 22:25 950 3,195 1,365 43 420 0/0 0/0/0/0/0/0 27-Jan 22:15 1,043 3,859 763 62 420 0/0 0/0/0/0/0/0 27-Jan 22:05 3,699 16,955 937 86 420 0/0 0/0/0/0/0/0 27-Jan 21:55 976 3,732 361 96 420 0/0 0/0/0/0/0/0 27-Jan 21:45 891 3,018 937 98 420 0/0 0/0/0/0/0/0 27-Jan 21:35 1,619 3,165 335 86 420 0/0 0/0/0/0/0/0 27-Jan 21:25 5,174 13,627 1,534 76 420 0/0 0/0/0/0/0/0

Segment Statistics Tells by type of action what segments are seeing most activity Tune objects by adding indexes, changing rows per block (RAC), tuning object related SQL

Segment Statistics Segments by Physical Reads DB/Inst: TEST/test Snaps: 24080-24107 -> Total Physical Reads: 27,349,451 -> Captured Segments account for 53.8% of Total Tablespace Subobject Obj. Physical Owner Name Object Name Name Type Reads %Total ------- ---------- -------------------- ---------- ----- ------------ ------- DNE4 MEP_D01 MODE_EXECUTION_POINT TABLE 3,350,928 12.25 FMS FMS_DATA ACCTNG_EVENT TABLE 2,720,574 9.95 DNE4 SHIPMENT_D SHIPMENT_PLAN TABLE 1,046,173 3.83 ISS ISS_DATA IB_SHIPMENT_STATUS_H TABLE 852,046 3.12 DNE4 TMS_AUDIT TEMP_SQL TABLE 552,676 2.02 -------------------------------------------------------------

Segment Statistics Segments by Direct Physical Reads DB/Inst: TEST/test Snaps: 24080-24107 -> Total Direct Physical Reads: 6,048,316 -> Captured Segments account for 77.9% of Total Tablespace Subobject Obj. Direct Owner Name Object Name Name Type Reads %Total ------ ---------- -------------------- ---------- ----- ------------ ------- FMS FMS_DATA ACCTNG_EVENT TABLE 2,720,470 44.98 DNE4 MEP_D01 MODE_EXECUTION_POINT TABLE 1,468,961 24.29 FMS FMS_DATA FIN_EVENT TABLE 521,600 8.62 FAM FAM_D SYS_LOB0000148435C00 LOB 360 .01 SYS SYSAUX SYS_LOB0000008958C00 LOB 19 .00 -------------------------------------------------------------

Segment Statistics Segments by Physical Writes DB/Inst: TEST/test Snaps: 24080-24107 -> Total Physical Writes: 5,084,963 -> Captured Segments account for 36.9% of Total Tablespace Subobject Obj. Physical Owner Name Object Name Name Type Writes %Total ---------- ---------- -------------------- ---------- ----- ------------ ------- ISD ISD_INDX ISD_SUPPORT_I03 INDEX 139,666 2.75 ISD ISD_INDX ISD_SUPPORT_I02 INDEX 134,617 2.65 DNE4 SHIPMENT_I SP_SUPP_INFO_I05 INDEX 113,698 2.24 DNE4 SHIPMENT_I SP_SUPP_INFO_I04 INDEX 112,988 2.22 DNE4_AUDIT TMS_AUDIT WORKFLOW_UNIT_AUDIT_ INDEX 109,555 2.15 -------------------------------------------------------------

Segment Statistics Segments by Direct Physical Writes DB/Inst: TEST/test Snaps: 24080-24107 -> Total Direct Physical Writes: 231,265 -> Captured Segments account for 9.8% of Total Tablespace Subobject Obj. Direct Owner Name Object Name Name Type Writes %Total ---------- ---------- -------------------- ---------- ----- ------------ ------- DNE4 DNE4_CLOB SYS_LOB0000050106C00 LOB 17,136 7.41 DNE4_AUDIT TMS_AUDIT SYS_LOB0000126341C00 LOB 4,274 1.85 FAM FAM_D SYS_LOB0000148435C00 LOB 719 .31 SYS SYSAUX WRH$_ACTIVE_SESSION_ 0822_24013 TABLE 239 .10 DNE4_AUDIT TMS_AUDIT SYS_LOB0000126341C00 LOB 184 .08 -------------------------------------------------------------

Segments by Table Scans DB/Inst: TMSP/tmsp Snaps: 24080-24107 -> Total Table Scans: 6,618 -> Captured Segments account for 2.5% of Total Tablespace Subobject Obj. Table Owner Name Object Name Name Type Scans %Total ---------- ---------- -------------------- ---------- ----- ------------ ------- DNE4 INDEX01 GP_CITY_I4 INDEX 41 .62 DNE4 RATE_INFO_ RATE_I3 INDEX 40 .60 DNE4 MEP_D01 MODE_EXECUTION_POINT TABLE 25 .38 FMS FMS_DATA ACCTNG_EVENT TABLE 10 .15 DNE4 DNE4_D02 ME_SERVICE_GROUP TABLE 6 .09 -------------------------------------------------------------

Initialization Parameters Only a couple deal with IO If the IO parameters are incorrect, may cost up to 30% of IO performance

Initialization Parameters Parameter Name Begin value ----------------------------- ---------------------------- control_files /TEST/ctl01/control01.ctl, /TEST/ db_2k_cache_size 268435456 db_block_size 32768 db_cache_size 15032385536 db_file_multiblock_read_count 16 db_files 5000 db_keep_cache_size 268435456 db_recovery_file_dest /archive/TEST db_recovery_file_dest_size 751619276800 log_archive_dest /archive/TEST log_archive_format tmsp_%t_%s_%r.arc log_archive_max_processes 4 log_buffer 262144000 memory_max_target 68719476736 memory_target 68719476736 streams_pool_size 2147483648 undo_management AUTO undo_retention 25200 undo_tablespace UNDO_TMS

What is Missing? Disk_async_io – set to async Filesystemio_options – set to setall SORT_AREA_SIZE – take care of thos non-PGAT sorts Since they are using MEMORY_MAX_SIZE and MEMORY_TARGET other items not set (PGAT) Why should they be?

Why is IO Important? Most performance issues can be tracked to IO issues IO Latency is the most critical issue Modern databases rapidly tax the most complex disk based system

Oracle and Queries Where does latency matter? Memory SGA & PGA Oracle Processes Reads - Cache miss Foreground Waits: DB file sequential read DB file scattered read 5 ms User’s Query Storage latency READS Tables & Indexes Logs

Oracle and Insert/update/delete Where does latency matter? Memory SGA & PGA Oracle Processes LGWR (foreground) DBWR (background) Users Insert Commit LOG WRITES Tables & Indexes Logs

RamSan - Reads Memory SGA & PGA Oracle Processes Reads - Cache miss Foreground Waits: DB file sequential read DB file scattered read 0.25 ms User’s Query SSD latency Logs

RamSan – Writes Insert, Update, Delete Memory SGA & PGA Oracle Processes LGWR SSD (foreground) DBWR (background) Disk speed Users Insert Commit

Thank You! Mike Ault [email_address] http:// www.statspackanalyzer.com /

Using AWR for IO Subsystem Analysis

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Using AWR for IO Subsystem Analysis (20)

Recently uploaded (20)

Using AWR for IO Subsystem Analysis