Ash architecture and advanced usage rmoug2014

ASH Architecture and Advanced Usage
John Beresniewicz, Graham Wood, Uri Shaft
Oracle America
RMOUG 2014

The following is intended to outline our
general product direction. It is intended for
information purposes only, and may not be
incorporated into any contract. It is not a
commitment to deliver any material, code, or
functionality, and should not be relied upon in
making purchasing decisions.
The development, release, and timing of any
features or functionality described for
Oracle’s products remains at the sole
discretion of Oracle.

Agenda
Part 1:
• The ASH mechanism
• DB Time estimates from ASH
• Top Activity and ASH Analytics
• Avoiding mistakes
Part 2:
• The ASH fix-up mechanism
• Estimating event counts and latencies from ASH

Motivation for ASH in 10g
Performance diagnostics tool
• Always on (unlike SQL trace)
• Keeps a history of what happened
• Keeps fine granularity of details
• Avoid averaging effects of coarse time granularity
Other requirements:
• Efficient: very small performance penalty for using it
• Robust: no locks of any kind, won’t stall
• Works well even when system is overloaded

Active Session History (time based)
A sampling of session activity at regular intervals
t1
Session 1
Session 2
Session 3
Session Time Duration Activity SQL Object
1 t1 null CPU SQL1 null
2 t1 100ms I/O SQL2 EMP
3 t1 5ms I/O SQL1 EMP
3 t2 1sec Row lock SQL1 DEPT
t2 t3

ASH Architecture
Every
1 hour
or
out-of-space
AWR
Circular buffer
in SGA
(2MB per CPU)
DBA_HIST_ACTIVE_SESS_HISTORY
V$ACTIVE_SESSION_HISTORY
Session
state
objects
MMON Lite
(MMNL)
V$SESSION
V$SESSION_WAIT
Variable
length rows
Direct-path
INSERTS
Write
1 out of 10
samples

ASH Architecture
Readers go
unlatched
Writer goes
one direction
Every
1 hour
or
out-of-space
AWR
Circular buffer
in SGA
(2MB per CPU)
DBA_HIST_ACTIVE_SESS_HISTORYV$ACTIVE_SESSION_HISTORY
MMON Lite
(MMNL)
Session
state
objects
V$SESSION
V$SESSION_WAIT
Readers go the
opposite way
Indexed
on time
Indexed
on time

Sampling Pseudo-code
1 FORALL SESSION STATE OBJECTS
2 IS SESSION CONNECTED?
NO => NEXT SESSION
YES:
3 IS SESSION ACTIVE?
NO => NEXT SESSION
YES:
4 MEMCPY SESSION STATE OBJ
5 CHECK CONSISTENCY OF COPY WITH LIVE SESSION
6 IS COPY CONSISTENT?
YES:
WRITE ASH ROW FROM COPY
NO:
IF FIRST COPY, REPEAT 4-6
ELSE => NEXT SESSION (NO ASH ROW WRITTEN)

Default Settings
• Sampling interval = 1000ms = 1 sec
• Disk filter ratio = 10 = 1 in 10 samples written to AWR
• ASH buffer size:
• Min( Max (5% shared pool, 2% SGA), 2MB per CPU)
• Absolute Max of 256MB
NOTE: the MMNL sampler session is not sampled

Control Parameters
• _ash_size : size of ASH buffer in bytes
• K/M notation works (e.g. 200M)
• _ash_sampling_interval : in milliseconds
• Min = 100, Max = 10,000
• _ash_disk_filter_ratio : every Nth sample to AWR
• MOD(sample_id, N) = 0 where N=disk filter ratio
• _sample_all : samples idle and active sessions

V$ASH_INFO (new in 11.2)
desc v$ash_info
Name Null Type
------------------------------ -------- --------------
TOTAL_SIZE NUMBER
FIXED_SIZE NUMBER
SAMPLING_INTERVAL NUMBER
OLDEST_SAMPLE_ID NUMBER
OLDEST_SAMPLE_TIME TIMESTAMP(9)
LATEST_SAMPLE_ID NUMBER
LATEST_SAMPLE_TIME TIMESTAMP(9)
SAMPLE_COUNT NUMBER
SAMPLED_BYTES NUMBER
SAMPLER_ELAPSED_TIME NUMBER
DISK_FILTER_RATIO NUMBER
AWR_FLUSH_BYTES NUMBER
AWR_FLUSH_ELAPSED_TIME NUMBER
AWR_FLUSH_COUNT NUMBER
AWR_FLUSH_EMERGENCY_COUNT NUMBER
Compute memory buffer
window size
Compute avg sampling
time in microseconds
DROPPED_SAMPLE_COUNT NUMBER

ASH Dimensions
desc v$active_session_history
Name Null Type
------------------------------ -------- ---------------
----------------
SAMPLE_ID NUMBER
SAMPLE_TIME TIMESTAMP(3)
IS_AWR_SAMPLE VARCHAR2(1)
SESSION_ID NUMBER
SESSION_SERIAL# NUMBER
SESSION_TYPE VARCHAR2(10)
FLAGS NUMBER
USER_ID NUMBER
.
.
.
93 rows selected

SQL Dimensions
SQL Analysis
-------------------------
SQL_ID VARCHAR2(13)
IS_SQLID_CURRENT VARCHAR2(1)
SQL_CHILD_NUMBER NUMBER
SQL_OPCODE NUMBER
SQL_OPNAME VARCHAR2(64)
FORCE_MATCHING_SIGNATURE NUMBER
TOP_LEVEL_SQL_ID VARCHAR2(13)
TOP_LEVEL_SQL_OPCODE NUMBER
SQL_PLAN_HASH_VALUE NUMBER
SQL_PLAN_LINE_ID NUMBER
SQL_PLAN_OPERATION VARCHAR2(30)
SQL_PLAN_OPTIONS VARCHAR2(30)
SQL_EXEC_ID NUMBER
SQL_EXEC_START DATE
PLSQL_ENTRY_OBJECT_ID NUMBER
PLSQL_ENTRY_SUBPROGRAM_ID NUMBER
PLSQL_OBJECT_ID NUMBER
PLSQL_SUBPROGRAM_ID NUMBER
QC_INSTANCE_ID NUMBER
QC_SESSION_ID NUMBER
QC_SESSION_SERIAL# NUMBER

Evolution of SQL_ID
• 11.1.0.7+ it is the currently executing SQL
• Except recursive server code SQL, triggers
• Previously there were various schemes to get the
bottom user-level SQL
• They all failed since it is very difficult to do.
• ENTRY_LEVEL columns added for applications with
generic PL/SQL entry points and SQL underneath
• Need to know what the application initially called

Wait Event Dimensions
Wait Event Analysis
--------------------------------------------------
EVENT VARCHAR2(64)
EVENT_ID NUMBER
EVENT# NUMBER
SEQ# NUMBER
P1TEXT VARCHAR2(64)
P1 NUMBER
P2TEXT VARCHAR2(64)
P2 NUMBER
P3TEXT VARCHAR2(64)
P3 NUMBER
WAIT_CLASS VARCHAR2(64)
WAIT_CLASS_ID NUMBER
WAIT_TIME NUMBER
SESSION_STATE VARCHAR2(7)
TIME_WAITED NUMBER

Blocking Session info
Locking/Blocking Analysis
--------------------------------------------------
BLOCKING_SESSION_STATUS
VARCHAR2(11)
BLOCKING_SESSION NUMBER
BLOCKING_SESSION_SERIAL# NUMBER
BLOCKING_INST_ID NUMBER
BLOCKING_HANGCHAIN_INFO
VARCHAR2(1)
Prior to 11.2:
• Finding blocking session could induce locking or crashes
• Therefore blocking session only in same instance, in some cases
From 11.2:
• The pre-11.2 path is taken for short local blocking
• Long blocking events (3 seconds in same instance, 10 seconds in
RAC) get accurate hang information copied into ASH
• Information comes from Hang Manager (working behind the scenes)

Object Dimensions
Object Analysis
--------------------------------------------------
CURRENT_OBJ# NUMBER
CURRENT_FILE# NUMBER
CURRENT_BLOCK# NUMBER
CURRENT_ROW# NUMBER
• CURRENT_OBJ# =
DBA_OBJECTS.DATA_OBJECT_ID of the segment
• It is only valid in specific wait events:
• I/O events on data blocks
• Cluster (global cache) events on data blocks
• Row Locks and Table Locks
• Buffer busy (and associated RAC events)
DANGER: the column is not cleared after waiting

Application Dimensions
Application Dimensions
------------------------------------------------
SERVICE_HASH NUMBER
PROGRAM VARCHAR2(48)
MODULE VARCHAR2(48)
ACTION VARCHAR2(32)
CLIENT_ID VARCHAR2(64)
MACHINE VARCHAR2(64)
PORT NUMBER
ECID VARCHAR2(64)
CONSUMER_GROUP_ID NUMBER
TOP_LEVEL_CALL# NUMBER
TOP_LEVEL_CALL_NAME VARCHAR2(64)
CONSUMER_GROUP_ID NUMBER
XID RAW(8)
REMOTE_INSTANCE# NUMBER
TIME_MODEL NUMBER

Session Statistics (SQL Monitoring)
Session Statistics
---------------------------------------------------
TM_DELTA_TIME NUMBER
TM_DELTA_CPU_TIME NUMBER
TM_DELTA_DB_TIME NUMBER
DELTA_TIME NUMBER
DELTA_READ_IO_REQUESTS NUMBER
DELTA_WRITE_IO_REQUESTS NUMBER
DELTA_READ_IO_BYTES NUMBER
DELTA_WRITE_IO_BYTES NUMBER
DELTA_INTERCONNECT_IO_BYTES NUMBER
PGA_ALLOCATED NUMBER
TEMP_SPACE_ALLOCATED NUMBER

Bit Vector Dimensions
Bitvec and Replay
---------------------------------------------------
IN_CONNECTION_MGMT VARCHAR2(1)
IN_PARSE VARCHAR2(1)
IN_HARD_PARSE VARCHAR2(1)
IN_SQL_EXECUTION VARCHAR2(1)
IN_PLSQL_EXECUTION VARCHAR2(1)
IN_PLSQL_RPC VARCHAR2(1)
IN_PLSQL_COMPILATION VARCHAR2(1)
IN_JAVA_EXECUTION VARCHAR2(1)
IN_BIND VARCHAR2(1)
IN_CURSOR_CLOSE VARCHAR2(1)
IN_SEQUENCE_LOAD VARCHAR2(1)
CAPTURE_OVERHEAD VARCHAR2(1)
REPLAY_OVERHEAD VARCHAR2(1)
IS_CAPTURED VARCHAR2(1)
IS_REPLAYED VARCHAR2(1)

Why does ASH work when the server
is CPU bound?
1. ASH sampler is very efficient and does not lock
• Should complete a sample within a single CPU slice
2. After sampling, the sampler computes next
scheduled sample time and sleeps until then
3. Upon scheduled wake-up, it waits for CPU (runq)
and samples again
• CPU bound sample times are shifted by one runq but
intervals stay close to 1 second

ASH Sampler and Run-queue
S_t0 S_t2S_t1
Run queue Run queue
A_t1A_t0
Run queue
A_t2
Sleep until next
time
Sleep until next
Sample Sample Sample
If run queue times are consistent sampling interval will be
preserved but sample times shifted

CPU Run-queue and Wait Latencies
Db file sequential read
Run-queue
On CPU Db file sequential read
Run-queue
On CPU
Event wait time is inflated when host is CPU-bound,
so ASH will sample more/longer waits
User 1
Actual wait time Actual wait time
Recorded wait time Recorded wait time

‘ON CPU’ and ASH
• ASH session status ‘ON CPU’ derived, not observed
• Session is in a database call
• Session is NOT in a wait event (idle or non-idle)
• Un-instrumented waits => ‘ON CPU’
• These are bugs and should be rare, but have happened
• Session on run queue may be ‘WAITING’ or ‘ON CPU’
• Depends on state prior to going onto run queue

ASH Pros and Cons
PROS
• Supports the DB Time method of performance analysis
• Historically (AWR) for large or systemic problems
• Recent (V$) for emergency or monitoring
• Always available (modulo licensing restrictions)
• Minimal cost to server performance
CONS
• Queries become estimates
• We query a sample and not the full data set
• Difficult to form semantically meaningful queries
• Probability distribution of a query result matters

Database Time and Active Sessions
• Database Time = Total time spent by sessions in the
database server actively working (on CPU) or actively
waiting (non-idle wait)
• Active Sessions = The number of sessions active
(working or waiting) in the database server at a
particular time
• Average Active Sessions = DB Time / Elapsed Time

ASH and Database Time
 iST:DBT
DB Time is approximated by multiplying sampling interval (:T) by
the total count of the samples
ASH)in(rowsT:DBT 
Each ASH row represents 1 :T slice of DB Time in
the units of :T

Basic ASH Math
COUNT(*) = DB Time (secs)
GROUP BY ?
Where samples are counted over some [time interval]

Non-default ASH Math
• For V$ACTIVE_SESSION_HISTORY:
• For DBA_HIST_ACTIVE_SESS_HISTORY:
SUM(:T/1000) = DB Time (secs)
:F x SUM(:T/1000) = DB Time (secs)
Where :T is sampling interval in millisecs and :F is the
disk filter ratio

Compare ASH and Time Model
select M.end_time
,ROUND(M.value / 100,3) as Metric_AAS
,ROUND(SUM(DECODE(A.session_type,'FOREGROUND',1,0)) /
((M.end_time - M.begin_time) * 86400 ),3)
as ASH_AAS
,COUNT(1) as ASH_count
from
v$active_session_history A
,v$sysmetric_history M
where
A.sample_time between M.begin_time and M.end_time
and M.metric_name = 'Database Time Per Sec' -- 10g metric
and M.group_id = 2
group by M.end_time,M.begin_time, M.value
order by M.end_time;

Compare ASH and Time Model
0.00
10.00
20.00
30.00
40.00
50.00
60.00
70.00
MDBTIME ASHDBTIME

Compare Estimated to Actual DB Time

Understanding Active Session Charts
• Chart of Average Active Sessions over time
• Broken down by Wait Class
• Green = CPU, Blue = I/O
• Area under curve = amount of DB Time
• Usage Model: “Click on the big stuff”

Active Sessions and DB Time
• The number of active sessions at any time is the rate
of change of the “DB Time function” at that time
• DB Time is the integral of the Active Session function

1
0
t
t
ionsActiveSessDBtime
ionsActiveSesstDBtime  /

timet0 t1
Active Sessions and DB Time
t = 1 sec
Active sessions
DB Time
Why does each ASH sample count for 1 second of DB Time?
(Why does ASH Math work?)

ASH DB Time Query (basic ASH math)
• What are the top 5 SQL by database time in the last 5
minutes?
SELECT * FROM
(SELECT NVL(SQL_ID,'NULL') as SQL_ID
,SUM(1) as DBtime_secs
FROM V$ACTIVE_SESSION_HISTORY
WHERE sample_time > SYSDATE - 5/1440
GROUP BY SQL_ID
ORDER BY 2 DESC
)
WHERE rownum < 6;
Substitute SUM(10) when querying DBA_HIST_ASH

“Click on the big stuff”
• EM usage model for DB Performance
• “click on the largest integral estimate of DB Time”
• The size of colored areas corresponds directly to the
important quantity
• ASH Math 3 Dim combinations ~ 903 = 729,000

ASH Timing for Nano-operations
• Some important operations are still too frequent and
short-lived for timing
• No wait event for “bind” operations
• A session-level bit vector is updated in binary fashion
before/after such operations
• Much cheaper than timer call
• The session bit vector is sampled into ASH
• “ASH Math” allows us to estimate time spent in these
un-timed transient operations

ASH Queries: Estimates Over Samples
• ASH does not contain all activity data, just a sample
• Multiple mechanisms would produce different results from
same true activity
• Aggregates over ASH data are random variables
• They estimate some property of the true activity
• Unbiased estimator means the expected value = true
property value
• Time sampling means ASH event samples are biased
over TIME_WAITED
• 100ms event 10x more likely to sample than 10ms event

Sample Size Matters
• Statistics over larger data sets have more “strength”
• Increases confidence that sample represents reality
• Rigorous sample sizing discussion is beyond scope
• ASH aggregates over fewer rows lose strength
• Beware of drawing conclusions from highly filtered data
• EM and AWR defaults were chosen carefully:
• 5-minute Top Activity detail window is 300 V$ samples
• 30-minute historical mode is 180 AWR samples

Common Mistakes (Bad ASH Math)
AVG(TIME_WAITED)
This does not estimate the average event latencies
because of sampling bias toward longer events
SUM(TIME_WAITED)
This does not compute total wait time in the database
since ASH does not sample all waits
ASH is neither a random sample nor a complete
sample of TIME_WAITED by session events

Bad ASH Math
• SQL observed using 9 secs of CPU every 10 secs

The ASH “Fix-up”
• ASH columns may be unknown at sampling time
• TIME_WAITED: session is still waiting
• PLAN_HASH: session is still optimizing SQL
• GC events: event details unknown at event initiation
• Certain time model bit vector columns
• ASH “fixes up” data during subsequent sampling
• TIME_WAITED fixed up in first sample after event completes
• Long events: last sample gets fixed up time_waited (all others stay 0)
• Querying current ASH may return un-fixed rows
• Should not be a problem generally

ASH Fix-up 1: Sampling
Session State Objects
ASH Sampler
MMNL
ON CPU
ON CPU
ON CPU
WAITING
WAITING
IO WAIT
V$ASH
TIME_WAITED
PLAN_HASH
TIME_WAITED
Fix-up List
Post-wait callback
registered for session
by Sampler

ASH Fix-up 2: Fixing Up
Callback
Session State Objects
ASH Sampler
MMNL
ON CPU
ON CPU
ON CPU
TIME_WAITED=n
WAITING
V$ASH
TIME_WAITED=n
PLAN_HASH
Fix-up List

Event Count Estimates From ASH

Correcting Bad ASH Math
• Why is AVG(TIME_WAITED) so commonly seen?
• Desire to identify average latencies for events
• Has a direct impact on performance
• Some latencies originate externally (e.g. I/O)
• TIME_WAITED can be used in latency estimates
• (it’s just not that simple)

Estimating Event Counts With ASH
:T = ASH sampling interval (millisec)
TIME_WAITED = length of sampled event (microsec)
N = Number of events the ASH sample
represents
:T ~ TIME_WAITED / 1000 * N
OR
N ~ :T * 1000 / TIME_WAITED
NOTE: Matching time units correctly is critical to correctness

Why is Count ~ :T / Time_Waited?
250 msSally
250 ms
250 ms
250 ms
250 ms
User1
User2
User3
User4
tN tN+:T:T elapsed time (1000 ms)
ASH sample time
Sally’s row in ASH estimates :T/250 ms = 4 waits during time interval tN to tN+:T

ASH Event Count Query (1)
• Top 5 objects by I/O requests last 15 minutes
SELECT W.*,O.object_name FROM
(SELECT current_obj#
,ROUND(SUM(1000000/time_waited)) as est_IOwaits
AND time_waited > 0 -- fixed up samples only
AND event IN (’db file sequential read’
,’db file scattered read’)
GROUP BY current_obj#
ORDER BY 2 DESC
) W
, DBA_OBJECTS O
WHERE o.data_object_id = W.current_obj#
AND ROWNUM < 6;
This query works when all waits are “short”

Long Events
• Events that are longer than :T are always sampled
• No sampler “bias” for these
• Event may be sampled multiple times (e.g. 3 second
enqueue)
• Only final (“fixed-up”) ASH row for long events has
TIME_WAITED > 0
• The event count of long events is known:
• 1 for each row with TIME_WAITED >= :T
• The DB Time estimate is still COUNT(*)

ASH Event Count Query (2)
• Top I/O event objects adjusted for long events
SELECT * FROM
(SELECT
current_obj#
,ROUND(SUM(CASE WHEN time_waited >= 1000000 THEN 1
ELSE (1000000/time_waited)
END)) as est_IOwaits
AND time_waited > 0
AND wait_class = 'User I/O'
GROUP BY current_obj#
ORDER BY 2 DESC
)
WHERE ROWNUM < 6;

Estimating Event Latencies
1. We estimate total DB time on events
2. We can now estimate total event occurrences
3. Therefore, we can compute average latency:
est_avg_latency_ms = est_Dbtime_ms / est_waits

User I/O Event Latencies (fixed samples)
SELECT event
,ROUND(est_DBtime_ms/est_waits,1)
as est_avg_latency_ms
FROM
(SELECT event
,ROUND(SUM(GREATEST(1, 1000000/time_waited)))
as est_waits
,SUM(GREATEST(1000, time_waited/1000))
as est_DBtime_ms
AND time_waited > 0 -- fixed-up events only
GROUP BY event
)
ORDER BY 2 DESC;

User I/O Event Latencies (all samples)
SELECT event
,ROUND(est_DBtime_ms/est_waits,1)
as est_avg_latency_ms
FROM
(SELECT event
,ROUND(SUM(CASE WHEN time_waited >0
THEN GREATEST(1, 1000000/time_waited)
ELSE 0 END )) as est_waits
,SUM(1000) as est_DBtime_ms
-- NOTE: all samples, no time_waited>0
GROUP BY event
)
ORDER BY 2 DESC;

Sample Results
EVENT EST_AVG_LATENCY_MS EST_WAITS EST_DBTIME_MS
--------------------------------- ---------------------- ---------------------- ------------------
direct path write 500 2 1000
db file parallel read 135.5 381 51616.854
direct path write temp 36.1 83 3000
db file sequential read 33.8 6903 233000
direct path read 23.7 2193 52000
db file scattered read 13.6 85280 1162000
6 rows selected
EVENT EST_AVG_LATENCY_MS EST_WAITS EST_DBTIME_MS
--------------------------------- ---------------------- ---------------------- ------------------
direct path write 500 2 1000
db file parallel read 136.5 381 52000
direct path write temp 36.1 83 3000
db file sequential read 33.9 6903 234000
direct path read 23.7 2193 52000
db file scattered read 13.6 85280 1163000
6 rows selected

How Bad is Bad ASH Math?
• Bad ASH math overestimates latency significantly
• Sampling biased to longer waits => averaging biased sample
• More samples => bigger issue
• More event latency variation => bigger issue
EVENT_NAME EST_WAIT_COUNT EST_DBTIME_SECS EST_AVG_WAIT_TIME_MS BAD_AVG_TIME_WAITED_MS
------------------------------- ---------------- ---------------- ---------------------- ----------------------
log file sync 86637 3468 40 204.3
log file parallel write 168586 2055 12.2 114.8
db file sequential read 230385 1709 7.4 108.9
db file async I/O submit 17367 1427 82.2 895
control file parallel write 2343 140 59.8 224.8
db file parallel read 15 2 129.5 130.6
direct path sync 8 1 120.6 120.6
7 rows selected

Lessons Learned (so far)
• ASH is time-based (not event-based) sample of
database activity
• ASH is an excellent representation of activity history
• How to estimate and rank DB Time spent over ASH
dimensions using basic ASH Math
• ASH estimates support DB Time performance
analyses exposed by EM
• The ASH fix-up is a critical (and unique) mechanism
• How to estimate event counts and latencies using
TIME_WAITED

Ash architecture and advanced usage rmoug2014

Ash architecture and advanced usage rmoug2014

Recommended

More Related Content

What's hot (20)

Viewers also liked (17)

Similar to Ash architecture and advanced usage rmoug2014 (20)

More from John Beresniewicz (7)

Recently uploaded (20)

Ash architecture and advanced usage rmoug2014