Same plan different performance

Same Plan
Different Performance
Mauro Pagano

SQL is slower….
• Same SQL experiences different performance
in systems that are identical (or supposed to)
• First check the execution plan
• Most of the time plan is different, address it
• But what if the plan is the same?
2

CBO is innocent (this time, maybe)
• Exec plan is where CBO’s job end (kind of)
• Same plan means CBO “worked” the same
• Doesn’t mean everything else IS the same
• Shift focus on next step, SQL execution
3

Apples vs oranges?
• Make sure the comparison is fair (data)
• All external factors should be similar
– CPU should be similar
– IO should be similar
– Memory should be similar
4

“Everything is the same!”
• Plan, data and hardware match, now what?
• Dig into how the SQL is executed
• Wait events and session statistics
• Factors
– configuration, storage layout, load
5

Old friends get-together
• Wait events
– Do they match?
– Are they close in cardinality?
– Do we spend the same time on them?
• Session statistics
– Do they match?
– Are they close in values?
6

Back to the plan for a second
• Exec plan is made of lots of small steps
• Each one produces/handles/consumes rows
• Same behaviors in short and long plans
• Keep it simple, focus on the step
• Remove the noise if possible (reduce TC)
7

Each scenario is a quiz
• SQL is provided
• Changes to the initial setup are disclosed
• Each run in one environment
• Identify what’s different and why
8

Setup
• Linux x86-64, 11.2.0.3
• 1 table, 1M rows, 3 columns, no index
– N1 unique
– N2 100 NDV
– C1 100chars long padded string
• Identical hardware, same DDL to create table
• Controlled environments to isolate behavior
• Simplest SQL to reproduce desired behavior
9

Scenario #1
• SQL
– select /*+ INDEX(TEST1M) */ count(*)
from test1m
where n1 between 1 and 1000000
• Environment
– Added index on N1
10

Scenario #1 – Run (A)
call count cpu elapsed disk query current rows
------- ------ -------- ---------- ---------- ---------- ---------- ----------
Parse 1 0.00 0.00 0 0 0 0
Execute 1 0.00 0.00 0 0 0 0
Fetch 2 0.31 0.45 2228 2228 0 1
------- ------ -------- ---------- ---------- ---------- ---------- ----------
total 4 0.32 0.45 2228 2228 0 1
Rows (1st) Rows (avg) Rows (max) Row Source Operation
---------- ---------- ---------- ---------------------------------------------------
1 1 1 SORT AGGREGATE (cr=2228 pr=2228 pw=0 time=451619 us)
1000000 1000000 1000000 INDEX RANGE SCAN TEST1M_IDX (cr=2228 pr=2228
Event waited on Times Max. Wait Total Waited
---------------------------------------- Waited ---------- ------------
db file sequential read 2228 0.00 0.15
11

Scenario #1 – Run (B)
------- ------ -------- ---------- ---------- ---------- ---------- ----------
Parse 1 0.00 0.00 0 0 0 0
Execute 1 0.00 0.00 0 0 0 0
Fetch 2 0.07 0.08 0 2228 0 1
------- ------ -------- ---------- ---------- ---------- ---------- ----------
total 4 0.08 0.08 0 2228 0 1
---------- ---------- ---------- ---------------------------------------------------
1000000 1000000 1000000 INDEX RANGE SCAN TEST1M_IDX (cr=2228 pr=0 pw=0
12

Scenario #1 Solution
• Buffer Cache cold/warm
• (Part of) the data already in memory
• Reduced number of physical reads (pr)
• Faster performance because less reads
• Number of (same) wait events is lower
• Isolated environment likely to read more
13

Scenario #2
• SQL
– select /*+ FULL(TEST1M) */ count(*)
from test1m
• Environment
– No changes from original setup
14

------- ------ -------- ---------- ---------- ---------- ---------- ----------
Parse 1 0.00 0.01 0 0 0 0
Execute 1 0.00 0.00 0 0 0 0
Fetch 2 0.57 1.51 28574 28584 0 1
------- ------ -------- ---------- ---------- ---------- ---------- ----------
total 4 0.57 1.52 28574 28584 0 1
---------- ---------- ---------- ---------------------------------------------------
1000000 1000000 1000000 TABLE ACCESS FULL TEST1M (cr=28584 pr=28574
---------------------------------------- Waited ---------- ------------
db file scattered read 240 0.02 1.07
15

------- ------ -------- ---------- ---------- ---------- ---------- ----------
Parse 1 0.00 0.00 0 0 0 0
Execute 1 0.00 0.00 0 0 0 0
Fetch 2 1.04 2.42 14286 28583 0 1
------- ------ -------- ---------- ---------- ---------- ---------- ----------
total 4 1.04 2.42 14286 28583 0 1
---------- ---------- ---------- ---------------------------------------------------
---------------------------------------- Waited ---------- ------------
16

Scenario #2 – Run (A) - Waits
WAIT #140245916217600: nam='db file scattered read' ela= 4834 file#=26 block#=16002 blocks=128
17

Scenario #2 – Run (B) - Waits
WAIT #140245916165224: nam='db file sequential read' ela= 124 file#=26 block#=16002 blocks=1
….<<another 38 waits here>>
18

Scenario #2 Solution
• Buffer cache status (cold/warm)
• Reduced number of physical reads (pr)
• Number of (same) wait events is higher
• Wait events details help track it down
– Non-contiguous blocks read
• Slower performance because smaller reads
19

Scenario #3
• SQL
from test1m
• Environment
– No changes
– BC warm
20

------- ------ -------- ---------- ---------- ---------- ---------- ----------
Parse 1 0.00 0.00 0 0 0 0
Execute 1 0.00 0.00 0 0 0 0
Fetch 2 0.92 2.96 14286 28583 0 1
------- ------ -------- ---------- ---------- ---------- ---------- ----------
total 4 0.92 2.96 14286 28583 0 1
---------- ---------- ---------- ---------------------------------------------------
---------------------------------------- Waited ---------- ------------
21

------- ------ -------- ---------- ---------- ---------- ---------- ----------
Parse 1 0.00 0.00 0 0 0 0
Execute 1 0.00 0.00 0 0 0 0
Fetch 2 0.11 1.01 28573 28575 0 1
------- ------ -------- ---------- ---------- ---------- ---------- ----------
total 4 0.11 1.02 28573 28575 0 1
---------- ---------- ---------- ---------------------------------------------------
---------------------------------------- Waited ---------- ------------
enq: KO - fast object checkpoint 2 0.00 0.00
direct path read 179 0.03 0.90
22

Scenario #3 – Solution
• Buffered vs Direct Path reads (different waits too)
• Direct Path
– skips Buffer Cache and reads whole table every time
– consistent performance
– number of wait events is consistent
• Buffered vs Direct Path decision is made AFTER
plan selection (several criteria)
23

Scenario #4
• SQL
from test1m
• Environment
– No changes
– BC cold
24

------- ------ -------- ---------- ---------- ---------- ---------- ----------
Parse 1 0.00 0.01 0 0 0 0
Execute 1 0.00 0.00 0 0 0 0
Fetch 2 0.57 3.08 15872 15884 1 1
------- ------ -------- ---------- ---------- ---------- ---------- ----------
total 4 0.57 3.10 15872 15884 1 1
---------- ---------- ---------- ---------------------------------------------------
---------------------------------------- Waited ---------- ------------
25

------- ------ -------- ---------- ---------- ---------- ---------- ----------
Parse 1 0.00 0.00 0 0 0 0
Execute 1 0.00 0.00 0 0 0 0
Fetch 2 0.32 1.66 15872 15881 0 1
------- ------ -------- ---------- ---------- ---------- ---------- ----------
total 4 0.32 1.66 15872 15881 0 1
---------- ---------- ---------- ---------------------------------------------------
---------------------------------------- Waited ---------- ------------
26

Scenario #4 – Run (A) - Waits
…..
27

Scenario #4 – Run (B) - Waits
28

Scenario #4 – Solution 1
• Different db_file_multiblock_read_count value
• Same number of blocks read from disk
– Contiguous blocks read
29

• Different extent size (64k vs 1M)
• Same number of blocks read from disk
– Contiguous blocks read
• Same params/stats but different storage org
30

Scenario #5
• SQL
from test1m
• Env changes
– No changes
– BC cold, MBRC and extent are identical
31

------- ------ -------- ---------- ---------- ---------- ---------- ----------
Parse 1 0.00 0.00 0 0 0 0
Execute 1 0.00 0.00 0 0 0 0
Fetch 2 0.26 0.72 14285 14297 1 1
------- ------ -------- ---------- ---------- ---------- ---------- ----------
total 4 0.27 0.72 14285 14297 1 1
---------- ---------- ---------- ---------------------------------------------------
---------------------------------------- Waited ---------- ------------
32

------- ------ -------- ---------- ---------- ---------- ---------- ----------
Parse 1 0.00 0.00 0 0 0 0
Execute 1 0.00 0.00 0 0 0 0
Fetch 2 0.44 1.29 28574 28586 1 1
------- ------ -------- ---------- ---------- ---------- ---------- ----------
total 4 0.44 1.29 28574 28586 1 1
---------- ---------- ---------- ---------------------------------------------------
---------------------------------------- Waited ---------- ------------
33

• Different PCTFREE (0 vs 50)
• Higher number of blocks read for same data
• Reads are of the same size hence more reads
• Data is more spread out, room for changes
• Slower performance because more reads
34

• Empty blocks below HWM
• Reads are of the same size hence more reads
• Data has been deleted, FTS reads everything
• Slower performance because more reads
35

Scenario #6
• SQL
from test1m
• Env changes
– No changes
– BC cold, MBRC, PCTFREE and extent are identical
36

------- ------ -------- ---------- ---------- ---------- ---------- ----------
Parse 1 0.00 0.00 0 0 0 0
Execute 1 0.00 0.00 0 0 0 0
Fetch 2 0.44 1.29 28574 28586 1 1
------- ------ -------- ---------- ---------- ---------- ---------- ----------
total 4 0.44 1.29 28574 28586 1 1
---------- ---------- ---------- ---------------------------------------------------
---------------------------------------- Waited ---------- ------------
37

------- ------ -------- ---------- ---------- ---------- ---------- ----------
Parse 1 0.00 0.00 0 0 0 0
Execute 1 0.00 0.00 0 0 0 0
Fetch 2 0.73 2.49 28803 58584 0 1
------- ------ -------- ---------- ---------- ---------- ---------- ----------
total 4 0.74 2.49 28803 58584 0 1
---------- ---------- ---------- ---------------------------------------------------
---------------------------------------- Waited ---------- ------------
cell single block physical read 230 0.01 0.06
38

Scenario #6 – Waits and SesStats
• Wait events show
– single block reads from UNDO tbs for obj#=0
WAIT #140029131327704: nam='db file scattered read' ela= 15412 file#=26 block#=15618 blocks=128 obj#=74828
WAIT #140029131327704: nam='cell single block physical read' ela= 220 … bytes=8192 obj#=0
• v$sesstat shows high
– data blocks consistent reads - undo records applied
39

Scenario #6 - Solution
• Different concurrency/workload
• Waits -> reads from UNDO tbs
• SesStats -> UNDO records applied
• Slower performance because more reads +
more work to recreate the correct image
40

Scenario #7
• SQL
– select /* 1st run */ n1,c1
from test1m
where n1 in (1,1000,5000)
• Env changes
– Index on TEST1M(N1)
– No concurrency at the time SQL is executed
41

Scenario #7 – Why so many cr/pr?
------- ------ -------- ---------- ---------- ---------- ---------- ----------
Parse 1 0.00 0.00 0 0 0 0
Execute 1 0.00 0.00 0 0 0 0
Fetch 2 0.00 0.75 11 18 0 3
------- ------ -------- ---------- ---------- ---------- ---------- ----------
total 4 0.00 0.75 11 18 0 3
---------- ---------- ---------- ---------------------------------------------------
3 3 3 INLIST ITERATOR (cr=18 pr=11 pw=0 time=235681 us)
3 3 3 TABLE ACCESS BY INDEX ROWID TEST1M (cr=18 pr=11
3 3 3 INDEX RANGE SCAN TEST1M_IDX (cr=9 pr=5 pw=0
---------------------------------------- Waited ---------- ------------
42

– single block reads from data tbs, same obj#
WAIT #140…: nam='db file sequential read' ela= 7414 file#=26 block#=2356 blocks=1 obj#=75022
– table fetch continued row
43

• Row migration, index points to original rowid
• Waits -> reads are from data tbs
• SesStats -> table fetch continued row
more work to find all the row pieces
• Similar behavior happens with chained rows
44

Scenario #8
• SQL
– select /* 2nd run */ n1,c1, ora_rowscn
from test1m
where rownum <= 5000
• Env changes
– Index on TEST1M(N1)
– No concurrency at the time SQL is executed
45

Scenario #8 - Why so many seq read?
------- ------ -------- ---------- ---------- ---------- ---------- ----------
Parse 1 0.00 0.00 0 0 0 0
Execute 1 0.00 0.00 0 0 0 0
Fetch 6 0.03 0.22 393 5378 0 5000
------- ------ -------- ---------- ---------- ---------- ---------- ----------
total 8 0.03 0.22 393 5378 0 5000
---------- ---------- ---------- -------------------------------------------
5000 5000 5000 COUNT STOPKEY (cr=5378 pr=393 pw=0 time=91193
---------------------------------------- Waited ---------- ------------
46

– single block reads from data tbs, same obj#
WAIT #1405…: nam='db file scattered read' ela= 6434 file#=26 block#=132 blocks=4 obj#=74828
– table fetch continued row
47

• Row migration, pseudo col needs row header
• Waits -> reads are from data tbs
• SesStats -> table fetch continued row
more work to find all the row pieces
• Similar behavior happens with chained rows
48

Other things to consider
• Same PHV with small differences
– Predicate ordering
– Column projection
• Exadata Optimizations
– Exadata Smart Flash Cache
– Storage indexes
• External to the database
– File system / SAN / Disk caching
– Read-ahead optimizations
49

Conclusions
• Same plan can still run differently
• Storage organization and concurrency impact
• Fix one scenario can introduce another, ie.
– low PCTFREE higher chance of row migration
– high caching slows down buffered mreads
• Find a balance to achieve optimal performance
50

References
• 'DB_FILE_MULTIBLOCK_READ_COUNT' AND EXTENTS
MANAGEMENT (Doc ID 181272.1)
• Higher 'direct path read' Waits in 11g when
Compared to 10g (Doc ID 793845.1)
• Why Is My Query Sometimes Slower Than Other
Times with Higher Consistent Gets Although No
Change in Execution Plan? (Doc ID 1558349.1)
• Row Chaining and Row Migration (Doc ID 122020.1)
52

Contact Information
• http://mauro-pagano.com
– Email
• mauro.pagano@gmail.com
– Download
• SQLd360 vYYMM (date)
– Pages
• SQLd360
53

Same plan different performance

Recommended

More Related Content

What's hot (20)

Similar to Same plan different performance (20)

Recently uploaded (20)

Same plan different performance