Why new hardware may not make Oracle databases faster
2 likes809 views
How can you know if hardware is the right answer to your Oracle database performance issues? How can you know for sure which hardware components will have the biggest impact? As a DBA or database developer, you should know that you can gain significant performance improvements without the time, money and risk associated with providing the latest server or flash storage array. Learn why new hardware may not make your Oracle database faster and what you can do instead.
![DETECT NUMA USE
[oracle@alpha1 ~]$ numactl --hardware
available: 1 nodes (0)
node 0 size: 4095 MB
node 9 free: 648MB
node distances:
node 0
0: 10
[oracle@alpha1 ~]$ numactl --show
policy: default
preferred node: current
physcpubind: 0
nodebind: 0
membind: 0
[dmorgan@lxorap1n5 ~]$ numactl --hardware
available: 2 nodes (0-1)
node 0 size: 48457 MB
node 0 free: 269 MB
node 1 size: 48480 MB
node 1 free: 47 MB
node distances:
node 0 1
0: 10 20
1: 20 10
[dmorgan@lxorap1n5 ~]$ numactl --show
policy: default
preferred node: current
physcpubind: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
cpubind: 0 1
nodebind: 0 1
membind: 0 1
Not Configured
Configured](https://image.slidesharecdn.com/whynewhardwaremaynotmakesqlserverfaster2-151009185143-lva1-app6892/85/Why-new-hardware-may-not-make-Oracle-databases-faster-14-320.jpg)
Ad
More Related Content
What's hot (20)
Similar to Why new hardware may not make Oracle databases faster (20)
![Oracle RAC 12c Practical Performance Management and Tuning OOW13 [CON8825]](https://cdn.slidesharecdn.com/ss_thumbnails/oraclerac12cpracticalperformancemanagementandtuningoow13con8825-131001011452-phpapp01-thumbnail.jpg?width=560&fit=bounds)
Ad
More from SolarWinds (20)
Ad
Recently uploaded (20)
Why new hardware may not make Oracle databases faster
- 1. THROWING HARDWARE AT ORACLE DATABASE PERFORMANCE PROBLEMS? DAN MORGAN, PERFORMANCE TUNING CORP May 13, 2015
- 2. INTRODUCTION: PRESENTER » Oracle ACE Director » Educator Wrote Oracle curriculum and primary program instructor at University of Washington Oracle Consultant: Harvard University Lecturer • APAC: University of Canterbury (NZ) • EMEA: University of Oslo (Norway) • Latin America: Universidad Latina de Panama and Technologico de Costa Rica » The Morgan behind Morgan's Library on the web www.morganslibrary.org » 10g, 11g, and 12c Beta tester » Retired chair Washington Software Assoc. Database SIG » Co-Founder International GoldenGate Oracle Users Group
- 3. INTRODUCTION: PERFORMANCE TUNING CORP. »Boutique Consultancy Specializations Oracle Database 11g and 12c High Availability: RAC, ASM, Data Guard Replication: GoldenGate Focus on Performance Tuning, Security, Stability, Migration, Upgrade, Backup and Recovery » Industry Specializations Telecommunications Healthcare Energy » Special Expertise Knowledge Transfer Training
- 4. THE #1 DATABASE PERFORMANCE TUNING METHODOLOGY Clearly the problem is a bad SQL statement
- 5. THE #1 DATABASE PERFORMANCE TUNING MYTHOLOGY This is from the worst Oracle Database I've seen this century
- 6. ONE SOLUTION
- 7. BAAG COMRADES Christian Antognini Karl Arao Mark Bobak Ronald Bradford Wolfgang Breitling Doug Burns Andrew Clarke Randolf Geist Alex Gorbachev Marco Gralike Frits Hoogland John Hurley Anjo Kolk David Kurtz Jonathan Lewis Robyn Sands Jared Still Jeremiah Wilton BAAG Membership
- 8. CLEARLY » It should be no surprise to anyone that I do not endorse guessing » But I am also not going to endorse AWR Reports » Nor will I support throwing an Exadata at everything » Nor will I stand with anyone that thinks the solution is ... another pretty GUI » So let's take a deep dive into performance tuning beginning with root cause analysis » And focus our efforts on what actually does work » Because Clearly: 30+ years of doing it poorly has not eliminated tuning problems
- 9. WHAT AFFECTS PERFORMANCE » Hardware Servers Storage Subsystems Networks » Software Operating System Configuration Virtual Machines Drivers » Database Memory Allocation Optimizer Configuration SQL Quality » Application Web Servers Application Servers Middleware Caching Application Code Quality Let's focus on what is important but not on the radar
- 10. HARDWARE » Servers Blade Servers I/O Cards NUMA Architecture Swapiness » Storage Controllers Read-Write Caches LUN Size and Layout » Networks TCP/IP UDP
- 11. BLADE SERVERS » Suitable for web and application servers » Possibly usable for small databases » A meltdown waiting to happen with RAC RAC: Frequently Asked Questions (Doc ID 220970.1) Cluster interconnect network separation can be satisfied either by using standalone, dedicated switches, which provide the highest degree of network isolation, or Virtual Local Area Networks defined on the Ethernet switch, which provide broadcast domain isolation between IP networks. VLANs are fully supported for Oracle Clusterware interconnect deployments. Partitioning the Ethernet switch with VLANs allows for: - Sharing the same switch for private and public communication. - Sharing the same switch for the private communication of more than one cluster. - Sharing the same switch for private communication and shared storage access. The following best practices should be followed: The Cluster Interconnect VLAN must be on a non-routed IP subnet. All Cluster Interconnect networks must be configured with non-routed IPs. The server-server communication should be single hop through the switch via the interconnect VLAN. There is no VLAN-VLAN communication. Oracle recommends maintaining a 1:1 mapping of subnet to VLAN. The most common VLAN deployments maintain a 1:1 mapping of subnet to VLAN. It is strongly recommended to avoid multi- subnet mapping to a single VLAN. Best practice recommends a single access VLAN port configured on the switch for the cluster interconnect VLAN. The server side network interface should have access to a single VLAN.
- 12. BLADE SERVERS » Blade servers, of which Cisco UCS is one example, do not have sufficient independent network cards to avoid the networking becoming a single point of failure » It is good when the public interface has a "keep alive" enabled but this is a fatal flaw for the cluster interconnect as fail-over will have a substantial delay » When different types of packets, public, storage, and interconnect are mixed low-level diagnostics are difficult if not impossible » When different types of packets, public, storage, and interconnect are mixed the latency of one is the latency of all » Traffic from any one blade can impact all blades Troubleshooting gc block lost and Poor Network Performance in a RAC Environment (Doc ID 563566.1) 6. Interconnect LAN non-dedicated Description: Shared public IP traffic and/or shared NAS IP traffic, configured on the interconnect LAN will result in degraded application performance, network congestion and, in extreme cases, global cache block loss. Action: The interconnect/clusterware traffic should be on a dedicated LAN defined by a non-routed subnet. Interconnect traffic should be isolated to the adjacent switch(es), e.g. interconnect traffic should not extend beyond the access layer switch(es) to which the links are attached. The interconnect traffic should not be shared with public or NAS traffic. If Virtual LANs (VLANS) are used, the interconnect should be on a single, dedicated VLAN mapped to a dedicated, non-routed subnet, which is isolated from public or NAS traffic.
- 13. NUMA MEMORY ALLOCATION » Non-Uniform Memory Access A memory design used in multiprocessing, where the memory access time depends on the memory location relative to the processor A processor can access its own local memory faster than non- local memory The benefits of NUMA are limited to particular workloads, notably on servers where the data are often associated strongly with certain tasks or users Diagram Source: Wikipedia
- 14. DETECT NUMA USE [oracle@alpha1 ~]$ numactl --hardware available: 1 nodes (0) node 0 size: 4095 MB node 9 free: 648MB node distances: node 0 0: 10 [oracle@alpha1 ~]$ numactl --show policy: default preferred node: current physcpubind: 0 nodebind: 0 membind: 0 [dmorgan@lxorap1n5 ~]$ numactl --hardware available: 2 nodes (0-1) node 0 size: 48457 MB node 0 free: 269 MB node 1 size: 48480 MB node 1 free: 47 MB node distances: node 0 1 0: 10 20 1: 20 10 [dmorgan@lxorap1n5 ~]$ numactl --show policy: default preferred node: current physcpubind: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 cpubind: 0 1 nodebind: 0 1 membind: 0 1 Not Configured Configured
- 15. IS YOUR DATABASE NUMA AWARE? SQL> SELECT a.ksppinm PNAME, c.ksppstvl PVAL, a.ksppdesc PDESC 2 FROM x$ksppi a, x$ksppcv b, x$ksppsv c 3 WHERE a.indx = b.indx 4 AND a.indx = c.indx 5 AND LOWER(a.ksppinm) LIKE '%numa%' 6* ORDER BY 1 SQL> / PNAME PVAL PDESC -------------------------- -------------- ------------------------------------------------------------------ _NUMA_instance_mapping Not specified Set of nodes that this instance should run on _NUMA_pool_size Not specified aggregate size in bytes of NUMA pool _db_block_numa 1 Number of NUMA nodes _enable_NUMA_interleave TRUE Enable NUMA interleave mode _enable_NUMA_optimization FALSE Enable NUMA specific optimizations _enable_NUMA_support FALSE Enable NUMA support and optimizations _numa_buffer_cache_stats 0 Configure NUMA buffer cache stats _numa_shift_enabled TRUE Enable NUMA shift _numa_shift_value 0 user defined value for numa nodes shift _numa_trace_level 0 numa trace event _px_numa_stealing_enabled TRUE enable/disable PQ granule stealing across NUMA nodes _px_numa_support_enabled FALSE enable/disable PQ NUMA support _rm_numa_sched_enable FALSE Is Resource Manager (RM) related NUMA scheduled policy enabled _rm_numa_simulation_cpus 0 number of cpus for each pg for numa simulation in resource manager _rm_numa_simulation_pgs 0 number of PGs for numa simulation in resource manager
- 16. ENABLE DATABASE NUMA SUPPORT conn / as sysdba ALTER SYSTEM SET "_enable_NUMA_support" = TRUE COMMENT= 'NUMA Support Enabled 15-Mar-2015' CONTAINER=ALL SCOPE=SPFILE SID='*';
- 17. » Also known as "Large Memory Pages" or just "Large Pages" » Each page table entry represents a “virtual to physical” translation of a process’s memory » Can be as large as 64 KB in size per entry » Can be huge for large memory systems See PageTables in /proc/meminfo As large a 1.5 GB » The entire SGA must fit inside the HugePages. If it does not fit none of it will use the HugePage memory and you will essentially have walled your database off from using a large portion of the server's memory HUGEPAGES
- 18. » Swapping (aka Paging) » In a sense the operating system's version of the Oracle Temp tablespace SWAPINESS Value Strategy vm.swapiness=0 The kernel will swap only to avoid an out of memory condition vm.swapiness=60 The default value vm.swapiness=100 The kernel will swap aggressively
- 19. OPTIMIZE TCP/IP STACK HANDLING --enable TCP kernel auto-tuning /proc/sys/net/ipv4/tcp_moderate_rcvbuf (1=on) -- tune TCP max memory: tune to 2xBDP (Bandwidth x Delay Product) -- For example, with 40 Mbits/sec bandwidth, 25 msec delay, -- BDP = (40 x 1000 / 8 Kbytes/sec) x (0.025 sec) ~ 128 Kbytes /proc/sys/net/ipv4/tcp_rmem and tcp_wmem 4096 87380 174760 -- tune the socket buffer sizes by setting to 2xBDP /proc/sys/net/core/rmem_max and wmem_max -- ensure that TCP Performance features are enabled /proc/sys/net/ipv4/tcp_sack /proc/sys/net/ipv4/tcp_window_scaling /proc/sys/net/ipv4/tcp_timestamps -- additionally be sure NIC cards have TCP off-loading capability
- 20. » Leave sufficient cpu resources for the bare-metal operating system to perform I/O and manage network traffic » Disable interrupt coalescing » Disable chipset power management » Read the docs http://www.vmware.com/pdf/Perf_Best_Practices_vSphere5.0.pdf » VMs on NUMA machines should be configured to enhance memory allocation This example is from vSphere where 0 and 1 are the processor sockets VIRTUAL MACHINES numa.nodeAffinity=0,1
- 21. STORAGE HARDWARE » Storage Spinning Disk Solid State Devices Controllers SAN Switches
- 22. STORAGE HEAT MAP
- 23. STORAGE LAYOUT LUN 204 LUN 200 LUN 205 LUN 208 LUN 74 LUN 94 LUN 81 LUN 78 LUN 101 LUN 57 LUN 123 LUN 82 LUN 77 LUN 35 LUN 92 LUN 62 LUN 106 LUN 107 LUN 51 LUN 102 LUN 105 LUN 49 LUN 104 LUN 53 LUN 58 LUN 47 LUN 126 LUN 103 LUN 85 LUN 206 LUN 124 LUN 130 LUN 84 LUN 43 LUN 44 LUN 56 LUN 45 LUN 67 LUN 14 LUN 17 LUN 30 LUN 19 LUN 88 LUN 207 LUN 26 LUN 8 LUN 2 LUN 4 LUN LUN 1 LUN 1 LUN 1 LUN 1 LUN 1 LUN 5 LUN 6 LUN 5 LUN LUN LUN LUN LUN LUN LUN LUN LUN LUN LUN LUN LUN LU LU LU LUN LUN LUN LUN LU LU LU LU LU LU LU LU L L L L L L L L L L L L L L L L L c0vmware01p, c0vmware... c0vmware02p, c0vmware... c0ora01p c0odsrac01p, c0odsrac02p c0ods04p c0orarac01p,c0orarac... c0visdb03p, c0visdb04p c0orademand01p c0vmware13p, c0vmware... c0file01p, c0file02p,... c0visrptdb01p c0baltimore01p c0ws01p, c0ws02p c0orademand02t
- 24. OBJECT OPTIMIZATIONS » Default tablespace settings waste disk and cause additional I/O Often a smallfile tablespace is a bad choice » Default table creations are almost always inefficient Often heap tables are a bad choice » Not every index should be a default B*Tree And poor choices lead to poor performance » Oracle sequence object defaults are almost always inefficient A default cache size of 20 is 99.9% of the time a bad choice » The vast majority of PL/SQL I see written would have been valid in version 7.3.4 I've almost never seen constants declared properly
- 25. © 2014 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. I'M NOT AFRAID TO SHOW YOU MINE
- 26. © 2014 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. PAGE ONE
- 27. © 2014 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. PAGE TWO
- 28. © 2014 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. PAGE THREE
- 29. © 2014 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. PAGE FOUR
- 30. © 2014 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. PAGE FIVE
- 31. © 2014 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. OPTIMIZER PLANS SELECT DISTINCT E1_2.OBJECT_ID FROM PMCM.ELEMENT_DETAIL E1_1, PMCM.ELEMENT_DETAIL E1_2, PMCM.MARK_NETW_HIERARCHY H1, PMCM.ELEMENT_DETAIL E2_1, PMCM.ELEMENT_DETAIL E2_2, PMCM.MARK_NETW_HIERARCHY H2 WHERE E1_1.OBJECT_ID = H1.PARENT_ID AND E1_2.OBJECT_ID = H1.OBJECT_ID AND E2_1.OBJECT_ID = H2.PARENT_ID AND E2_2.OBJECT_ID = H2.OBJECT_ID AND E1_1.CURRENT_IND = 'Y' AND E2_1.CURRENT_IND = 'Y' AND E2_1.CURRENT_IND = 'Y' AND E2_2.CURRENT_IND = 'Y' AND H1.CURRENT_IND = 'Y' AND H2.CURRENT_IND = 'Y' AND H1.HIERARCHY_TYPE = 'NETWORK' AND H2.HIERARCHY_TYPE = 'NETWORK' AND H1.PARENT_TYPE IN ('BSC','RNC') AND H2.PARENT_TYPE IN ('BSC','RNC') AND E2_2.ELEMENT_TYPE = 'CELL' AND E1_2.ELEMENT_TYPE = 'CELL' AND H1.PARENT_TYPE IN ('BSC','RNC') AND E1_1.ELEMENT_NAME = E2_1.ELEMENT_NAME AND E1_1.ELEMENT_ID = E2_1.ELEMENT_ID AND E1_2.ELEMENT_NAME = E2_2.ELEMENT_NAME AND E1_2.ELEMENT_ID = E2_2.ELEMENT_ID AND E1_2.USEID LIKE '*%' AND E2_2.USEID NOT LIKE '*%'; | Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time | Pstart| Pstop | | 0 | SELECT STATEMENT | | 1 | 78 | | 74M (40)| 50:54:42 | | | | 1 | TEMP TABLE TRANSFORMATION | | | | | | | | | | 2 | LOAD AS SELECT | | | | | | | | | | 3 | PARTITION RANGE ALL | | 22M| 1111M| | 38153 (11)| 00:01:34 | 1 | 29 | |* 4 | TABLE ACCESS FULL | ELEMENT_DETAIL | 22M| 1111M| | 38153 (11)| 00:01:34 | | | | 5 | LOAD AS SELECT | | | | | | | | | | 6 | PARTITION HASH ALL | | 337K| 9231K| | 3514 (15)| 00:00:09 | 1 | 16 | |* 7 | TABLE ACCESS FULL | MARK_NETW_HIERARCHY | 337K| 9231K| | 3514 (15)| 00:00:09 | | | | 8 | SORT AGGREGATE | | 1 | 78 | | | | | | |* 9 | HASH JOIN | | 927G| 65T| 534M| 74M (40)| 50:53:00 | | | | 10 | VIEW | | 22M| 277M| | 16808 (12)| 00:00:42 | | | | 11 | TABLE ACCESS FULL | SYS_TEMP_0FDA7485F_6A66C42E | 22M| 1111M| | 16808 (12)| 00 | | | |* 12 | HASH JOIN | | 21G| 1272G| 534M| 1616K (43)| 01:06:04 | | | | 13 | VIEW | | 22M| 277M| | 16808 (12)| 00:00:42 | | | | 14 | TABLE ACCESS FULL | SYS_TEMP_0FDA7485F_6A66C42E | 22M| 1111M| | 16808 (12)| 0 | | | |* 15 | HASH JOIN | | 476M| 23G| 524M| 97327 (22)| 00:03:59 | | | |* 16 | HASH JOIN | | 10M| 401M| 8704K| 34520 (10)| 00:01:25 | | | |* 17 | HASH JOIN | | 234K| 5948K| 8256K| 783 (10)| 00:00:02 | | | | 18 | VIEW | | 337K| 4286K| | 142 (14)| 00:00:01 | | | | 19 | TABLE ACCESS FULL | SYS_TEMP_0FDA74860_6A66C42E | 337K| 3956K| | 142 (14)| 00:00:01 | | | | 20 | VIEW | | 337K| 4286K| | 142 (14)| 00:00:01 | | | | 21 | TABLE ACCESS FULL | SYS_TEMP_0FDA74860_6A66C42E | 337K| 3956K| | 142 (14)| 00:00:01 | | | | 22 | VIEW | | 22M| 277M| | 16808 (12)| 00:00:42 | | | | 23 | TABLE ACCESS FULL | SYS_TEMP_0FDA7485F_6A66C42E | 22M| 1111M| | 16808 (12)| 00:00:42 | | | | 24 | VIEW | | 22M| 277M| | 16808 (12)| 00:00:42 | | | | 25 | TABLE ACCESS FULL | SYS_TEMP_0FDA7485F_6A66C42E | 22M| 1111M| | 16808 (12)| 0 | | |
- 32. © 2014 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. OPTIMIZER PLANS ---------------------------------------------------------------------------------------------------------------------------------- | Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time | Pstart| Pstop | ---------------------------------------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | 78 | | 14T(100)|999:59:59 | | | | 1 | TEMP TABLE TRANSFORMATION | | | | | | | | | | 2 | LOAD AS SELECT | | | | | | | | | | 3 | PARTITION RANGE ALL | | 22M| 1111M| | 38153 (11)| 00:01:34 | 1 | 29 | |* 4 | TABLE ACCESS FULL | ELEMENT_DETAIL | 22M| 1111M| | 38153 (11)| 00:01:34 | | | | 5 | LOAD AS SELECT | | | | | | | | | | 6 | PARTITION HASH ALL | | 337K| 9231K| | 3514 (15)| 00:00:09 | 1 | 16 | |* 7 | TABLE ACCESS FULL | MARK_NETW_HIERARCHY | 337K| 9231K| | 3514 (15)| 00:00:09 | | | | 8 | SORT AGGREGATE | | 1 | 78 | | | | | | | 9 | MERGE JOIN | | 471P| 15E| | 14T(100)|999:59:59 | | | | 10 | MERGE JOIN | | 10P| 616P| | 694G (81)|999:59:59 | | | | 11 | MERGE JOIN | | 231T| 10P| | 377G (64)|999:59:59 | | | | 12 | SORT JOIN | | 334T| 11P| 28P| 377G (64)|999:59:59 | | | | 13 | MERGE JOIN CARTESIAN| | 334T| 11P| | 140G (14)|999:59:59 | | | |* 14 | HASH JOIN | | 989M| 23G| 534M| 96010 (38)| 00:03:56 | | | | 15 | VIEW | | 22M| 277M| | 16808 (12)| 00:00:42 | | | | 16 | TABLE ACCESS FULL| SYS_TEMP_0FDA7485B_6A66C42E | 22M| 1111M| | 16808 (12)| 00:00:42 | | | | 17 | VIEW | | 22M| 277M| | 16808 (12)| 00:00:42 | | | | 18 | TABLE ACCESS FULL| SYS_TEMP_0FDA7485B_6A66C42E | 22M| 1111M| | 16808 (12)| 00:00:42 | | | | 19 | BUFFER SORT | | 337K| 4286K| | 140G (14)|999:59:59 | | | | 20 | VIEW | | 337K| 4286K| | 142 (14)| 00:00:01 | | | | 21 | TABLE ACCESS FULL| SYS_TEMP_0FDA7485C_6A66C42E | 337K| 3956K| | 142 (14)| 00:00:01 | | | |* 22 | SORT JOIN | | 337K| 4286K| 12M| 844 (14)| 00:00:03 | | | | 23 | VIEW | | 337K| 4286K| | 142 (14)| 00:00:01 | | | | 24 | TABLE ACCESS FULL | SYS_TEMP_0FDA7485C_6A66C42E | 337K| 3956K| | 142 (14)| 00:00:01 | | | |* 25 | SORT JOIN | | 22M| 277M| 855M| 65084 (16)| 00:02:40 | | | | 26 | VIEW | | 22M| 277M| | 16808 (12)| 00:00:42 | | | | 27 | TABLE ACCESS FULL | SYS_TEMP_0FDA7485B_6A66C42E | 22M| 1111M| | 16808 (12)| 0 | | | |* 28 | SORT JOIN | | 22M| 277M| 855M| 65084 (16)| 00:02:40 | | | | 29 | VIEW | | 22M| 277M| | 16808 (12)| 00:00:42 | | | | 30 | TABLE ACCESS FULL | SYS_TEMP_0FDA7485B_6A66C42E | 22M| 1111M| | 16808 (12)| 0 | | | ----------------------------------------------------------------------------------------------------------------------------------
- 33. © 2014 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. OPTIMIZER PLANS WITH ed AS (SELECT object_id, element_id, element_name, element_type, useid FROM pmcm.element_detail WHERE element_type = 'CELL' AND current_ind = 'Y'), mnh AS (SELECT parent_id, object_id FROM pmcm.mark_netw_hierarchy WHERE current_ind = 'Y' AND hierarchy_type = 'NETWORK' AND parent_type IN ('BSC', 'RNC')) SELECT COUNT(*) FROM ed e1_1, ed e1_2, ed e2_1, ed e2_2, mnh h1, mnh h2 WHERE e1_1.object_id = h1.parent_id AND e1_2.object_id = h1.object_id AND e2_1.object_id = h2.parent_id AND e2_2.object_id = h2.object_id AND e1_1.element_name = e2_1.element_name AND e1_1.element_id = e2_1.element_id AND e1_2.element_name = e2_2.element_name AND e1_2.element_id = e2_2.element_id AND e1_2.useid LIKE '*%' AND e2_2.useid NOT LIKE '*%'; ----------------------------------------------------------------------------------------------------------------------- | Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time | ------------------------------------------------------------------------------------------------------------------------ | 0 | SELECT STATEMENT | | 1 | 214 | | 100K (6)| 00:04:08 | | 1 | HASH UNIQUE | | 1 | 214 | | 100K (6)| 00:04:08 | |* 2 | HASH JOIN | | 1 | 214 | 12M| 100K (6)| 00:04:08 | | 3 | PARTITION HASH ALL | | 337K| 9231K| | 3514 (15)| 00:00:09 | |* 4 | TABLE ACCESS FULL | MARK_NETW_HIERARCHY | 337K| 9231K| | 3514 (15)| 00:00:00 | |* 5 | HASH JOIN | | 207K| 36M| 22M| 95860 (6)| 00:03:56 | | 6 | PARTITION RANGE ALL | | 586K| 15M| | 16233 (2)| 00:00:40 | | 7 | TABLE ACCESS BY LOCAL INDEX ROWID | ELEMENT_DETAIL | 586K| 15M| | 16233 | ??:??:?? | |* 8 | INDEX SKIP SCAN | ED_ET_TECH_CI | 586K| | | 12791 (1)| 00:00:3? | |* 9 | HASH JOIN | | 207K| 31M| 22M| 77982 (7)| 00:03:12 | | 10 | PARTITION RANGE ALL | | 586K| 15M| | 16233 (2)| 00:00:40 | | 11 | TABLE ACCESS BY LOCAL INDEX ROWID | ELEMENT_DETAIL | 586K| 15M| | 16233 | ??:??:?? | |* 12 | INDEX SKIP SCAN | ED_ET_TECH_CI | 586K| | | 12791 (1)| 00:00:?? | |* 13 | HASH JOIN | | 179K| 22M| 12M| 60372 (8)| 00:02:29 | | 14 | PARTITION HASH ALL | | 337K| 9231K| | 3514 (15)| 00:00:09 | |* 15 | TABLE ACCESS FULL | MARK_NETW_HIERARCHY | 337K| 9231K| | 3514 (15)| 00:00:?? | |* 16 | HASH JOIN | | 184K| 17M| 10M| 55886 (8)| 00:02:18 | | 17 | PARTITION RANGE ALL | | 184K| 9008K| | 37137 (8)| 00:01:32 | |* 18 | TABLE ACCESS FULL | ELEMENT_DETAIL | 184K| 9008K| | 37137 (8)| 00:01:32 | | 19 | PARTITION RANGE ALL | | 576K| 28M| | 17383 (8)| 00:00:43 | |* 20 | TABLE ACCESS BY LOCAL INDEX ROWID| ELEMENT_DETAIL | 576K| 28M| | 17383 (8)| ??:??:?? | |* 21 | INDEX SKIP SCAN | ED_ET_TECH_CI | 583K| | | 13939 (9)| 00:00:35 | ------------------------------------------------------------------------------------------------------------------------
- 34. © 2014 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. OPTIMIZER PLANS --------------------------------------------------------------------------------------------------------------------------------- | Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time | --------------------------------------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | | | | 264T(100)| | |* 1 | VIEW | | 156P| 15E| | 264T (79)|999:59:59 | |* 2 | WINDOW SORT PUSHED RANK | | 156P| 15E| 15E| 264T (79)|999:59:59 | | 3 | MERGE JOIN CARTESIAN | | 156P| 15E| | 68T (16)|999:59:59 | | 4 | MERGE JOIN CARTESIAN | | 220G| 205T| | 96M (16)| 26:57:48 | | 5 | MERGE JOIN CARTESIAN | | 310K| 302M| | 232 (11)| 00:00:01 | | 6 | MERGE JOIN CARTESIAN | | 779 | 777K| | 22 (0)| 00:00:01 | | 7 | NESTED LOOPS | | | | | | | | 8 | NESTED LOOPS | | 2 | 2044 | | 20 (0)| 00:00:01 | | 9 | NESTED LOOPS OUTER | | 2 | 1990 | | 18 (0)| 00:00:01 | | 10 | NESTED LOOPS | | 2 | 1868 | | 17 (0)| 00:00:01 | | 11 | NESTED LOOPS | | 2 | 1712 | | 15 (0)| 00:00:01 | | 12 | NESTED LOOPS | | 2 | 1564 | | 13 (0)| 00:00:01 | | 13 | MERGE JOIN CARTESIAN | | 2 | 1442 | | 11 (0)| 00:00:01 | | 14 | NESTED LOOPS OUTER | | 1 | 625 | | 8 (0)| 00:00:01 | | 15 | NESTED LOOPS OUTER | | 1 | 613 | | 7 (0)| 00:00:01 | | 16 | NESTED LOOPS | | 1 | 580 | | 6 (0)| 00:00:01 | | 17 | NESTED LOOPS OUTER | | 1 | 539 | | 5 (0)| 00:00:01 | | 18 | NESTED LOOPS OUTER | | 1 | 340 | | 5 (0)| 00:00:01 | | 19 | TABLE ACCESS BY INDEX ROWID| PA_STUDENT | 1 | 316 | | 3 (0)| 00:00:01 | |* 20 | INDEX UNIQUE SCAN | PK_STUDENT | 1 | | | 2 (0)| 00:00:01 | | 21 | TABLE ACCESS BY INDEX ROWID| PA_STUD_USER | 1 | 24 | | 2 (0)| 00:00:01 | |* 22 | INDEX UNIQUE SCAN | PK_STUD_USER | 1 | | | 1 (0)| 00:00:01 | | 23 | TABLE ACCESS BY INDEX ROWID | PA_ORG | 1 | 199 | | 0 (0)| | |* 24 | INDEX UNIQUE SCAN | PK_ORG | 1 | | | 0 (0)| | | 25 | TABLE ACCESS BY INDEX ROWID | PA_DOMAIN | 13 | 533 | | 1 (0)| 00:00:01 | |* 26 | INDEX UNIQUE SCAN | PK_DOMAIN | 1 | | | 0 (0)| | | 27 | TABLE ACCESS BY INDEX ROWID | PA_USRRF_STUD | 100 | 3300 | | 1 (0)| 00:00:01 | |* 28 | INDEX UNIQUE SCAN | PK_USRRF_STUD | 1 | | | 0 (0)| | | 29 | VIEW PUSHED PREDICATE | PV_STUD_USER | 1 | 12 | | 1 (0)| 00:00:01 | |* 30 | FILTER | | | | | | | | 31 | NESTED LOOPS OUTER | | 1 | 22 | | 264 (11)| 00:00:01 | |* 32 | INDEX UNIQUE SCAN | PK_STUDENT | 1 | 10 | | 2 (0)| 00:00:01 | |* 33 | MAT_VIEW ACCESS FULL | PV_AP_STUD_USER | 1 | 12 | | 262 (11)| 00:00:01 | | 34 | BUFFER SORT | | 2 | 192 | | 10 (0)| 00:00:01 | | 35 | TABLE ACCESS BY INDEX ROWID | PA_CPNT_COMPLIANCE_DATA | 2 | 192 | | 3 (0)| 00:00:01 | |* 36 | INDEX RANGE SCAN | IX_CPNT_CD__EVTHST | 2 | | | 1 (0)| 00:00:01 | | 37 | TABLE ACCESS BY INDEX ROWID | PA_CPNT_TYPE | 1 | 61 | | 1 (0)| 00:00:01 | |* 38 | INDEX UNIQUE SCAN | PK_CPNT_TYPE | 1 | | | 0 (0)| | | 39 | TABLE ACCESS BY INDEX ROWID | PA_RQMT_TYPE | 1 | 74 | | 1 (0)| 00:00:01 | |* 40 | INDEX UNIQUE SCAN | PK_RQMT_TYPE | 1 | | | 0 (0)| | | 41 | TABLE ACCESS BY INDEX ROWID | PA_CMPL_STAT | 1 | 78 | | 1 (0)| 00:00:01 | |* 42 | INDEX UNIQUE SCAN | PK_CMPL_STAT | 1 | | | 0 (0)| | | 43 | TABLE ACCESS BY INDEX ROWID | PA_QUAL | 1 | 61 | | 1 (0)| 00:00:01 | |* 44 | INDEX UNIQUE SCAN | PK_QUAL | 1 | | | 0 (0)| | |* 45 | INDEX UNIQUE SCAN | PK_CPNT | 1 | | | 0 (0)| | | 46 | TABLE ACCESS BY INDEX ROWID | PA_CPNT | 1 | 27 | | 1 (0)| 00:00:01 | | 47 | BUFFER SORT | | 399 | | | 21 (0)| 00:00:01 | | 48 | INDEX FAST FULL SCAN | PK_USRRF_STUD | 399 | | | 1 (0)| 00:00:01 | | 49 | BUFFER SORT | | 399 | | | 231 (11)| 00:00:01 | | 50 | INDEX FAST FULL SCAN | PK_USRRF_STUD | 399 | | | 0 (0)| | | 51 | BUFFER SORT | | 710K| | | 96M (16)| 26:57:48 | | 52 | INDEX FAST FULL SCAN | IX_STUD_USER__STUDENT | 710K| | | 309 (16)| 00:00:01 | | 53 | BUFFER SORT | | 710K| | | 264T (79)|999:59:59 | | 54 | INDEX FAST FULL SCAN | IX_STUD_USER__STUDENT | 710K| | | 309 (16)| 00:00:01 | ---------------------------------------------------------------------------------------------------------------------------------
- 35. © 2014 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. POORLY WRITTEN APPLICATIONS SELECT /*+ RESULT_CACHE */ srvr_id FROM ( SELECT srvr_id, SUM(cnt) SUMCNT FROM ( SELECT DISTINCT srvr_id, 1 AS CNT FROM servers UNION ALL SELECT DISTINCT srvr_id, 1 FROM serv_inst) GROUP BY srvr_id) WHERE sumcnt = 2;
- 36. SQL> show parameter optimizer_mode NAME TYPE VALUE ------------------------------------ ----------- --------- optimizer_mode string ALL_ROWS SQL> ALTER SYSTEM SET OPTIMIZER_MODE='FIRST_ROWS_10'; System altered. SQL> show parameter optimizer_mode NAME TYPE VALUE ------------------------ ----------- -------------- optimizer_mode string FIRST_ROWS_10 OPTIMIZER SETTINGS » Default Oracle install favors data warehouse not OLTP NAME TYPE VALUE ------------------------------------ ----------- ----------- optimizer_adaptive_features boolean TRUE optimizer_adaptive_reporting_only boolean FALSE optimizer_capture_sql_plan_baselines boolean FALSE optimizer_dynamic_sampling integer 2 optimizer_features_enable string 12.1.0.1 optimizer_index_caching integer 0 optimizer_index_cost_adj integer 100 optimizer_mode string ALL_ROWS optimizer_secure_view_merging boolean TRUE optimizer_use_invisible_indexes boolean FALSE optimizer_use_pending_statistics boolean FALSE optimizer_use_sql_plan_baselines boolean TRUE
- 37. OPTIMIZER STATISTICS » Gather System Stats » Gather Fixed Object Stats » Gather Dictionary Stats » Gather Table Stats » Gather Column Stats » Gather Index Stats » Gather Extended Stats » Stat Generation Copy Stats Set Stats Fixing Stats SQL> select type, count(*) 2 from v$fixed_table 3 group by type 4 order by 1; TYPE COUNT(*) ----- ---------- TABLE 1144 VIEW 1261 SQL> select name 2 from v$fixed_table 3 where rownum < 11; NAME -------------------------- X$KQFTA X$KQFVI X$KQFVT X$KQFDT X$KQFCO X$KQFOPT X$KYWMPCTAB X$KYWMWRCTAB X$KYWMCLTAB X$KYWMNF
- 38. GATHERING OPTIMIZER STATISTICS SELECT dbms_stats.create_extended_stats(USER, 'SERV_INST', '(srvr_id, si_status)') FROM dual; exec dbms_stats.set_table_prefs(USER, 'SERVERS', 'ESTIMATE_PERCENT','90'); exec dbms_stats.set_table_prefs(USER, 'SERVERS', 'DEGREE','8'); exec dbms_stats.gather_table_stats(USER, 'SERVERS', 'P2010103106', estimate_percent => dbms_stats.auto_sample_size, block_sample=>TRUE, method_opt=>'FOR ALL INDEXED COLUMNS', degree=>2, cascade=>TRUE force=>TRUE);
- 39. MANUFACTURING OPTIMIZER STATISTICS exec dbms_stats.set_table_stats(USER, 'EMP', numrows=>1000000, numblks=>10000, avgrlen=>74); exec dbms_stats.set_index_stats(USER, 'ix_emp_deptno', numrows=>1000000, numlblks=>1000, numdist=>10000, clstfct=>1); exec dbms_stats.set_column_stats(USER, 'emp', 'deptno', distcnt=>10000); exec dbms_stats.set_table_stats(USER, 'dept', numrows=>100, numblks=>100);
- 40. ROOT CAUSE ANALYSIS BY SOPHISTICATED GUESSING » Check data dictionary for collected stats » Explain Plans Very few people can read them » AWR Reports Most AWR Reports produced are just a pile of numbers of no value » ASH Reports » SQL Tuning Advisor » SQL Trace and TKPROF » Baselines and Evolving Baselines » DBMS_TRACE, DBMS_MONITOR, TRCSESS, TRACE ANALYZER, SQLTXPLAIN
- 41. » Very few people can read an explain plan » Here are two plans from identical tables with identical stats » Which one would you rely on? READING EXPLAIN PLANS -------------------------------------------------------------------------- | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| -------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 141 | 4560 | 6 (84)| | 1 | INTERSECTION | | | | | | 2 | SORT UNIQUE NOSORT | | 141 | 564 | 2 (50)| | 3 | INDEX FULL SCAN | PK_SERVERS | 141 | 564 | 1 (0)| | 4 | SORT UNIQUE | | 999 | 3996 | 4 (25)| | 5 | INDEX FAST FULL SCAN| IX_SERV_INST | 999 | 3996 | 3 (0)| -------------------------------------------------------------------------- --------------------------------------------------------------------------- | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| --------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 141 | 4560 | 20 (10)| | 1 | INTERSECTION | | | | | | 2 | SORT UNIQUE | | 141 | 564 | 10 (10)| | 3 | TABLE ACCESS FULL | SERVERS | 141 | 564 | 9 (0)| | 4 | SORT UNIQUE | | 999 | 3996 | 10 (10)| | 5 | TABLE ACCESS FULL | SERV_INST | 999 | 3996 | 9 (0)| --------------------------------------------------------------------------- EXPLAIN PLAN FOR SELECT srvr_id FROM servers INTERSECT SELECT srvr_id FROM serv_inst; SQL> SELECT table_name, blocks 2 FROM user_tables 3* WHERE table_name IN ('SERVERS', 'SERV_INST'); TABLE_NAME BLOCKS ------------------------------ ---------- SERV_INST 28 SERVERS 28
- 42. © 2014 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. TAKE THE GUESS-WORK OUT OF ROOT CAUSE ANALYSIS » What you want is a tool that Provides the ability to compare good days with bad: Good times with bad times Provides WAIT analysis Provides TREND analysis Does not hit the Oracle SYSAUX tables requiring the purchase of Diagnostic and Tuning Pack licenses Provides a repository with historical data to use for comparisons
- 43. SUMMARY » Do not guess » If your problem is not caused by slow hardware then faster hardware will just create a problems faster » Gather information from a date-time range when everything was good and compare with the date-time range when the problem was observed » If the problem is slow SQL ... is the SQL statement itself slow or is it the entire environment that should be examined? » If the problem is related to CPU ... do you need more/faster CPUs or a better design? » If the problem is related to I/O ... do you need a new SAN or better indexing? » If the problem is related to network bandwidth ... do you need new NIC cards and to stop network virtualization ... yeah you probably do
- 44. RESOLVE PERFORMANCE ISSUES QUICKLY • Try Database Performance Analyzer FREE for 14 days • Improve root cause of slow performance • Quickly identify root cause of issues that impact end-user response time • See historical trends over days, months, and years • Understand impact of VMware® performance • Agentless architecture, installs in minutes 44 DO W NLO AD FRE E TRI AL © 2015 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. www.solarwinds.com/dpa-download/
- 45. Questions? © 2014 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
- 46. THANK YOU © 2015 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. The SOLARWINDS and SOLARWINDS & Design marks are the exclusive property of SolarWinds Worldwide, LLC, are registered with the U.S. Patent and Trademark Office, and may be registered or pending registration in other countries. All other SolarWinds trademarks, service marks, and logos may be common law marks, registered or pending registration in the United States or in other countries. All other trademarks mentioned herein are used for identification purposes only and may be or are trademarks or registered trademarks of their respective companies. 46