Myth busters - performance tuning 103 2008

December 2008 MythBusters - performance tuning 103 Paul Guerin Bachelor of Engineering (Computer) Oracle DBA Certified Professional DBA 10+ years Employers: Origin Energy, Bluescope Steel, BHP Billiton

Topics Capacity Waste and contention. NETS performance initiatives.

How data is stored Data is stored in a table. A database contains many tables. Tables are comprised of blocks (Oracle: 2k,4k,8k,16k,32k bytes) Ideally place as many rows in each block as practical. In NETS an average row is 75 bytes long, but many are <40 bytes and >150 bytes. Can be 10s or 100s of rows in each block. NETS: ~1000 application tables. Tables are not sorted - inserts are placed where there is space available. Indexes are also comprised of blocks An index is a copy of a table column or concatenated table columns. Indexes are sorted ascending (default) or descending column. NETS: ~800 application indexes. Inserts into an index must preserve the sort order.

Waste from Indexes Everyone loves indexes….. Indexes are often created to improve query performance, but misuse wastes capacity…. Example TX.EVENT_QUEUE : 6453 blocks used (~100MB) 6420 blocks (99.5%) are free most of the time. I/O profile: 1.7million inserts occur around the 20th of each month. CBO statistics will be inaccurate at this time – not an issue.

Waste from Indexes Example: TX.EVENT_QUEUE TX.EVENT_QUEUE_IDX1 (EVENT_HANDLER_TYPE_ID, DAY, CONTRACT_ID, SITE_ID, TNI_CODE, STATE, STATUS, EVENT_QUEUE_ID) = 101MB TX.EVENT_QUEUE_IDX2 (EVENT_HANDLER_TYPE_ID, PRIORITY, CREATED_DATE) = 54MB TX.PK_EVENT_QUEUE (EVENT_QUEUE_ID) = 24MB Total=179MB

Waste from Indexes Example Dropped - TX.EVENT_QUEUE_IDX1 Dropped - TX.EVENT_QUEUE_IDX2 Retained - TX.PK_EVENT_QUEUE (EVENT_QUEUE_ID) = 24MB Created - TX.EVENT_QUEUE_IDX (EVENT_HANDLER_TYPE_ID, DAY) = 31MB Waste reduction = 179 - 55MB = 124MB Waste was greater than the size of the table (ie master data~100MB) !!! Poor index placement leads to: Wasted capacity Inefficient execution plans leading to poor performance.

Table inserts: Each table contains blocks available for insert. These blocks are identified in a freelist. Default: A block is i) available for an insert if <40% full. ii) not available for an insert if >90% full. Also each table has a High Water Mark (HWM) to indicate the maximum number of blocks that contained rows. Index: Row is inserted in the correct place in the leaf block. Table deletes: Table blocks that fall underneath the lower threshold (default: 40%) are placed on the freelist. Index: Row is also deleted.

Waste from nologging inserts All Inserts/deletes/updates generate archive logs (SQLserver: transaction logs) Archive log: a record of changes that occurred in a database. Required for data recovery. Exception: Can perform a nologging insert with an Append hint.

Waste from nologging inserts Nologging inserts (also known as a Direct Path Load) The insert is always performed above the High Water Mark of the table. Advantages: Less overhead leading to potential increase in overall database performance. Insert compressed rows into a table - can't use a normal insert to compress. Disadvantages: Nologging inserts are not cached in memory. Redo entries still created for any indexes on the table. Potentially wastes space as free blocks below the HWM are not reused after a delete.

Methods to delete everything from a table i) DELETE TABLE … Creates redo entries and a possible performance inhibitor for a large table. Also can be CPU capacity intensive. No extent deallocation takes place – so the table and index do NOT shrinking and the tablespace does NOT shrink. HWM remains as is. *** Potential wastage of CPU capacity and storage space due to archive log creation *** ii) DROP TABLE … CREATE TABLE … CREATE INDEX1, CREATE INDEX2, etc … Deallocate extents from table and indexes but re-allocate for inserts. Deallocation/allocation is a performance inhibitor. The tablespace does NOT shrink. iii) TRUNCATE TABLE … Deallocate extents from table and indexes (default: DROP STORAGE) but re-allocate for inserts. Resets the HWM. Tables and indexes shrink – but the tablespace does NOT shrink. ??? DDL becomes severely slower where the number of extents >4096 ??? iv) TRUNCATE TABLE … REUSE STORAGE Arguably the best option as extents are not deallocated – why deallocate if just going to allocate on insert? Resets the HWM. Tables and indexes do NOT shrink – the tablespace also does NOT shrink.

Myth: Tablespaces do not extend. Each tablespace will extend to its natural limit. e.g. NETS = 16GB. To extend beyond the limit another datafile or tempfile must be added. More tempfiles are needed if the workspace in memory and disk is exceeded. Large or inefficient queries need larger workareas - more waste. Myth: Oracle doesn’t reuse space. Oracle does reuse space (on table freelists) but…. Nologging inserts will waste space if misused. Indexes may waste more space than a table. Myth: Truncates, Drops, and Deletes free space. Truncates and drops only deallocate space to the tablespace, not to the storage medium. Deletes only free space within the table (and any indexes). For a delete the table (and any indexes) do not shrink in size - even if every row has been deleted.

Waste from statement processing Myth: Inserts/deletes/updates lock the whole table. Myth: Oracle escalates locks. In Oracle, writes to a table don't block reads to the same table. Oracle doesn't have a lock manager (unlike SQLserver and DB2). Locks are stored with the data. Therefore Oracle can not escalate locks. e.g. several row locks consolidated into a whole table lock. Also locks occur at block level (“hot block”) where a session attempts to read a block that another session is already reading from disk. e.g. multiple PKG_EVENT_QUEUE.DO_DAY_EVENT queued jobs concurrently reference TX.TNI_CUST_AGG_DATA. The wait time for a “hot block” is often >50% greater than the read time from disk. Therefore multi-session processing leads to slower elapsed times for each session. Note: Multi-session processing is NOT the same as parallel processing. Significant amounts of time are wasted where multi-session processing occurs.

Waste from excessively large workareas The database is comprised of storage for data and separate allocations for workareas. NETS: 473GB (52%) is dedicated to copies (MVs, indexes) or temporary work areas (temporary tablespace, undo tablespace). NETS temporary tablespace (~30GB) Sorts or hash joins above >25MB spill from memory to disk. NETS Undo tablespace (~64GB) Need more for long running queries and updates/deletes/inserts. GETS Undo tablespace (~102GB) more than 71% of all GETSP datafiles. Poor performing queries need larger workareas, therefore more storage capacity.

Reclaiming waste NETS: 442GB (48%) is dedicated to table storage. However only 287GB (31%) is actual data. Wastage = 442GB – 287GB = 155GB Large differential due to sparse blocks (ie few rows per block). Solution: Rebuild the tables and indexes to increase block density. Method: Create new tablespace (initial allocation). Rebuild (allocate more): Large tables into their own tablespace. Indexes and MVs into their own tablespaces. Rebuilds of tables require an outage (or at least locking of that table). Rebuilds of indexes can be performed online without locking. Drop old tablespace (deallocation). Note: need more storage space to reclaim storage space.

Reclaiming waste Rebuilds: Increase block density and decrease the number of sparse blocks. Potentially less physical I/O to get the same data. Potentially more rows cached in memory. Enhanced when using compressed tables. Lower cluster factor so indexes are more likely to be used by the CBO. More rows per block can result in better query performance. … . However also a chance of poorer query performance: Execution plan will change but perhaps for the worse….. Similar consequences for other performance initiatives: Parameter changes, statistics gather, new index on table, ... Corrective action : Affected queries need SQL tuning.

Contention for capacity Wasted capacity is a symptom of poor performance but…. contention for capacity can also be significant…. All our databases are on shared storage (SAN): adv: reliability disadv: I/O contention between databases Myth: High workloads in the MTMs don't affect NETS. All production databases share the same storage. I/O processing leads to contention amongst databases. Contention for I/O capacity leads to reduced response times on all production databases. Soln: SQL tuning to reduce I/O. Also datafile rebuilds and compressed objects to reduce the number of blocks required.

Contention related myths “ Can you make my session run faster?” Limited ability to prioritise sessions with Windows. Other operating systems allow easy setting of process priority for the CPU: Solaris/unix/linux: monitoring tools, identify resource utilisation profile, prioritise processes for critical jobs. Only queued jobs (not user sessions) can be prioritised in Oracle 10g. Limited ability to influence CPU contention between user sessions. Myth: All data should be placed on a single database Contention for CPU time Segregate amongst different hosts. Contention for I/O because all hosts on the same SAN: SQL tuning or place data on local storage. Same database inevitably leads higher workloads, contention on CPU, and poorer performance. Placing all data on a single database results in minor savings in storage space. Data consolidation leads to more CPU contention.

Good database practices….. SQL tuning: Less block retrieval and sorting. Less need for large work areas. Instance tuning: Rightsizing memory components. Reducing “hot blocks” and block clones. Database tuning: Correct index placement. Object rebuilds to increase block density. Statistics on objects are also important. Tip: Less contention on a weekend (>8am <6pm) = faster queries

Operational improvements on NETS Sustained effort to improve NETS operating performance over the past two years. Scores of initiatives employed to “clean up the mess”…..

Operational improvements on NETS Query optimisation Problems Rule-based optimiser not supported by Oracle - bugs present that will never be rectified. Generation of poor execution plans. Soln: Gather statistics on all objects Better execution plans lead to faster queries.

Operational improvements on NETS Inserting data Problems Nologging inserts are not compatible with an online backup strategy. Insert only above the HWM - large potential for wastage. Soln: Forced redo entries prevents inserts above the HWM Forced logging is transparent to the application.

Operational improvements on NETS Performance monitoring Problems Difficult to determine which objects are DML intensive. Also difficult to view condition of database days later. Soln: enable monitoring for each table and take performance snapshots Gathering statistics where >10% of changes made to the table. Also monitor index usage.

Operational improvements on NETS Security and auditing Problems Many schemas had an unrestricted ability to change or drop any object. Most schemas had an unrestricted ability to grant any privilege to another schema/user. DBA roles and privileges given to schemas and users. Some schemas had access to DBA type views. Unrestricted access to create public database links. A defacto DBA account had been created. ie DBO. Soln: over 17,000 grants and revokes in 4 stages Improved functionality as direct object privileges are granted instead of system privs via a role. Better ability to determine which objects are not accessed (without needing to audit). e.g. ETGWEB schema in NETS. Enforce the use of correct dynamic views. e.g. user_tab_partitions instead of dba_tab_partitions. Better housekeeping as object location can be restricted. Between 90-70 objects invalid instead of 110-90.

Operational improvements on NETS Backups Problems Taking a database offline destroys any data that was cached in memory. Note: Accessing a block in memory is faster than retrieving from the storage. Offline backups prevent access to data (including db links). Leads to poor availability, if backup every night then links fail every night. As NETS references remote databases, taking a remote database offline every night results in job failures in NETS every night. On failure any transaction needs to rollback. ie double the workload for no net result. Often rerun urgent jobs during business hours which poor performance due to extra workloads. Soln: backup while the database is online Now NETS is open at all times - preserving data in memory. Improved reliability. Parallel file copies reduce the completion time.

Operational improvements on NETS Disaster recovery Problems Intensive administrative burden to test data recovery. No regular disaster recovery. Soln: Use the RMAN utility Easy to test the disaster recovery procedure and datafiles. The backup files become the new database (don't need twice the storage space).

Operational improvements on NETS Full duplicate Problems Intensive administrative burden to provide non-production environments. Need to synchronise and fracture the shared storage. The production database needs to freeze for several minutes. Soln: Use the RMAN utility Less administrative burden. Now no impact on NETS, also tests datafiles used for disaster recovery. Uses a copy of the backup files.

Operational improvements on NETS Partial duplicate Problems Administration intensive partial duplicates Soln: Use the RMAN utility Less administration required. Smaller footprint on shared storage. e.g. TP database only contains the RETAIL schema.

Operational improvements on NETS Month-end snapshot Problems Required all processing on NETS to cease for several hours. Slow process as parallel processing is not used. After-hours monitoring required by the DBA. The hierachy structure of the scripts made improvements complex to implement. Soln: Use the RMAN utility New method has no impact on NETS processing. No impact on Transfer Pricing. Improved reliability. Parallel processing reduces the completion time. Uses a copy of the backup files. Also NETS analysis processing reduced from 2 wks to 4 days.

Operational improvements on NETS Future - continue to proactively improve reliability and performance.... Capacity planning Problems Difficult or impossible to reclaim wastage from low density blocks and nologging inserts. Difficult object management if 100s of objects are in the same tablespace. Large production environments also result in large development/test environments. Soln: Reclaim capacity from the database Relocate (and rebuild) objects. Less space occupied on the shared storage. Smaller backup sizes. Smaller test and development environments so they take less time to refresh. Improved manageability, and potentially performance. Also proved that PSS can be shrunk from 900GB to 200GB.

Operational improvements on NETS Migrate to better O/S Problems Windows has difficulty with copying large files. No longer acceptable for production databases. Larger databases require greater analysis of storage efficiency. Soln: Migrate to a non-Windows environment Investigate Solaris/Unix/Linux for better reliability and monitoring tools.

Operational improvements on NETS Queued jobs Problems Some jobs take days to complete due to inefficient processing. Many old blocks need to be available until long running jobs complete. Require >100GB of Undo tablespace in NETS (+ GETS). Dynamic SQL don't use bind variables leading to memory fragmentation. Soln: SQL tuning activities Reduce CPU utilisation via SQL tuning activities.

Summary Strong correlation between the following: Slow query performance. Wasted data capacity (including unused indexes, inefficient index placement, redundant objects). Low block density (including nologging inserts). Excessive temporary workarea (memory and disk) due to sorting. Excessive undo workareas (memory and disk) due to slow queries. Contention for CPU and I/O capacity amongst sessions. Multiple sessions on the same table creating "hot" blocks and block clones. Waste in production is duplicated in test and development environments. Contention for resources affect all databases.

Summary: NETS is being overhauled and the results so far: Higher availability. Improved overall database performance. Less support group intervention. More initiatives to come… Improvements in NETS are also replicated in GETS, PSS, MTMs…

Myth busters - performance tuning 103 2008

Recommended

More Related Content

What's hot (18)

Similar to Myth busters - performance tuning 103 2008 (20)

More from paulguerin (6)

Recently uploaded (20)

Myth busters - performance tuning 103 2008