Ad
More Related Content
What's hot (20)
Viewers also liked (20)
Ad
Similar to Real Application Testing (20)
Ad
More from oracleonthebrain (7)
Recently uploaded (20)
Real Application Testing
- 1. Oracle Database 11g Real Application Testing and Enterprise Manager Virtual Classroom Lynn Ferrante and Michael Whelan
- 2. Oracle Database 11g Real Application Testing and Enterprise Manager Virtual Classroom Lynn Ferrante and Michael Whelan
- 3. Database Testing Agenda • Overview • Database testing • Eliminate the risk of change by assessing the impact of change on production databases • Database masking • Eliminate risk of moving data to non-production environments 3
- 4. Database Testing • Despite expensive testing success rate low • Many issues go undetected • System availability and performance negatively impacted • Cause of low success rate • Existing tools provide inadequate testing • Simulate synthetic workload instead of replaying actual production workload • Provide partial workflow coverage Database Replay makes real-world testing possible 4
- 5. Typical Steps in Test Phase 1 2 3 4 5 5
- 6. 1: Install & Setup Test System Duration (Days) 24 Days Traditional Approach Real Application 4 Days Testing 0 0 0 0 0 0 10 20 30 40 50 60 70 80 90 0 10 11 12 13 14 15 6
- 7. 2: Understand Application Usage Duration (Days) +20 Days Traditional Approach Real Application +0 Days Testing 0 0 0 0 0 0 10 20 30 40 50 60 70 80 90 0 10 11 12 13 14 15 7
- 8. 3: Identify Key Transactions Duration (Days) +20 Days Traditional Approach Real Application +0 Days Testing 0 0 0 0 0 0 10 20 30 40 50 60 70 80 90 0 10 11 12 13 14 15 8
- 9. 4: Develop Test Scripts Duration (Days) +80 Days Traditional Approach Real Application +2 Days Testing 0 0 0 0 0 0 10 20 30 40 50 60 70 80 90 0 10 11 12 13 14 15 9
- 10. 5: Run Test Workload Duration (Days) +5 Days Traditional Approach Real Application +5 Days Testing 0 0 0 0 0 0 10 20 30 40 50 60 70 80 90 0 10 11 12 13 14 15 10
- 11. Innovate Faster Duration (Days) 149 Days Traditional Approach Real Application 11 Days Testing 0 0 0 0 0 0 10 20 30 40 50 60 70 80 90 0 10 11 12 13 14 15 11
- 12. Comparison of LoadRunner & Oracle Real Application with e-Business Suite 80 80 Time Taken (Days) 60 40 24 20 20 20 5 LoadRunner 4 0 0 2 5 DB Replay 0 Install & Setup Understand Identify Key Generate Run Test Application Transactions Workload Usage Total Testing Time DB Replay: 2 weeks “Traditional” Testing: 30 weeks 12
- 13. Oracle Enterprise Manager Top-Down Application Management Application Application Operations Quality Management Management Real User Experience Monitoring Load Testing and Accelerators for Service Level Packaged Apps & Management Web Services Application Packaged Application Testing Suite Functional Testing and Management Accelerators for Packaged Apps & SOA Application Management Web Services Diagnostic and Tuning Test Management Provisioning, Configuration Management Real Application Testing Change Tracking Data Masking and Synchronization 13
- 15. Real Application Testing • Value • Rapid technology adoption Deploy • Higher testing quality • Business Benefit Test Change • Lower cost • Lower risk Remediate 15
- 16. ROI Ovum Summit Oracle Real Application Testing – business agility through superior testing http://www.oracle.com/corporate/analyst/reports/infrastructure/dbms/ovum -real-application-testing.pdf 16
- 17. Real Application Testing Identify SQL statement with Captures a production performance regressions workload and allows replay of caused by system changes the workload on a test system Execute SQL Execute SQL serially SQL concurrently DATABASE PERFORMANCE REPLAY ANALYZER 17
- 18. Real Application Testing Database Replay 18
- 19. Database Replay • Replay actual production database workload in test environment • Identify, analyze and fix potential instabilities before making changes to production • Capture Workload in Production • Capture full production workload with real load, timing & concurrency characteristics • Move the captured workload to test system • Replay Workload in Test • Make the desired changes in test system • Replay workload with full production characteristics • Honor commit ordering • Analyze & Report • Errors • Data divergence Analysis & Reporting • Performance divergence 19
- 20. Database Replay: Supported Changes Client Client Client … Changes Unsupported Middle Tier Changes Supported •Database Upgrades, Patches •Schema, Parameters Recording of External Client •RAC nodes, Interconnect Requests •OS Platforms, OS Upgrades Storage •CPU, Memory •Storage •Etc. 20
- 21. Common Usage Scenarios • Database initialization/config parameter changes • Schema changes • Changes to operating systems and hardware • Gathering statistics • Database upgrades including patch deployments • Implementation of tuning recommendations • Partitioning • Compression • etc 21
- 22. Database Replay Workflow Production (10.2.0.4) Test (11.1) Clients … Replay Driver Mid-Tier … Storage Storage Analysis & Capture Process Replay Reporting 22
- 23. Step 1: Workload Capture Production System • All external client requests captured in binary files Client Client Client • System background, internal … activity excluded File System • Minimal performance overhead for capture Middle Tier • For RAC, shared and local File 1 file system supported File 2 • Specify interesting time period for capture, e.g., peak … workload, month-end processing, etc. File n Storage • Can capture on 10.2.0.4 and replay on 11g 23
- 24. Step 2: Process Workload Files • Setup test system Test System • Test DB is at same point in time as before production capture • Use RMAN to physically restore production db from backup File 1 File 1 • Use Snapshot standby File 2 • Use imp/exp, Data Pump, etc. File 2 … • Processing transforms captured data into replayable format … File n • Once processed, workload can be File n replayed many times • For RAC copy all capture files to Metadata single location for processing Replay Files Capture Files 25
- 25. Step 3: Replay Workload Test System • Replays workload preserving timing, concurrency and dependencies of the capture system Replay Driver • Replay Driver is a special client program that consumes processed File 1 workload and sends requests to the replay system File 2 … • Replay Driver consists of one or more clients. For File n workloads with high concurrency, it may be Metadata necessary to start multiple Replay Files clients to drive workload 26
- 26. Analysis & Reporting • Comprehensive reports are provided for analysis purposes • There (3) types of divergences are reported • Data Divergence: Number of rows returned by each call are compared and divergences reported • Error Divergence: For each call error divergence is reported • New: Error encountered during replay not seen during capture • Not Found: Error encountered during capture not seen during replay • Mutated: Different error produced in replay than during capture • Performance Divergence • Capture and Replay Report: Provides high-level performance information • ADDM Report: Provides in-depth performance analysis • AWR, ASH Report: Facilitates comparative or skew analysis 27
- 27. Demo 28
- 28. Real Application Testing SQL Performance Analyzer 29
- 29. SPA Benefits • Enables identification of SQL performance regressions before end-users can be impacted • SPA can help with any change that impacts SQL execution plan • DB upgrades • Optimizer statistics refresh • New indexes, Materialized Views, Partitions, etc. • Automates SQL performance tracking of hundreds of thousands of SQL statements – impossible to do manually • Captures SQL workload with low overhead • Integrated with SQL Tuning Advisor and SQL Plan Baselines for regression remediation 30
- 30. Production Test Client Client Client … Middle Tier Re-execute SQL Queries Capture SQL … … Use SQL Tuning Oracle DB Advisor to tune regression Storage 31
- 31. SQL Performance Analyzer: Workflow Production Test Make Change Steps (1) (2) (3) (4) (5) Capture Transport Execute SQL Execute SQL Compare SQL (STS) STS Pre-change Post-change Perf. (6) Reiterate No Done? (7) Yes Tuned System Production Change / (7) Tuning Deployment 32
- 32. Step 1: Capture SQL Workload • SQL Tuning Set (STS) used to store SQL workload • STS includes: • SQL Text • Bind variables • Execution plans Cursor Cache • Execution statistics Incremental Capture • Incremental capture used to populate STS from cursor cache over a time period • SQL tuning set’s filtering and ranking SQL Tuning Set capabilities filters out undesirable SQL • SQL workload captured in 10.2.0.1 and Production Database higher can be used for SPA tasks in 11g 33
- 33. Step 2: Move SQL Workload to Test System Cursor Cache SQL Tuning Set Export/Import SQL Tuning Set Production Database Test Database • Use Enterprise Manager 34
- 34. Step 3: Execute SQL Before Making Change SQL Tuning Set • Establishes SQL workload performance baseline Fetch Next SQL • SQL execution plan and statistics captured Test Execute • SQL executed serially (no concurrency) Execution Plan & • Each SQL executed only once Statistics Save Result • DDL/DML skipped • Option to do Explain Plan only analysis SQL Performance Analyzer 35
- 35. Step 4: Execute SQL After Making Change SQL Tuning Set Fetch Next SQL • Manually implement the planned change • Database upgrade, patches • Optimizer statistics refresh • Schema changes Test Execute • Database parameter changes • Tuning actions, e.g., SQL Profile creation Execution Plan & Statistics Save Result • Re-execute SQL after change Completed • Gathers new SQL execution plans and statistics SQL Performance Analyzer 36
- 36. Step 5: Compare & Analyze Performance Completed Completed • Compare performance using different metrics, e.g., • Elapsed Time • CPU Time • Optimizer Cost • Buffer Gets Compare SQL Performance • SPA Report shows impact of change for each SQL • Improved SQL Analysis Report • Regressed SQL • Unchanged SQL • Fix regressed SQL using SQL Tuning Advisor or SQL Plan Baselines SQL Performance Analyzer 37
- 37. SPA Report 38
- 38. SPA Report 39
- 39. Demo 40
- 40. Real Application Testing Summary • Provides cost-effective and easy-to-use solution of assessing impact of change on production systems • Holistic and actual workload testing results in lower risk • Brings down testing cycle from months down to days • Lowers hardware costs by eliminating need for mid-tier and application setup on test system • Maximizes ROI by leveraging Diagnostics and Tuning Pack to remediate regressions • With Real Application Testing businesses can • Stay competitive • Improve profitability • Be compliant 41
- 41. Real Application Testing: Tools of the Trade SQL Performance Analyzer Database Replay What is it? • Predicts SQL performance deviations • Replays real database workload before end-users can be impacted, on test system, helps assess helps assess impact of change on impact of change on workload SQL response time throughput How it works? • Executes each SQL, stored in SQL • Captures workloads and replays it Tuning Set, in isolation using with production characteristics production context and then including concurrency, compares before and after execution synchronization & plans and run-time statistics dependencies When to use? • Unit testing of SQL with the goal to • Comprehensive testing of all identify the set of SQL statements sub-systems of the database with improved/regressed performance server using real production workload SQL SQL Concurrency Dependency Speed up/down 42
- 43. Database Diagnostic Pack Introduction • Part of the database management pack family • Provides unique functionalities • Industry’s first performance self-diagnosing engine • Lightweight, Automatic, Out-of-box performance data warehouse • Guided problem resolution • Introduces a new, easy to use performance tuning methodology • Not just another tool • Core functionality built right into the database server • Makes it most efficient, integrated and accurate 44
- 44. Database Diagnostic Pack Key Features • Automatic Workload Repository (AWR) • Active Session History (ASH) • Automatic Database Diagnostic Monitor (ADDM) 45
- 45. How Does ADDM Work? • Top Down Analysis Using Snapshots in AWR Snapshots Automatic Workload Repository • Throughput centric - Focus on reducing time ‘DB time’ Automatic Diagnostic Engine Self-Diagnostic Engine • Classification Tree - based on decades of Oracle performance tuning expertise • Real-time results • Don’t need to wait hours to see the results • Pinpoints root cause High-load IO / CPU RAC issues • Distinguishes symptoms SQL issues from the root cause • Reports non-problem areas System Network + SQL Advisor Resource DB config • E.g. I/O is not a problem Advice Advice 46
- 46. Database Tuning Pack Introduction • Part of the database management pack family • Provides unique, automatic and deterministic SQL tuning functionality • Core functionality built in the Oracle Database 10g kernel • SQL Tuning Advisor • SQL Access Advisor • Requires diagnostic pack 47
- 47. SQL Profiles Contains auxiliary information specific to a given SQL statement – Customized optimizer settings Based on past execution history – Compensation for missing or stale statistics – Compensation for errors in optimizer estimates Estimation errors occur due to data skews and correlations, complex filters and joins Doesn’t freeze the execution plan Doesn’t require any change to the SQL text – Ideal for Packaged Apps Persistence: Works across shutdowns & upgrades Transportable across databases (10.2) 48
- 49. Oracle Data Masking Data Masking for Non-Production Environments • Eliminate risk of moving data to non-production environments LAST_NAME SSN SALARY AGUILAR 203-33-3234 40,000 • Share data with partners BENSON 323-22-2943 60,000 without exposing sensitive data Production • No impact on production environments • Sensitive data never leaves the database (masked in place) Non-Production • Automated masking process LAST_NAME SSN SALARY ANSKEKSL 111—23-1111 40,000 preserves application integrity BKJHHEIEDK 111-34-1345 60,000 50
- 50. Summary • Real Application Testing for managing risk • Provides cost-effective and easy-to-use solution of assessing impact of change on production systems • Lowers hardware costs by eliminating need for mid-tier and application setup on test system • Maximizes ROI by leveraging Diagnostics and Tuning Pack to remediate regressions • Diagnostic and Tuning Pack for proactive management • Industry’s first performance self-diagnosing engine • Data Masking for security of data 51
- 51. Comparison of Traditional Approach & Oracle Real Application with e-Business Suite 80 80 Time Taken (Days) 60 40 24 20 20 20 5 LoadRunner 4 0 0 2 5 DB Replay 0 Install & Setup Understand Identify Key Generate Run Test Application Transactions Workload Usage Total Testing Time DB Replay: 2 weeks “Traditional” Testing: 30 weeks 52