Oracle 19c Upgrade

Tips and Tricks

www.viscosityna.com @ViscosityNA
 Sean Scott
25 years working with Oracle technology
UTOUG Board ⁘ RAC SIG Board
Oracle OpenWorld ⁘ Collaborate/IOUG ⁘ Regional UG
RAC/MAA ⁘ DR/HA ⁘ TFA/AHF ⁘ Exadata/ODA
Automation ⁘ DevOps ⁘ Containers ⁘ Virtualization
Ultramarathon Running ⁘ Bouldering

www.viscosityna.com @ViscosityNA
 Why Upgrade?

www.viscosityna.com @ViscosityNA
2009: Oracle 11gR2 Released. Also…

“In May 2009, Facebook

overtook MySpace in
U.S.-based web traffic
for the first time."
www.viscosityna.com @ViscosityNA
2013: Oracle 12cR1 Released. Also…

“Twerk” and “Selfie”

added to the dictionary

Best Comedy of 2013*

* Live action comedy released in the three-month window between Melissa McCarthy films “Identity Thief” and “The Heat”

www.viscosityna.com @ViscosityNA
Oracle 19c Support Roadmap
• Oracle strongly recommends customers upgrade to Oracle Database 19c
• Terminal release for Oracle Database 12c Release 2
• No Extended Support for 12.2.0.1 or 18c
• Extended and Long-Term Support will only be available for 19c
• Premier Support through April 2024 *
• Extended Support through April 2027 *

“Oracle 19c is the 11.2.0.4 of the 12c family”

* Per MOS Note 742060.1, updated August 3 2020

www.viscosityna.com @ViscosityNA
 Upgrade vs Migration

www.viscosityna.com @ViscosityNA
Upgrade vs. Migrate
Upgrade Migrate
• Existing hardware • Change hardware, OS, storage
• Side-by-side Oracle homes • Source, target Oracle homes
• Database metadata is updated • Source remains unchanged
• Data unchanged in-place* • Data moves* and may change*
• Single copy of data • Multiple copies of data*

www.viscosityna.com @ViscosityNA
Modern upgrades usually include migrations
Migration methods determined by:
• Tolerance for downtime
• Source and target hardware, OS, architecture
• Database size
• Licensing
• Integrations and dependencies

www.viscosityna.com @ViscosityNA
Select a trusted, experienced partner to shepherd the process

www.viscosityna.com @ViscosityNA
 Upgrade Methods

www.viscosityna.com @ViscosityNA
Upgrade Tools
• catupgrade, command line scripts
• PUU: Parallel Upgrade Utility
• Data Pump
• CTAS: Create Table as SQL
• ZDG: Zero Downtime Upgrade with GoldenGate
• FPP: Fleet Patching & Provisioning—formerly Rapid Home Provisioning
• DBUA: Database Upgrade Assistant—GUI, Silent
• AutoUpgrade

www.viscosityna.com @ViscosityNA
AutoUpgrade vs. DBUA
DBUA AutoUpgrade
• Parameters on command line • Fine-grained configuration
• Difficult to resume • Easy to resume
• Good error correction • Excellent error correction
• Good pre-checks, fixes • Better pre-checks, fixes
• GRP optional • GRP by default
• Pre-, post-tasks optional • Pre-, post-tasks by default
• Works w/NOARCHIVELOG • Superior feedback
• Better diagnostic files

www.viscosityna.com @ViscosityNA
 When possible, use AutoUpgrade

www.viscosityna.com @ViscosityNA
My Dad Taught Me…
Don’t buy cheap tools

If you don’t have time to do it right…

….you don’t have time to do it over

Use the right tool for the job

Measure twice, cut once

www.viscosityna.com @ViscosityNA
 Before the Upgrade

www.viscosityna.com @ViscosityNA
Before the Upgrade - Check and Review

• Contents and size of FRA

• Redo rates
• Restore Points
• NOLOGGING objects and tablespaces
• Datafiles, Tempfiles, Redo Log files

www.viscosityna.com @ViscosityNA
Before the Upgrade - Check and Review

• Database links
• Directories
• External tables
• Jobs and Scheduler Jobs
• Review and document hidden parameters

www.viscosityna.com @ViscosityNA
Before the Upgrade - Check and Review

• Environment variables
– SQLPATH
– TWO_TASK
• .profile, .bash_profile, .bashrc, etc.
• Shell aliases

www.viscosityna.com @ViscosityNA
Before the Upgrade - Changes

• Set Force Logging

• Set Block Change Tracking
• Extend ADR retention policy
• Extend AWR retention policy
• Resize FRA
• Remove hidden parameters

www.viscosityna.com @ViscosityNA
Before the Upgrade - Changes

• Unset SQLPATH
• Remove login.sql
• Restore default glogin.sql
• Remove aliases to ORACLE_HOME/bin

www.viscosityna.com @ViscosityNA
Before the Upgrade - Backup or Capture

• Capture IPCS reports (semaphore, shared memory)

• Backup SAR directory
• Backup crontab
• Backup spfile and memory to pfile
• Backup /etc/oratab

www.viscosityna.com @ViscosityNA
Before the Upgrade - Backup or Capture

• $ORACLE_HOME/dbs
• $ORACLE_HOME/network/admin
• Wallets
• Diagnostic directories
• Capture SGA and PGA target advice
• Report Opatch inventory, patch registry

www.viscosityna.com @ViscosityNA
Before the Upgrade - Listener

• Report status and services

• Review listener logs
• Review and backup configurations

www.viscosityna.com @ViscosityNA
Before the Upgrade - RMAN

show all
report unrecoverable
report need backup
report schema
list backup

www.viscosityna.com @ViscosityNA
Before the Upgrade - APEX

Upgrade or remove APEX

• APEX is not upgraded with the database (Oracle 18c +)
• Must be upgraded or removed prior to database upgrade
• Drop old APEX schema prior to TZ upgrade

www.viscosityna.com @ViscosityNA
Before the Upgrade - Review Features

DBMS_JOBS
• Replaced by DBMS_SCHEDULER
• Supported for backward compatibility only
• Upgrade will create DBMS_SCHEDULER entries
• Pre-create DBMS_SCHEDULER entries if possible
• Control, review
• Reduces work performed in upgrade

www.viscosityna.com @ViscosityNA
Before the Upgrade - Review Features

Oracle Media
• Removed
Oracle Streams
• Replace with:
• Oracle GoldenGate
• Quest Shareplex
• HVR

www.viscosityna.com @ViscosityNA
Before the Upgrade - AutoUpgrade/DBUA

• Run analyze/pre-upgrade JAR

• Confirm the following:
• DB_RECOVERY_FILE_DEST_SIZE
• DB_RECOVERY_FILE_DEST

www.viscosityna.com @ViscosityNA
Before Upgrading - Last Steps

• Run ultrp.sql
• Collect dictionary, fixed object status
• Empty recycle bin
• Run the pre-upgrade JAR or AutoUpgrade -analyze

www.viscosityna.com @ViscosityNA
 Before the Upgrade - Advanced

www.viscosityna.com @ViscosityNA
Reduce Technical Debt

• Remove demo schemas

• Reduces database size, attack surface
• Plan for service account credential/password changes
• Upgrade wallets & certificates
• Upgrade listener security
• Review SQL*Net client/host allowed versions

www.viscosityna.com @ViscosityNA
Review Access Control

Access Control Entries (ACE) replace Access Control Lists (ACL)

• ACE are more granular
• ACL are mostly compatible with ACE
• ACE does not recognize port-specific privileges for resolve
• Root, intermediate certificates sometimes handled differently
• Check for deprecated encryption implementations

www.viscosityna.com @ViscosityNA
Review Database History

dba_feature_usage_statistics
• Identify features that were installed, removed
• Confirm features were removed completely, correctly
• Look for artifacts

dba_registry_sqlpatch, dba_registry_history
• Review upgrade and patch apply history

www.viscosityna.com @ViscosityNA
Download & Stage Files

• Latest version of Opatch - Patch 6880880

• Latest version of AutoUpgrade - MOS 2485457.1
• Latest version of Pre-Upgrade JAR - MOS 884522.1
• Latest version of AHF/TFA (includes OraChk) - MOS 2550798.1

www.viscosityna.com @ViscosityNA
Get Familiar with AHF/TFA

Did you know?

• OraChk/ExaChk includes a pre-install check:
orachk -u -o pre
• AHF includes SRDC for install issues:
$TFA_HOME/bin/tfactl diagcollect -srdc dbinstall

www.viscosityna.com @ViscosityNA
 Recognize Cognitive Bias

www.viscosityna.com @ViscosityNA
 A cognitive bias is a systematic pattern of deviation from
norm or rationality in judgment. Individuals create their
own "subjective reality" from their perception of the input.
An individual's construction of reality, not the objective
input, may dictate their behavior in the world.

www.viscosityna.com @ViscosityNA
 Bias: We have documentation

Is it current?
Is it accurate?

Documentation doesn’t address:

Stress Coordination Research
Urgency Phone calls Management
Confusion Messaging Doubt, Panic
Multitasking Alerts Conflict
www.viscosityna.com @ViscosityNA
 Bias: We have backups
Restoring a database is like cooking.
Backups are merely ingredients.

Success requires all the right ingredients,

a recipe, a properly equipped kitchen,
and skilled chefs to bring it all together.

www.viscosityna.com @ViscosityNA
 Bias: If it works in pre-prod, it will work in prod

Production environments may include

exotic complexities or configurations that
aren’t well duplicated in lower environments.

www.viscosityna.com @ViscosityNA
 Bias: If it works in pre-prod, it will work in prod

Production upgrades rarely go without incident.

The best way to prepare for the unexpected

is to make the fundamentals second-nature.

www.viscosityna.com @ViscosityNA
 Bias: Production is under change control

Older systems are more likely to contain

undocumented or unexpected idiosyncrasies
that introduce brittleness and fragility.

www.viscosityna.com @ViscosityNA
 Upgrades = World Cup

www.viscosityna.com @ViscosityNA
Upgrades are the World Cup for DBAs

• High stakes

• Infrequent

• Scrutiny

• High pressure!

www.viscosityna.com @ViscosityNA
 Everyone is an expert

www.viscosityna.com @ViscosityNA
High expectations eclipse
Expectations
exceptional performances

www.viscosityna.com @ViscosityNA
Practice, Practice, Practice

Prepare like elite athletes and teams

• Build confidence in yourself, others
• Develop “muscle memory” for your upgrade
• Respond instinctively
• Improve reaction time
• Know your tools and systems

www.viscosityna.com @ViscosityNA
 Expectations Not all losses
are equal

www.viscosityna.com @ViscosityNA
Learn to Fix Things by Breaking Things

Something will go wrong

• Learn to fix things before reaching production
• Use Docker, virtualization to create sandbox systems
• Break, fix, burn, repeat
• Use for Gamification
• Create challenges
• Build teamwork and skills

www.viscosityna.com @ViscosityNA
 After the Upgrade

www.viscosityna.com @ViscosityNA
Post-upgrade tasks

• Report and compare datafiles, tempfiles, redo log files

• Update .profile, .bash_profile, .bashrc, etc.
• Update crontab

www.viscosityna.com @ViscosityNA
 Avoiding the Most
Common Issues

www.viscosityna.com @ViscosityNA
Automate!

Most common upgrade issue - user error

• Steps:
• Missed
• Performed 2x+
• Run out of order
• Run concurrently
• Wrong environment values (copy/paste commands)

www.viscosityna.com @ViscosityNA
Preview Optimizer Features & Behavior

Most common post-upgrade issue - performance!

• Know the new optimizer features
• Which features will go live?
• How are they disabled?
• Understand new optimizer behavior
• Default settings may change across point releases

www.viscosityna.com @ViscosityNA
 Questions

www.viscosityna.com @ViscosityNA
 e oraclesean.com

C https://www.linkedin.com/in/soscott/

D @oraclesean

) https://github.com/oraclesean

k [email protected]

P Search "OracleSean" on YouTube

www.viscosityna.com @ViscosityNA