Oracle GoldenGate 21c New Features and Best Practices

Copyright, 2021 RheoData and affiliates
www.rheodata.com @rheodatallc
Oracle GoldenGate 21c
New Features and Best Practices
8/18/2021
Presentation will start 3 minutes after

About RheoData
A global systems integrator headquartered in Atlanta, Georgia. Providing solutions across the enterprise with
architecture design/assessments, implementation, data replication, and professional services to ensure high-
quality enterprise systems leveraging Oracle and Microsoft technologies. RheoData works hand-in-hand with
our strategic partners and their sales teams to build and deploy high performance and scalable enterprise
solutions in addition to providing clients with cost savings professional services for turnkey enterprise
management.

Our Mission:
Enabling businesses to transform through
automation at a time when companies must
change in order to scale

License Management for Oracle,
HashiCorp, and Cloud
Get the most out of your license
DBA
Services
Remote and
On-Site
Professional Services
Where and when you need it the most
Migrations
Zero Downtime
Health Checks
How are you running?
On-Call Support
Managed Services
Tactical Assistance Program (TAP)
Experts to help design and manage complex architecture
Take Cloud to next level
Automation of Cloud

Speaker
Bobby Curtis, MBA
Atlanta, GA
• Author
• Speaker
• Former Product Manger for GoldenGate
• Oracle ACE Director (Alumni)
@dbasolved
https://dbasolved.com
bobby.curtis@rheodata.com

Agenda
� GoldenGate for Oracle
 GoldenGate (Microservices) Enhancements
 Microservices (Best Practices)
 Deprecated and Desupported Features

OGG 21c New Features (Oracle)
� Automatic Table Extract – DDL to enable GoldenGate to start/stop capture on a table (DB 21c feature)
� Unified Builds – Bundles all database clients and supports Oracle databases per cert matrix
• ADW Capture Support – ATP (dedicated), ADW (dedicated) and ATP/ADW on ExaCC/ExaCS
• Kerberos Authentication – External authentication for database login (DBLOGIN) for Kerberos Accounts
• Simplified Heartbeat Administration – External
• Native JSON Support – Capture (extract) and Apply (replicat) now support the native JSON datatype (DB 21c feature)
• AutoCDR Enhancements – Earliest Timestamp, Delete always wins, & Site priority resolution (DB 21c feature)
• Partition Name Filtering – Filtering by partition name (PARTITION/PARTITIONEXCLUDE)
• DB_UNIQUE_NAME – Available in trail files and heartbeat tables
• DDL > 4MB – Large DDL text is supported (up to 4MB)
• Enhanced PR and IR reports – Enhanced diagnosis capabilities for Parallel and Integrated Replicat
• Certificate Management – Create and Manage server and client certificates from web interface
• Multiple Certs for WSS Protocol – Store and use multiple client certificates for authentication
� Microservices Only – Starting in 21c, Oracle GoldenGate is available with Microservices Only (Classic and Microservices released)

Automatic Table Extract
DDL to enable OGG to start/stop capturing a table
• Requires Oracle Database 21c or later
• Disabled by default
OGG Parameter:
TRANLOGOPTIONS INTEGRATEDPARAMS (enable_auto_capture Y)
Use INFO EXTRACT to see setting for each Extract
Database:
• Enabling Supplemental Logging on a table will trigger auto-capture
• Setting ENABLE LOGICAL REPLICATION on a table with supplemental logging, triggering auto-capture
• Can be done as a CREATE or ALTER TABLE ddl command

Unified Builds
Microservices Architecture:
Database Instant Client Libraries - $OGG_HOME/bin/instantclient
Environment Variables:
ORACLE_HOME=$OGG_HOME/lib/instantclient
LD_LIBRARY_PATH=$OGG_HOME/lib:$OGG_HOME/lib/instantclient

Agenda
� GoldenGate (Microservices) Enhancements
 Microservices (Best Practices)

OGG 21c Microservices Enhancements
� Unified Web UI for Deployments – No more different tabs within browser
� Simplified Certificate Management – no more direct manipulation of Oracle Wallet to add/replace certificates
• Client Certificate Selection – Support for choosing certificates for Distribution Paths
• Migration Utility – Classic to Microservices migration utility (pending)
• Security Improvements – Unix Domain Sockets for inter-process communication with Performance Metric Service
� Microservices for Heterogenous – Starting in 21c, bring more flexibility to data architectures

Microservices for Heterogenous
• More flexibility for heterogenous environments
• Cleaner interoperability between environments
• Work remotely across the enterprise

Unified Web UI
• All Services of deployment in a single browser page

Simplified Certificate Management
• Web UI and REST APIs:
• View/Replace Certificates
• Add multiple client certificates
• Add trust points (CA certs)
• Not Supported – Direct manipulation of wallets after installation
• Distribution Paths can select which client certificate to present to
remote site

Agenda
� Microservices (Best Practices)

Oracle GoldenGate Microservices
Source
Target
Uni-Directional or Active/Active
Extract Replicat
Trail
Files
Trail
Files
Active/Active
AdminSrvr DistroSrvr RecSrvr
ServiceMgr PMSrvr
Nginx (Reverse Proxy)
CMD
REST
HTML
PL/SQL

To Hub or Not To Hub
What is a HUB architecture?
What is Data Mesh?

ServiceManager
Deployment Home
TF
Nginx
Extract Replicat
Hub Architecture
Unified Libraries
Streaming and DataOps
ETL
&ML
DBMS
Cloud
Big Data
NoSQL
Streams
Object
Storage

Best Practice 1: Deployments
Microservices brought a huge change to GoldenGate:
• Microservices
• Oracle GoldenGate Home
• Deployment Home Structure
• New Environment Variables

Oracle GoldenGate Home
/opt/app/oracle/product/19.1.0/oggcore_19c
[dr-xr-xr-x oracle oinstall] OPatch
[dr-xr-xr-x oracle oinstall] bin
[dr-xr-xr-x oracle oinstall] cfgtoollogs
[dr-xr-xr-x oracle oinstall] deinstall
[dr-xr-xr-x oracle oinstall] diagnostics
[dr-xr-xr-x oracle oinstall] include
[dr-xr-xr-x oracle oinstall] install
[dr-xr-x--- oracle oinstall] inventory
[dr-xr-xr-x oracle oinstall] jdk
[dr-xr-xr-x oracle oinstall] jlib
[dr-xr-xr-x oracle oinstall] lib
[dr-xr-xr-x oracle oinstall] oui
[dr-xr-xr-x oracle oinstall] srvm
• Directory structure now like Oracle Home
($ORACLE_HOME)
• After installation $OGG_HOME should be changed to
Read-Only
• Prevents unauthorized changes

Oracle GoldenGate Deployment Home
/u01/gg_deployments
|-- [drwxr-x--- oracle oinstall] Atlanta
| |-- [drwxr-x--- oracle oinstall] bdb
| |-- [drwxr-x--- oracle oinstall] etc
| `-- [drwxr-x--- oracle oinstall] var
|-- [drwxr-x--- oracle oinstall] NewYork
| |-- [drwxr-x--- oracle oinstall] bdb
| |-- [drwxr-x--- oracle oinstall] etc
| `-- [drwxr-x--- oracle oinstall] var
`-- [drwxr-x--- oracle oinstall] ServiceManager
|-- [drwxr-x--- oracle oinstall] bin
|-- [drwxr-x--- oracle oinstall] etc
`-- [drwxr-x--- oracle oinstall] var
• Directory structure resides outside of $OGG_HOME
• All deployments should reside in a standard location
• File System (NFS, SAN, DBFS, ACFS, etc.)
• Docker Volumes (supported)
• Directory should be read-write for Oracle user
(assumption)
• Only appears in ServiceManager directory
• registerServiceManager.sh
• only if set to daemon
• startSM.sh (only if manual/XAG)
• stopSM.sh (only if manual/XAG)

Deployment Home
ServiceManager
Deployment Home
TF
Nginx
Extract Replicat
Design Tips
• Locations
• Regional Approaches
• Sub-Setting
• Application
• Type of Application
Unified Libraries

Best Practice 2: Upgrades
Microservices enable easier upgrades:
• Microservices
• Utilities
• Leap Frogging
• Upgrades from Microservices

Utilities
• August 2017 – Microservices release, no upgrade utility
• Planned upgrade utility
• Started in 2019 (late)/2021 (pending)
• Limited use-cases
• Oracle Supported via My Oracle Support (MOS) Note (pending)

Leap Frogging
Source
Extract Pump
Trail
Files
Manager
Target
Replicat
Trail
Files
Admin
Srvr
Distro
Srvr
Rec
Srvr
Service
Mgr
PM
Srvr
Nginx (Reverse Proxy)
1. Classic Extract captures from Source and builds trail file
2. Pump ship trails to Microservices Receiver Service
3. Microservices Replicat reads trail and applies to Target
4. Convert Target to Source, New Target built with Microservices
Classic to Microservices

$OGG_HOME
12.3/18.1/19.1
(MA)
etc
$DEPLOYMENT_HOME
(ServiceManager)
var
bin
etc
$DEPLOYMENT_HOME
var
etc
$DEPLOYMENT_HOME
var
Microservices Upgrade
curl -L -X PATCH
'https://localhost/services/v2/deployments/ServiceManager'
-H 'Authorization: Basic b2dnYWRtaW46V2VsY29tZTE='
-H 'Content-Type: text/plain'
--data-raw '{
"oggHome":"/opt/app/oracle/product/21.3.0/oggcore_1",
"status":"restart"
}'

Copyright © 2019 Oracle and/or its affiliates.
etc
$DEPLOYMENT_HOME
(ServiceManager)
var
bin
etc
$DEPLOYMENT_HOME
var
etc
$DEPLOYMENT_HOME
var
$OGG_HOME
21.3.0.x (MA)
curl -L -X PATCH 'https://localhost/services/v2/deployments/Atlanta'
-H 'Authorization: Basic b2dnYWRtaW46V2VsY29tZTE='
--data-raw '{
"oggHome":"/opt/app/oracle/product/21.3.0/oggcore_1",
"oggEtcHome":"/opt/gg_deployments/Atlanta/etc",
"oggVarHome":"/opt/gg_deployments/Atlanta/var",
"environment"{
"tns_admin":"/opt/app/oracle/product/18.1.0/network/admin"
}
"status":"restart"
}'
$OGG_HOME
12.3/18.1/19.1
(MA)

Copyright © 2019 Oracle and/or its affiliates.
etc
$DEPLOYMENT_HOME
(ServiceManager)
var
bin
etc
$DEPLOYMENT_HOME
var
etc
$DEPLOYMENT_HOME
var
$OGG_HOME
21.3.0.x (MA)
$OGG_HOME
12.3/18.1/19.1
(MA)

Best Practice 3: Security
Security is a theme in Microservices:
• Microservices
• ServiceManager
• Nginx
• User Security – Management Layer
• User Security – Configuration Layer

Unsecure or Secure - ServiceManager
ServiceManager
Deployment Home
TF
Nginx
Extract Replicat
11g 12c 19c
18c
Secure
• SSL/TLS security enables and affects all
underlying deployment homes
• Must have Oracle Wallet with Certificates
for host and distribution path
Unsecure
• SSL/TLS security disabled and affects all
underlying deployment homes
Cannot mix security, all or nothing approach

Unsecure or Secure - Nginx
ServiceManager
Deployment Home
TF
Nginx
Extract Replicat
11g 12c 19c
18c
Secure
• Consolidates ports required for services
down to a single port
• Keeps Firewall Admins friendly
Unsecure
• Required to open a minimum of 6 ports for
first deployment, plus 5 additional ports
for each deployment added.

Unsecure or Secure - Nginx
ServiceManager
Deployment Home
TF
Nginx
Extract Replicat
11g 12c 19c
18c
#!/bin/bash
# Copyright (c) 2020 RheoData, LLC and/or its affiliates. All rights reserved.
#
# Since: March 2019
# Author: Bobby Curtis <bobby.curtis@rheodata.com>
# Description: configureNginx
#
# DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS HEADER.
#
vServiceManagerUser=$1
vPassword=$2
vPort=$3
export JAVA_HOME=${OGG_HOME}/jdk
echo ${JAVA_HOME}
echo ${OGG_HOME}
cd ${OGG_HOME}/lib/utl/reverseproxy
./ReverseProxySettings -u $vServiceManagerUser -P $vPassword -o ogg.conf
https://localhost:$vPort
sudo cp ogg.conf /etc/nginx/conf.d/nginx.conf
sudo sh /etc/ssl/certs/make-dummy-cert /etc/nginx/ogg.pem
sudo nginx
sudo nginx -t
sudo nginx -s reload
• Nginx is not installed by default with OGGMA.
• Requires a new yum repo to be configured.
• Nginx must be refreshed after each deployment
added
• Port 80 or 443 will be based on ServiceManager

Users Security – Management Layer
ServiceManager
Deployment 1 … N
Security
Administrator
Operator
User
Security
Administrator
Operator
User
Access (ServiceManager)
Access (Deployment)
ServiceManager roles have
access to all underlying
deployments at the permissions
granted by the role.
Assign to DBAs and Mgmt
Roles assigned within a
deployment have the underlying
permissions granted to only the
deployment which granted.
Assign to Jr. DBAs and DevOps

User Security – Configuration Layer
Oracle GoldenGate Classic
User Alias Purpose
C##GGATE CDB User
GGATE PDB User
SGGATE Database UserIDAlias
Oracle GoldenGate Microservices
User Alias Purpose
C##GGATE CDB User
GGATE PDB User
SGGATE Database
UserIDAlias
PDBGGATE PDB UserIDAlias
OGGADMIN ServiceManager
STREAMADMIN Distribution Service
curl -L -X POST
'https://localhost/Atlanta/adminsrvr/services/v2/credentials/
OracleGoldenGate/SGGATE'
-H 'Authorization: Basic b2dnYWRtaW46V0VsY29tZTEyMzQ1IyM='
--data-raw '{
"userid":"c##ggate@10.20.1.2:1539/orcl.rheodata.com",
"password":"WElcome12345##"
}'
add credentialstore add user
c##ggate@10.20.1.2:1539/orcl.rheodata.com",
password WElcome12345##
alias SGGATE
domain OracleGoldenGate

Best Practice 4: APIs
APIs are the optimal way of working with Oracle GoldenGate Microservices:
• Microservices
• Write once, Run Multiple
• Script it out

APIs – General Information
• Oracle GoldenGate 12c, 18c, 19c & 21c for Microservices
• All administration functions can be performed with RESTful APIs
• Core Documentation
• https://docs.oracle.com/en/middleware/goldengate/core/21.3/index.html
• RESTful API Documentation (21.1.0.0.x)
• https://docs.oracle.com/en/middleware/goldengate/core/21.1/oggra/

API
curl -X POST
http://{hostname}:{port}/services/v2/connections/{Domain}.{Alias}/trandata/table
--user ‘{ServiceManagerUser}:{password}'
-H 'cache-control: no-cache'
-d '{
"operation":"add",
"tableName":"pdb1.inittest.optype_test"
}'
Database - Trandata

API
curl -X POST
http://{hostname}:{port}/services/v2/connections/{Domain}.{Alias}/trandata/schema
-- user ‘{ServiceManagerUser}:{password}'
-d '{
"operation":"add",
"schemaName":"oggtst1.soe"
}'
Database - SchemaTrandata

API
curl -X POST
http://{hostname}:{port}/services/v2/extracts/{Ex
tractName}
-d '{
"description":"Integrated Extract",
"config":[
"Extract IEPKTST",
"ExtTrail bc",
"UseridAlias SGGATE",
"sourcecatalog pdb1;",
"Table inittest.pktst_table, getbeforecols(on
update all, on delete all);"
],
"source":{
"tranlogs":"integrated"
},
"credentials":{
"alias":"SGGATE"
},
"registration":{
"containers": [ "pdb1" ],
"optimized":false
},
"begin":"now",
"targets":[
{
"name":"bc",
"sizeMB":250
}
],
"status":"start"
}'
• Configure, Build, and Run from a
single command
• Source Controlled Extract
Extracts

API
• Configure, Build, and Run from a
single command
• Source Controlled Replicat
curl -X POST
http://{hostname}:{port}/services/v2/replicats/{Repli
catName}
-d '{
"config":[
"Replicat IRPKTST",
"UseridAlias TGGATE",
"Map pdb1.inittest.pktst_table, Target
inittest.pktst_table, colmap(usedefaults,
optype=@getenv('''ggheader''','''optype'''),
beforerid=@before(rid));"
],
"source":{
"name":"bd"
},
"credentials":{
"alias":"TGGATE"
},
"checkpoint":{
"table":"ggate.checkpoint"
},
"mode":{
"type":"integrated",
"parallel": false
},
"registration": "standard",
"begin":"now",
"status":"running"
}'
Replicats

Agenda
� Microservices (Best Practices)
� Deprecated and Desupported Features

Deprecated and Desupported Features
• Deprecated Features
• Oracle Sharding using OGG (still supported in 21c)
� Oracle GoldenGate Classic Architecture for Oracle and Heterogenous (DB2 z/OS, MySQL, MS SQL Server, and
PostgresSQL) (as of 21c)
• Desupprted Features
• Classic Extract for Oracle (deprecated as of 18c)
• Removal of NO_OBJECTDEFS and USE_TRAILDEFS parameters
• UDT Protocol in Distro Service and Receiver Service
• RMTTASK
• OPTIMIZE_PROGRESS_TABLE parameter
• NO_OBJECTDEFS and NO_TRAILDEFS parameters

In Closing
The future of Data Integration Automation is through
Microservices, which enables organizations to
transform, automate, and scale their data movement
environments.

https://www.linkedin.com/company/rheodata
@rheodatallc
info@rheodata.com
LinkedIn
Twitter
Email

Contact RheoData
info@rheodata.com
@rheodatallc http://rheodata.com
Transform, Automate, Scale … today and in the future!

Oracle GoldenGate 21c New Features and Best Practices

Recommended

More Related Content

What's hot (20)

Similar to Oracle GoldenGate 21c New Features and Best Practices (20)

More from Bobby Curtis (20)

Recently uploaded (20)

Oracle GoldenGate 21c New Features and Best Practices

Editor's Notes