Recreate Physical Standby Database After Recovery of Primary Database
Recreate Physical Standby Database After Recovery of Primary Database
In Oracle 11g it is possible to create a Physical Standby Database from the Primary Database
using RMAN. Furthermore, the same procedure can be adopted in an existing Data Guard
environment to recreate the Physical Standby Database following recovery of the Primary
Database.
A number of reasons may influence your decision to recreate the Standby Database, these
include:
All the above have one thing in common; data synchronisation required between Primary and
Standby. In terms of restoration time, it may be advantageous to recreate the Standby Database
from the Primary. This is easily accomplished using RMAN‟s Oracle 11g new feature;
“duplicate target database for standby from active database”.
CRS, RDBMS & ASM Oracle Homes are intact on standby nodes
Original password file exists on the standby nodes [$ORACLE_HOME/dbs]
Original TNS configuration files exist on Primary and Standby nodes
CRS nodeapps and ASM instances are online on both Standby nodes
Working network exists between Primary and Standby nodes
The Primary Database can be opened by at least 1 instance
The Primary Database unique name is PRD
The Standby Database unique name is PRDSBY
In this example, both Physical and Standby databases are 2 node RAC using ASM shared
storage.
Delete Standby Database Files From ASM (if existing)
1. Ensure the Standby Database is shutdown (if available). As oracle user logon to Standby
Database Node A and type following command:
ps –ef | grep pmon | grep –v grep
If text similar to the following is returned by the above command, shutdown the database.
oracle 20844 1 0 Feb16 ? 00:00:07 ora_pmon_PRDSBY1
2. Logon to Standby Database Node A as oracle user and set ASM environment variables as
follows:
. oraenv
ORACLE_SID = [PRDSBY1] ? +ASM1
3. Delete the database files from ASM +DATA diskgroup using ASMCMD utility:
asmcmd rm ’+DATA/PRDSBY/CONTROLFILE/*’
asmcmd rm ’+DATA/PRDSBY/DATAFILE/*’
asmcmd rm ’+DATA/PRDSBY/ONLINELOG/*’
asmcmd rm ’+DATA/PRDSBY/TEMPFILE/*’
asmcmd rm ’+DATA/PRDSBY/PARAMETERFILE/*’
4. Delete the database files from ASM +FLASH diskgroup using ASMCMD utility:
asmcmd rm ’+FLASH/PRDSBY/CONTROLFILE/*’
asmcmd rm ’+FLASH/PRDSBY/BACKUPSET/*’
asmcmd rm ’+FLASH/PRDSBY/ONLINELOG/*’
Tip:
RMAN will not overwrite database files on the Standby, they must be removed first.
ASM will not allow the deletion of files if the database is open (files are accessed)
ASM will implicitly remove empty sub-directories, so only need to delete the files.
2. Create a controlfile for the Standby Database from the Primary Database. As the oracle user,
logon to Primary node A and execute the following commands:
srvctl stop database –d PRD
export ORACLE_SID=PRD1
sqlplus '/as sysdba'
SQL> STARTUP MOUNT;
SQL> ALTER DATABASE CREATE STANDBY CONTROLFILE AS '/tmp/PRDSBY.ctl';
SQL> ALTER DATABASE OPEN;
SQL> exit
3. As the oracle user, copy the control file to the Standby site node A:
4. Temporarily defer the shipping of logs to the Standby nodes. As oracle user, logon to
Primary node A and execute the following commands:
export ORACLE_SID=PRD1
sqlplus '/as sysdba'
SQL> alter system set log_archive_dest_state_2=defer scope=memory sid='*';
5. As the oracle user, logon to Standby node A and create a temporary database initialisation
file as follows:
vi /tmp/initPRDSBY.ora
db_name=PRD
remote_login_passwordfile=EXCLUSIVE
compatible=11.1.0
6. As the oracle user, logon to Standby node A and startup nomount Standby instance 1 using
the following commands:
export ORACLE_SID='PRDSBY1'
sqlplus '/ as sysdba'
startup nomount pfile='/tmp/initPRDSBY.ora'
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of Duplicate Db command at 02/24/2010 08:43:43
RMAN-03015: error occurred in stored script Memory Script
RMAN-03009: failure of backup command on prmy2 channel at 02/24/2010 08:43:43
ORA-17628: Oracle error 19505 returned by remote Oracle server
10. As oracle user, logon to Primary node A and test the connection to the Standby instance:
sqlplus sys/<password>@STBY_PRD_DGMGRL as sysdba
11. If connection succeeds, use RMAN to create a Physical Standby Database from the active
PRD (Primary) Database.
As oracle user, connect to the Primary Database using RMAN:
export ORACLE_SID=PRD1
rman target / auxiliary sys/<password>@STBY_PRD_DGMGRL
12. Then execute the following RMAN script to duplicate the Primary Database across the
network to the Standby:
run {
allocate channel prmy1 type disk;
allocate channel prmy2 type disk;
allocate channel prmy3 type disk;
allocate channel prmy4 type disk;
allocate auxiliary channel stby type disk;
duplicate target database for standby from active database
spfile
set db_unique_name='PRDSBY'
set control_files='/tmp/PRDSBY.ctl'
set instance_number='1'
set audit_file_dest='/u01/app/oracle/admin/PRD/adump'
set remote_listener='LISTENERS_PRDSBY'
nofilenamecheck;
}
exit
export ORACLE_SID=PRDSBY1
sqlplus '/as sysdba'
SQL> shutdown
14. Then startup mount the Standby Instance 1, create SPFILE on ASM shared storage from
temporary PFILE (see example in Appendix A) and shutdown the Instance:
SQL> startup mount pfile='/tmp/initPRDSBY1.ora'
SQL> create spfile='+DATA/PRDSBY/spfilePRDSBY.ora' from
pfile='/tmp/initPRDSBY1.ora';
SQL> shutdown
SQL> exit
15. Now rename the SPFILE created by RMAN to prevent Oracle using it to start the Standby
instance. As oracle user execute the following command on Standby Node A:
mv $ORACLE_HOME/dbs/spfilePRDSBY1.ora $ORACLE_HOME/dbs/spfilePRDSBY1.bak
16. As oracle user logon to Standby nodes A and B and check the database initialisation file
contains the following.
SPFILE='+DATA/PRDSBY/spfilePRDSBY.ora'
Node B
cat $ORACLE_HOME/dbs/initPRDSBY2.ora
SPFILE='+DATA/PRDSBY/spfilePRDSBY.ora'
18. Then change the value of the control_files Standby Database initialisation parameter to
„+DATA‟,‟ +FLASH‟:
19. Remaining on Standby node A, use RMAN to restore the Physical Standby Database
controlfile to ASM shared storage [+DATA and +FLASH diskgroups]:
rman target /
restore controlfile from ’/tmp/PRDSBY.ctl’
exit
22. From Standby node 1, place the Standby Database into Automatic Managed Recovery mode,
known as starting the Managed Recovery Process (MRP):
export ORACLE_SID=PRDSBY1
sqlplus '/as sysdba'
SQL> alter database recover managed standby database disconnect from session;
SQL> exit
23. Logon to Primary Database instance1 as SYSDBA and re-enable log shipping for Data
Guard as follows:
export ORACLE_SID=PRD1
sqlplus '/as sysdba'
SQL> alter system set log_archive_dest_state_2=enable scope=memory sid='*';
SQL> alter system archive log current;
SQL> exit
24. After a short while (enough time for Standby Database to synchronise with the Primary), stop
the Managed Recovery Process as follows:
export ORACLE_SID=PRDSBY1
sqlplus '/as sysdba'
SQL> alter database recover managed standby database cancel;
26. Logon Standby node 2 and open the database from instance 2:
export ORACLE_SID=PRDSBY2
sqlplus '/as sysdba'
SQL> alter database open;
SQL> exit
28. As oracle user, log back onto Primary node A and start database instance 2 using srvctl:
30. Execute the following SQL on the Standby Database to verify that redo data is being
received from the Primary, then registered and applied on the Standby :
export ORACLE_SID=PRDSBY1
sqlplus '/as sysdba'
SQL> alter session set nls_date_format=’YYYY-MON-DD HH24:MI:SS’;
SQL> SELECT SEQUENCE#, THREAD#, FIRST_TIME, NEXT_TIME, APPLIED
FROM V$ARCHIVED_LOG
ORDER BY THREAD#, SEQUENCE#;
Example:
SEQUENCE# THREAD# FIRST_TIME NEXT_TIME APPLIED
---------- ---------- -------------------- ----------------------------
1 1 2009-MAR-30 18:11:04 2009-MAR-30 18:14:30 YES
2 1 2009-MAR-30 18:14:30 2009-MAR-31 13:49:18 YES
3 1 2009-MAR-31 14:33:18 2009-MAR-31 14:33:28 YES
4 1 2009-MAR-31 14:33:28 2009-MAR-31 15:28:26 YES
5 1 2009-MAR-31 15:45:44 2009-MAR-31 15:45:50 YES
6 1 2009-MAR-31 15:45:50 2009-MAR-31 15:53:17 YES
7 1 2009-MAR-31 15:53:17 2009-MAR-31 16:00:37 YES
tail –f /u01/app/oracle/diag/rdbms/PRDSBY/PRDSBY1/trace/alert_PRDSBY1.log
Tip: Of course, the time out issue can be completely avoided by running the RMAN clone
commands within a shell script in the background. See Appendix C for an example script.
E.g.
To run the script in the background:
nohup RMAN_clone_script.ksh &
Should the time-out occur during the foreground execution, RMAN will continue to copy the
datafiles from the Primary site to the Standby site. However, RMAN will be unable to conduct
the post datafile copy operations which include switching to the datafile copy on the Standby
site.
Let‟s take a closer look at the datafile names and what has to done before the Physical Standby
database can be opened.
Below is an extract from the Physical Standby instance 1 alert log when attempting to start the
Managed Recovery Process (MRP):
alert_PRDSBY1.ora
-----------------
Completed: ALTER DATABASE MOUNT
Tue May 04 18:04:28 2010
alter database recover managed standby database disconnect from session
Attempt to start background Managed Standby Recovery process (PRDSBY1)
Tue May 04 18:04:28 2010
MRP0 started with pid=36, OS id=2762
MRP0: Background Managed Standby Recovery process started (PRDSBY1)
Fast Parallel Media Recovery enabled
Tue May 04 18:04:33 2010
Managed Standby Recovery not using Real Time Apply
Tue May 04 18:04:33 2010
Errors in file
/u01/app/oracle/diag/rdbms/prdsby/PRDSBY1/trace/PRDSBY1_dbw0_1357.trc:
ORA-01157: cannot identify/lock data file 1 - see DBWR trace file
ORA-01110: data file 1: '+DATA/PRD/datafile/system.263.701555551'
Note that Oracle cannot identify data file 1 that supports the System tablespace.
When we query v$datafile from the mounted instance we see all files have zero bytes:
sqlplus / as sysdba
SQL> select file#, bytes/(1024*1024) mbytes from v$datafile;
FILE# MBYTES
---------- ----------
1 0
2 0
3 0
4 0
5 0
6 0
7 0
8 0
9 0
10 0
11 0
12 0
13 0
14 0
This is because the datafiles need to be renamed to match those in the Standby control file (used
to start the instance). The original datafile and the output filenames can be found in the stdout
from the RMAN clone script. (See Appendix B)
E.g.
Another method to obtain the required datafile names is to look in ASM on the Standby site at
the physical files, using ASM command line utility asmcmd:
$ . oraenv
ORACLE_SID = [PRDSBY1] ? +ASM1
The Oracle base for ORACLE_HOME=/u01/app/oracle/product/11.1.0/asm is
/u01/app/oracle
$ asmcmd
ASMCMD> cd data/prdsby/datafile
ASMCMD> ls
PRODUCTION_ARCHIVE_DATA.287.718097165
PRODUCTION_DATA.262.718097165
PRODUCTION_INDEX.286.718097167
PRODUCTION_LOB.266.718104189
SNMP_DATA.276.718104929
$ . oraenv
ORACLE_SID = [+ASM1] ? PRDSBY1
The Oracle base for ORACLE_HOME=/u01/app/oracle/product/11.1.0/db_1 is
/u01/app/oracle
sqlplus / as sysdba
SQL> alter system set standby_file_management=MANUAL scope=spfile sid='*';
System altered.
SQL>shutdown
..
SQL> startup mount
E.g.
System altered.
FILE# MBYTES
---------- ----------
1 6280
2 1095.125
3 8300
4 3165
5 5
6 100
7 500
8 100
9 149052
10 115054.75
11 1024
12 136777.063
13 259.125
14 139.875
5. Yes, we have filesizes! Now start the MRP on Physical Standby database instance 1:
SQL> alter database recover managed standby database disconnect from session;
Database altered.
6. To open the database read-only, the MRP has to be stopped, database opened and MRP
restarted. This is dependant on the Primary and Standby databases being synchronised:
SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL; -- stop MRP
Database altered.
SQL> alter database recover managed standby database disconnect from session;
Database altered.
7. You do not need to rename the database tempfile as this is implicitly created by Oracle on
instance startup, as seen in the Physical Standby instance 1 alert log:
alert_PRDSBY1.ora
-----------------
ORA-01186: file 201 failed verification tests
ORA-01157: cannot identify/lock data file 201 - see DBWR trace file
ORA-01110: data file 201: '+DATA'
File 201 not verified due to error ORA-01157
Dictionary check complete
Re-creating tempfile +DATA as +DATA/prdsby/tempfile/temp.301.718189579
Another Gotcha!
Despite successfully cloning your Primary Database using RMAN, it is possible that when you
attempt to start the Managed Recovery Process (MRP) on the Physical Standby, it fails to the
following error seen in the database instance alert log:
This is due to the Primary Database having only 1 incarnation, while the Standby has many, as
seen below:
The ORA-19909 error can occur if the RMAN clone has been performed before, or the Standby
Database has been flashed back.
Solution
The solution is to find out the CURRENT incarnation from the Primary Database and reset the
Standby Database incarnation to match, using the following RMAN command:
rman target=/
RMAN > reset database to incarnation 1
RMAN > exit
sqlplus / as sysdba
SQL> alter database recover managed standby database disconnect from session;
Appendix A
Standby Database Initialisation Parameter File
*.archive_lag_target=0
*.audit_file_dest='/u01/app/oracle/admin/PRD/adump'
*.audit_trail='db'
*.cluster_database_instances=2
*.cluster_database=TRUE
*.compatible='11.1.0.0.0'
*.control_files='/tmp/PRDSBY.ctl'
*.db_block_size=8192
*.db_create_file_dest='+DATA'
*.db_domain=''
*.db_name='PRD'
*.db_recovery_file_dest='+FLASH'
*.db_recovery_file_dest_size=822486237184
*.db_unique_name='PRDSBY'
*.diagnostic_dest='/u01/app/oracle'
*.dispatchers='(PROTOCOL=TCP) (SERVICE=PRDSBYXDB)'
*.fal_client='PRDSBY'
*.fal_server='PRD'
PRD2.instance_number=2
PRD1.instance_number=1
PRDSBY1.instance_number=1
PRDSBY2.instance_number=2
PRDSBY1.local_listener='LISTENER_PRDSBY1'
PRDSBY2.local_listener='LISTENER_PRDSBY2'
*.log_archive_config='SEND, RECEIVE, DG_CONFIG=(PRD,PRDSBY)'
*.log_archive_dest_1='LOCATION=+FLASH VALID_FOR=(ALL_LOGFILES,ALL_ROLES)
DB_UNIQUE_NAME=PRDSBY'
*.log_archive_dest_2='SERVICE=PRD LGWR ASYNC
VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=PRD'
*.log_archive_dest_state_1='ENABLE'
*.log_archive_dest_state_2='ENABLE'
*.log_archive_format='prd_arch_%t_%s_%r.log'
*.log_archive_max_processes=4
*.max_dump_file_size='100M'
*.memory_target=21474836480
*.nls_language='ENGLISH'
*.nls_territory='UNITED KINGDOM'
*.open_cursors=300
*.processes=1000
*.remote_listener='LISTENERS_PRDSBY'
*.remote_login_passwordfile='exclusive'
*.service_names=''
Appendix B
Successful completion of RMAN duplicate database script
Recovery Manager: Release 11.1.0.7.0 - Production on Wed Feb 24 09:05:53 2010
RMAN> 2> 3> 4> 5> 6> 7> 8> 9> 10> 11> 12> 13> 14> 15>
using target database control file instead of recovery catalog
allocated channel: prmy1
channel prmy1: SID=1040 instance=PRD1 device type=DISK
RMAN>
export ORACLE_SID=PRD1
export ORACLE_HOME=/u01/app/oracle/product/11.1.0/db_1