I recently had the situation where a physical standby (running 10.2.0.3) had got itself so far out of date, that transferring the required archived redo logs would have necessitated retrieving them from tape. As a far quicker method of getting this standby back up-to-date (as opposed to a complete rebuild), I decided to apply an incremental backup to the standby.
This procedure is actually reasonably well documented. Except this leaves the controlfile at one SCN while all your datafiles have marched up to the new scn. Upon restarting the managed recovery process, the standby is still asking for ancient archived logs that your datafiles already have the changes applied.
It is interesting to contrast the instructions for 10g above with similar, though not the same instructions for performing this on 11g. The 11g instructions ask for the controlfile to be restored as well. Except the restore standby controlfile did not work on 10.2.0.3 instance.
So here is my guide to applying an rman incremental backup to a physical standby.
- Find out where your standby thinks you have applied up to:
standby> alter database recover managed standby database cancel;
standby> select current_scn from v$database;
- Make an rman incremental backup on the primary from the above scn:
primary_rman> backup incremental from scn #### database format '/backup/standby_%U';
- Once this backup file is copied to the standby, catalog it with the standby controlfile:
standby_rman> catalog start with '/backup/standby';
- Now actually apply the changes from the backup to the standby:
standby_rman> recover database noredo;
- At this point your datafiles will be up-to-date, but your controlfile is still out of date, create new standby controlfile from the primary:
primary> alter database create standby controlfile as '/tmp/standby.ctl';
Copy this to the standby and replace your current standby controlfiles with this new one. Once this is done you should be able to restart managed recovery and it will take off from your backup scn.
It is this final stage that does not seem to be covered to well with the documentation. Certainly I found I needed this new standby controlfile after applying an incremental backup to my standby.
jarneil 
I’ve the same problem, but it was more complicated to solve because we’re using OMF and the new standby controlfile contains new names for the datafiles and online redo logs, so renaming them is an additional step.
Hi Enrique,
Thanks for reading!
Yep, if the paths to the datafiles differs between primary and standby then you have extra work on your hands – as you would have done when initially creating the standby.
jason.
Do you know if this will work without any level 0 or previous backup at all?
Thanks.
Hi mdinh,
Yep I think so. Normally when doing an incremental backup, you don’t specify the scn number, so doing an incremental level 1 for example needs a level 0 and rman when doing the level 1 uses the scn of the level 0 backup and then chooses the blocks that have newer scns than this for the level 1 backup.
In the example above you are actually telling rman which scn to use to compare with all the blocks so it should not really care. The documentation does not state a level 0 must exist for this strategy.
Why you would not have a previous backup though, scares me somewhat!
jason,
Good post Jason!
In my opinion, another even more annoying, quirks of this procedure is that the incremental backup for recovery of standby has to be performed to disk [according to RMAN manual, the main reason is that “Backups on tape cannot be cataloged.” and I see the point]. However in practice this can be an operational problem when using ASM (where file transfer is not too easy in 10g) and/or when the incremental backup is quite large.
Do you see that also as a possible problem?
I have the feeling that it should be not too difficult to come up with a work around to make use of tape backups for recovery of standby DBs. Any ideas?
Cheers,
Luca
Hi Luca!
Thanks for dropping by.
This procedure was conducted on db’s utilizing ASM, but I was lucky in that the local OS filesystem had enough room to accomodate the incremental backup, it was then a simple matter of scp to the standby, register and away.
I would envisage more pain to transfer if the backup is on ASM.
I guess there comes a cut off point when the incremental gets so huge that you are as well off doing a complete re-install of your standby and this obviously can use a backup to tape. I suppose when your incremental is in the small 10’s of GB local filesystem might be viable.
Actually Luca, have a look at http://download.oracle.com/docs/cd/B19306_01/backup.102/b14191/rcmdupdb008.htm
That seems to be happy with an incremental from scn going to tape. I have not attempted this, so you may have more information than me!
jason.
My point is that in the URL you linked I read “You must create this incremental backup on disk for it to be useful. When you move the incremental backup to the standby, you must catalog it at the standby as described in “Step 3: Catalog the Incremental Backup Files at the Standby Database”. Backups on tape cannot be cataloged.”
So the ‘problem’ is: how to catalog a tape backup on the standby control file..
L.
Hi Luca,
Ooops. My bad for not actually reading past the first rman command. Hmm. I see that an incremental from scn backup does not get registered with the controlfile when you do the backup, so the primary has no knowledge of it.
When you find out, you should blog it Luca!
jason.
Hey man, thanks for the life saver. I have a near TB database that somewhere lost a log a couple weeks ago and it was too late for me to get it back by the time I discovered it. Your part with the control file create saved me. I did not want to have to re-create from scratch and I was running into the same thing you did. By a stroke of luck I found your blog. Cheers!
Hi Chris,
I’m glad this post was of use!
Thanks for your appreciative comment!
cheers,
jason.
Neat solution !
Hi Hemant,
Glad you liked it!
This technique really can save you a lot of time, if you have lost an archivelog or indeeed have many to apply to the standby.
Thanks for reading.
cheers,
jason.
Hi,
I am trying to apply the incremental backup to my physical standby. I followed the steps but I get an error at step recover database noredo;
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of recover command at 07/23/2008 10:14:11
ORA-19870: error reading backup piece /tmp/ForStandby_c4jm6hi4_1_1
ORA-19573: cannot obtain exclusive enqueue for datafile 2
The files are definately there and owned by the user.
I am confused if the standby db should be in mount stage or not ??
If not, how can RMAN connect to it.
also, if I replace the newly created controlfile, what is the next step. Can you advise.
hi,
I have followed the steps and applied the incremental backup’s. I have created standby controlfile on primary, shutdown the standby and then copied the standby controlfile as control01.ctl, control02.clt and then startup mount.
but I get the following error:
RMAN> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECT;
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-00558: error encountered while parsing input commands
RMAN-01009: syntax error: found “recover”: expecting one of: “mount, open”
RMAN-01007: at line 1 column 16 file: standard input
Also,
On primary :
SQL> alter system switch logfile;
SQL> SELECT MAX(R.SEQUENCE#) LAST_SEQ_RECD, MAX(L.SEQUENCE#) LAST_SEQ_SENT FROM
V$ARCHIVED_LOG R, V$LOG L WHERE
R.DEST_ID=2 AND L.ARCHIVED=’YES’;
LAST_SEQ_RECD LAST_SEQ_SENT
————- ————-
8950 8950
on standby
idle> SELECT MAX(R.SEQUENCE#) LAST_SEQ_RECD, MAX(L.SEQUENCE#) LAST_SEQ_SENT FROM
V$ARCHIVED_LOG R, V$LOG L WHERE
R.DEST_ID=2 AND L.ARCHIVED=’YES’;
LAST_SEQ_RECD LAST_SEQ_SENT
————- ————-
8949 8951
Why is the LAST_SEQ_SENT on the standby ahead of the primary ??
Can you please advise. Thanks
Hello Karnash,
I try and answer these questions in order:
1. What version of Oracle are you using? If it’s got a 9 in it, I might not be able to help!
2. Yes, the database must be in the mount stage.
3 ORA-19571 – you are sure nothing else has this datafile open? The standby must be at the mount stage.
4. Once you have replaced the now old controlfile(s) with a new standby controlfile, just startup mount the database and start managed recovery.
5. You start the managed recovery from sql NOT an rman connection – it’s a sql command not an rman one.
6. hmm, just look for what the current logfile is on the primary with
select * from v$log;
if the current is 8951 that is good, are you using LGWR to transport the redo?
jason.
Hi Jason,
Thanks for your reply.
— I am having a 10.2.0.3 db on solaris.
— I have tried to issue managed recovery from sql but I got this error:
ERROR at line 1:
ORA-01153: an incompatible media recovery is active
so I shutdown immediate and then started the db in mount stage.
— I am using data guard to keep the standby in sync with primary
— EM shows the logs are being applied. So on primary, I do switch logfile and then that is shipped and applied to standby.
— the alert log on the standby has
Redo Shipping Client Connected as PUBLIC
— Connected User is Valid
RFS[1]: Assigned to RFS process 10750
RFS[1]: Identified database type as ‘physical standby’
Primary database is in MAXIMUM PERFORMANCE mode
Thu Jul 24 08:51:19 2008
RFS LogMiner: Client disabled from further notification
Primary database is in MAXIMUM PERFORMANCE mode
RFS[1]: Successfully opened standby log 4: ‘/u02/oradata/prd1/stby_redo_g4_m1.dbf’
I think I am fine now….
Jason,
Do I need to rename the data file and log files after using control file from primary on standby?
Hi,
Nope you should not have to rename them.
If they are in a different location you can use a db_file_name_convert & log_file_name_convert.
renaming should be last resort.
jason.
Hi,
Id like to implement the same incremental setup on my standby database, problem is when i do a backup of the scn that reflects the scn from the standby i get the RMAN-06755 error. Question here is will this cause a problem? how do i get the backups so that it ties in with the SCN on the standby?
jaydes
Hi Jaydes,
As far as I can see the RMAN error is saying that the SCN you are trying to take the incremental from is a later SCN than the SCN that your datafiles are at.
So 2 questions:
How are you determining which SCN to take the incremental backup from?- this should be determined on the standby.
You are taking the backup on the primary, right?
cheers,
jason.
Hi,
Though I cataloged, it still reads from tape backup. How to avoid this.
RMAN> catalog start with ‘/ORACLE-DATA/data03/stage/’;
…..
RMAN> recover database;
destination for restore of datafile 00242: /ORACLE-DATA/data03/PRWADG/datafile/o1_mf_xwiki_da_41ywnnvc_.dbf
channel ORA_SBT_TAPE_1: reading from backup piece bk_un4jr9uof_s29412_p1_t666172175
Thanks
Joseph
Hi Joseph,
You want to talk to Luca, as he want’s to recover from tape on the standby!
Heck, check your defaults:
rman> show all;
any clues?
jason.
Hi (just 2 years later!)
Just a quick comment on points 2 and 17/18. You are applying an incremental backup. That will update changed blocks in the existing datafiles on disk. At least in the case of a filesystem you won’t need to rename datafiles because, er, they’ve already been renamed when you originally created the standby and recovery is applied to datafile 1,2… not datafile /path/to/file1,/path/to/file2 etc. I’d expect this also to apply to OMF datafiles as well – though I’ll admit to not doing this myself yet.
Hi Niall,
Yes, when you apply the incremental you are using the controlfile that is already on the standby, and as you said this is already aware of where the datafiles are. It’s a bit of a pain in that you then have to create a new standby controlfile from the primary and obviously if your standby is using different paths you have a little bit more work on your hands at this stage.
jason.
Any update on the RMAN-06755 error, Specify a smaller incremental-start SCN problem ?
I tried to stop applying, switch log 3 times on primary, to make the gap, then make a backup with the last scn on standby, immediately I ran rman on prod, I get this error,
question is will be the backup based on an earlier SCN cause problem when I use this “earlier” incremental backup to restore on standby.
any comment is appreciated.
Hi Oliver,
Where are you getting the scn to make the backup up to? When you have the scn, you can run a scn_to_timestamp on this scn to find what time the primary thinks this is corresponding to, check this makes sense. Also in the above testing you have done, it might be prudent to update some records.
jason.