Applying an incremental backup to a physical standby

I recently had the situation where a physical standby (running 10.2.0.3) had got itself so far out of date, that transferring the required archived redo logs would have necessitated retrieving them from tape. As a far quicker method of getting this standby back up-to-date (as opposed to a complete rebuild), I decided to apply an incremental backup to the standby.

This procedure is actually reasonably well documented. Except this leaves the controlfile at one SCN while all your datafiles have marched up to the new scn. Upon restarting the managed recovery process, the standby is still asking for ancient archived logs that your datafiles already have the changes applied.

It is interesting to contrast the instructions for 10g above with similar, though not the same instructions for performing this on 11g. The 11g instructions ask for the controlfile to be restored as well. Except the restore standby controlfile did not work on 10.2.0.3 instance.

So here is my guide to applying an rman incremental backup to a physical standby.

    Find out where your standby thinks you have applied up to:

standby> alter database recover managed standby database cancel;

standby> select current_scn from v$database;

    Make an rman incremental backup on the primary from the above scn:

primary_rman> backup incremental from scn #### database format '/backup/standby_%U';

    Once this backup file is copied to the standby, catalog it with the standby controlfile:

standby_rman> catalog start with '/backup/standby';

    Now actually apply the changes from the backup to the standby:

standby_rman> recover database noredo;

    At this point your datafiles will be up-to-date, but your controlfile is still out of date, create new standby controlfile from the primary:

primary> alter database create standby controlfile as '/tmp/standby.ctl';

Copy this to the standby and replace your current standby controlfiles with this new one. Once this is done you should be able to restart managed recovery and it will take off from your backup scn.

It is this final stage that does not seem to be covered to well with the documentation. Certainly I found I needed this new standby controlfile after applying an incremental backup to my standby.

About these ads
Leave a comment

27 Comments

  1. Enrique Orbegozo

     /  June 3, 2008

    I’ve the same problem, but it was more complicated to solve because we’re using OMF and the new standby controlfile contains new names for the datafiles and online redo logs, so renaming them is an additional step.

    Reply
  2. jarneil

     /  June 3, 2008

    Hi Enrique,

    Thanks for reading!

    Yep, if the paths to the datafiles differs between primary and standby then you have extra work on your hands – as you would have done when initially creating the standby.

    jason.

    Reply
  3. Do you know if this will work without any level 0 or previous backup at all?

    Thanks.

    Reply
  4. jarneil

     /  June 3, 2008

    Hi mdinh,

    Yep I think so. Normally when doing an incremental backup, you don’t specify the scn number, so doing an incremental level 1 for example needs a level 0 and rman when doing the level 1 uses the scn of the level 0 backup and then chooses the blocks that have newer scns than this for the level 1 backup.

    In the example above you are actually telling rman which scn to use to compare with all the blocks so it should not really care. The documentation does not state a level 0 must exist for this strategy.

    Why you would not have a previous backup though, scares me somewhat!

    jason,

    Reply
  5. Luca

     /  June 4, 2008

    Good post Jason!
    In my opinion, another even more annoying, quirks of this procedure is that the incremental backup for recovery of standby has to be performed to disk [according to RMAN manual, the main reason is that "Backups on tape cannot be cataloged." and I see the point]. However in practice this can be an operational problem when using ASM (where file transfer is not too easy in 10g) and/or when the incremental backup is quite large.
    Do you see that also as a possible problem?
    I have the feeling that it should be not too difficult to come up with a work around to make use of tape backups for recovery of standby DBs. Any ideas?

    Cheers,
    Luca

    Reply
  6. jarneil

     /  June 4, 2008

    Hi Luca!

    Thanks for dropping by.

    This procedure was conducted on db’s utilizing ASM, but I was lucky in that the local OS filesystem had enough room to accomodate the incremental backup, it was then a simple matter of scp to the standby, register and away.

    I would envisage more pain to transfer if the backup is on ASM.

    I guess there comes a cut off point when the incremental gets so huge that you are as well off doing a complete re-install of your standby and this obviously can use a backup to tape. I suppose when your incremental is in the small 10’s of GB local filesystem might be viable.

    Actually Luca, have a look at http://download.oracle.com/docs/cd/B19306_01/backup.102/b14191/rcmdupdb008.htm

    That seems to be happy with an incremental from scn going to tape. I have not attempted this, so you may have more information than me!

    jason.

    Reply
  7. Luca

     /  June 4, 2008

    My point is that in the URL you linked I read “You must create this incremental backup on disk for it to be useful. When you move the incremental backup to the standby, you must catalog it at the standby as described in “Step 3: Catalog the Incremental Backup Files at the Standby Database”. Backups on tape cannot be cataloged.”

    So the ‘problem’ is: how to catalog a tape backup on the standby control file..

    L.

    Reply
  8. jarneil

     /  June 5, 2008

    Hi Luca,

    Ooops. My bad for not actually reading past the first rman command. Hmm. I see that an incremental from scn backup does not get registered with the controlfile when you do the backup, so the primary has no knowledge of it.

    When you find out, you should blog it Luca!

    jason.

    Reply
  9. Chris Ruel

     /  June 27, 2008

    Hey man, thanks for the life saver. I have a near TB database that somewhere lost a log a couple weeks ago and it was too late for me to get it back by the time I discovered it. Your part with the control file create saved me. I did not want to have to re-create from scratch and I was running into the same thing you did. By a stroke of luck I found your blog. Cheers!

    Reply
  10. jarneil

     /  June 27, 2008

    Hi Chris,

    I’m glad this post was of use!

    Thanks for your appreciative comment!

    cheers,

    jason.

    Reply
  11. Neat solution !

    Reply
  12. jarneil

     /  July 22, 2008

    Hi Hemant,

    Glad you liked it!

    This technique really can save you a lot of time, if you have lost an archivelog or indeeed have many to apply to the standby.

    Thanks for reading.

    cheers,

    jason.

    Reply
  13. karnash

     /  July 23, 2008

    Hi,
    I am trying to apply the incremental backup to my physical standby. I followed the steps but I get an error at step recover database noredo;
    RMAN-00571: ===========================================================
    RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
    RMAN-00571: ===========================================================
    RMAN-03002: failure of recover command at 07/23/2008 10:14:11
    ORA-19870: error reading backup piece /tmp/ForStandby_c4jm6hi4_1_1
    ORA-19573: cannot obtain exclusive enqueue for datafile 2
    The files are definately there and owned by the user.
    I am confused if the standby db should be in mount stage or not ??
    If not, how can RMAN connect to it.
    also, if I replace the newly created controlfile, what is the next step. Can you advise.

    Reply
  14. karnash

     /  July 23, 2008

    hi,
    I have followed the steps and applied the incremental backup’s. I have created standby controlfile on primary, shutdown the standby and then copied the standby controlfile as control01.ctl, control02.clt and then startup mount.

    but I get the following error:
    RMAN> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECT;

    RMAN-00571: ===========================================================
    RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
    RMAN-00571: ===========================================================
    RMAN-00558: error encountered while parsing input commands
    RMAN-01009: syntax error: found “recover”: expecting one of: “mount, open”
    RMAN-01007: at line 1 column 16 file: standard input

    Also,
    On primary :
    SQL> alter system switch logfile;
    SQL> SELECT MAX(R.SEQUENCE#) LAST_SEQ_RECD, MAX(L.SEQUENCE#) LAST_SEQ_SENT FROM
    V$ARCHIVED_LOG R, V$LOG L WHERE
    R.DEST_ID=2 AND L.ARCHIVED=’YES';

    LAST_SEQ_RECD LAST_SEQ_SENT
    ————- ————-
    8950 8950

    on standby

    idle> SELECT MAX(R.SEQUENCE#) LAST_SEQ_RECD, MAX(L.SEQUENCE#) LAST_SEQ_SENT FROM
    V$ARCHIVED_LOG R, V$LOG L WHERE
    R.DEST_ID=2 AND L.ARCHIVED=’YES';

    LAST_SEQ_RECD LAST_SEQ_SENT
    ————- ————-
    8949 8951

    Why is the LAST_SEQ_SENT on the standby ahead of the primary ??

    Can you please advise. Thanks

    Reply
  15. jarneil

     /  July 23, 2008

    Hello Karnash,

    I try and answer these questions in order:

    1. What version of Oracle are you using? If it’s got a 9 in it, I might not be able to help!

    2. Yes, the database must be in the mount stage.

    3 ORA-19571 – you are sure nothing else has this datafile open? The standby must be at the mount stage.

    4. Once you have replaced the now old controlfile(s) with a new standby controlfile, just startup mount the database and start managed recovery.

    5. You start the managed recovery from sql NOT an rman connection – it’s a sql command not an rman one.

    6. hmm, just look for what the current logfile is on the primary with

    select * from v$log;

    if the current is 8951 that is good, are you using LGWR to transport the redo?

    jason.

    Reply
  16. karnash

     /  July 23, 2008

    Hi Jason,
    Thanks for your reply.
    — I am having a 10.2.0.3 db on solaris.
    — I have tried to issue managed recovery from sql but I got this error:
    ERROR at line 1:
    ORA-01153: an incompatible media recovery is active
    so I shutdown immediate and then started the db in mount stage.
    — I am using data guard to keep the standby in sync with primary
    — EM shows the logs are being applied. So on primary, I do switch logfile and then that is shipped and applied to standby.
    — the alert log on the standby has
    Redo Shipping Client Connected as PUBLIC
    — Connected User is Valid
    RFS[1]: Assigned to RFS process 10750
    RFS[1]: Identified database type as ‘physical standby’
    Primary database is in MAXIMUM PERFORMANCE mode
    Thu Jul 24 08:51:19 2008
    RFS LogMiner: Client disabled from further notification
    Primary database is in MAXIMUM PERFORMANCE mode
    RFS[1]: Successfully opened standby log 4: ‘/u02/oradata/prd1/stby_redo_g4_m1.dbf’
    I think I am fine now….

    Reply
  17. karnash

     /  July 23, 2008

    Jason,
    Do I need to rename the data file and log files after using control file from primary on standby?

    Reply
  18. jarneil

     /  July 24, 2008

    Hi,

    Nope you should not have to rename them.

    If they are in a different location you can use a db_file_name_convert & log_file_name_convert.

    renaming should be last resort.

    jason.

    Reply
  19. jaydes

     /  August 21, 2008

    Hi,

    Id like to implement the same incremental setup on my standby database, problem is when i do a backup of the scn that reflects the scn from the standby i get the RMAN-06755 error. Question here is will this cause a problem? how do i get the backups so that it ties in with the SCN on the standby?

    jaydes

    Reply
  20. jarneil

     /  August 21, 2008

    Hi Jaydes,

    As far as I can see the RMAN error is saying that the SCN you are trying to take the incremental from is a later SCN than the SCN that your datafiles are at.

    So 2 questions:

    How are you determining which SCN to take the incremental backup from?- this should be determined on the standby.

    You are taking the backup on the primary, right?

    cheers,

    jason.

    Reply
  21. Joseph

     /  October 1, 2008

    Hi,
    Though I cataloged, it still reads from tape backup. How to avoid this.

    RMAN> catalog start with ‘/ORACLE-DATA/data03/stage/';

    …..

    RMAN> recover database;

    destination for restore of datafile 00242: /ORACLE-DATA/data03/PRWADG/datafile/o1_mf_xwiki_da_41ywnnvc_.dbf
    channel ORA_SBT_TAPE_1: reading from backup piece bk_un4jr9uof_s29412_p1_t666172175

    Thanks
    Joseph

    Reply
  22. jarneil

     /  October 3, 2008

    Hi Joseph,

    You want to talk to Luca, as he want’s to recover from tape on the standby!

    Heck, check your defaults:

    rman> show all;

    any clues?

    jason.

    Reply
  23. Hi (just 2 years later!)

    Just a quick comment on points 2 and 17/18. You are applying an incremental backup. That will update changed blocks in the existing datafiles on disk. At least in the case of a filesystem you won’t need to rename datafiles because, er, they’ve already been renamed when you originally created the standby and recovery is applied to datafile 1,2… not datafile /path/to/file1,/path/to/file2 etc. I’d expect this also to apply to OMF datafiles as well – though I’ll admit to not doing this myself yet.

    Reply
    • Hi Niall,

      Yes, when you apply the incremental you are using the controlfile that is already on the standby, and as you said this is already aware of where the datafiles are. It’s a bit of a pain in that you then have to create a new standby controlfile from the primary and obviously if your standby is using different paths you have a little bit more work on your hands at this stage.

      jason.

      Reply
  24. Any update on the RMAN-06755 error, Specify a smaller incremental-start SCN problem ?

    I tried to stop applying, switch log 3 times on primary, to make the gap, then make a backup with the last scn on standby, immediately I ran rman on prod, I get this error,

    question is will be the backup based on an earlier SCN cause problem when I use this “earlier” incremental backup to restore on standby.

    Reply
  25. any comment is appreciated.

    Reply
  26. jarneil

     /  September 5, 2011

    Hi Oliver,

    Where are you getting the scn to make the backup up to? When you have the scn, you can run a scn_to_timestamp on this scn to find what time the primary thinks this is corresponding to, check this makes sense. Also in the above testing you have done, it might be prudent to update some records.

    jason.

    Reply

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 53 other followers

%d bloggers like this: