DR Gap resolution on RAC with ASM

Hi Everyone,

Due to data center network issues, primary RAC nodes were disconnected from DR RAC for few days. Since no space to hold the archive logs generated, RMAN pushed older archive logs to tape.

Once network issues have been resovled between the data center, DR was not catching up.
Configuration had 5 primary RAC nodes and DR of 5 nodes. preferred apply node was set at node5 in dgmgrl broker configuration.

You can read DR Gap detection article at
http://applicationsdba.blogspot.com/2009/03/dr-log-apply-gap-detection.html

Apply node alert log was reporting

Fri Oct 7 07:59:29 2011
FAL[client]: Failed to request gap sequence
GAP - thread 4 sequence 14878-14878
DBID 236715958 branch 723637176
FAL[client]: All defined FAL servers have been attempted.
-------------------------------------------------------------
Check that the CONTROL_FILE_RECORD_KEEP_TIME initialization
parameter is defined to a value that is sufficiently large
enough to maintain adequate log switch information to resolve
archivelog gaps.
-------------------------------------------------------------

Resolution:

If cooked file system is used then it would be easy to resolve the gap as scp the archivelogs over to DR and register will help.

With ASM, it is not simple scp but extra required.

Here is what I have done.

1. restore archive logs from tape
2. copy to cooked file system
3. scp over to DR
4. register the archived logs
5. start the MRP processes

-bash-3.2$ rman target=/
Recovery Manager: Release 10.2.0.4.0 - Production on Fri Oct 7 10:01:37 2011
Copyright (c) 1982, 2007, Oracle. All rights reserved.
connected to target database: ELPRD (DBID=236715958)

RMAN> connect catalog rman/rmanpa55w0rd@rman10g

connected to recovery catalog database

RMAN>

...
.. set policy etc.
.. I had to restore many archive logs from all 5 threads
.. I will show one here
...
RMAN> restore archivelog sequence 17531 thread 1;

Starting restore at 07-OCT-11
allocated channel: ORA_SBT_TAPE_1
channel ORA_SBT_TAPE_1: sid=1043 instance=ELPRD5 devtype=SBT_TAPE
channel ORA_SBT_TAPE_1: Veritas NetBackup for Oracle - Release 6.5 (2010042404)
allocated channel: ORA_SBT_TAPE_2
channel ORA_SBT_TAPE_2: sid=1082 instance=ELPRD5 devtype=SBT_TAPE
channel ORA_SBT_TAPE_2: Veritas NetBackup for Oracle - Release 6.5 (2010042404)
allocated channel: ORA_SBT_TAPE_3
channel ORA_SBT_TAPE_3: sid=1052 instance=ELPRD5 devtype=SBT_TAPE
channel ORA_SBT_TAPE_3: Veritas NetBackup for Oracle - Release 6.5 (2010042404)
allocated channel: ORA_SBT_TAPE_4
channel ORA_SBT_TAPE_4: sid=983 instance=ELPRD5 devtype=SBT_TAPE
channel ORA_SBT_TAPE_4: Veritas NetBackup for Oracle - Release 6.5 (2010042404)
using channel ORA_DISK_1
using channel ORA_DISK_2
using channel ORA_DISK_3
using channel ORA_DISK_4

channel ORA_SBT_TAPE_1: starting archive log restore to default destination
channel ORA_SBT_TAPE_1: restoring archive log
archive log thread=1 sequence=17531
channel ORA_SBT_TAPE_1: reading from backup piece 0bmo6saq_1_1_763588954
channel ORA_SBT_TAPE_1: restored backup piece 1
piece handle=0bmo6saq_1_1_763588954 tag=ELPRD5 ARCH 10/03/11 20:00
channel ORA_SBT_TAPE_1: restore complete, elapsed time: 00:00:56
Finished restore at 07-OCT-11

RMAN>
RMAN> copy archivelog '+FLASH/elprd/archivelog/2011_10_07/thread_1_seq_17531.24012.763888087' to '/u01/app/oracle/diag/DBA/thread_1_seq_17531.24012.763888087';

Starting backup at 07-OCT-11
released channel: ORA_SBT_TAPE_1
released channel: ORA_SBT_TAPE_2
released channel: ORA_SBT_TAPE_3
released channel: ORA_SBT_TAPE_4
using channel ORA_DISK_1
using channel ORA_DISK_2
using channel ORA_DISK_3
using channel ORA_DISK_4
channel ORA_DISK_1: starting archive copy
input archive log thread=1 sequence=17531 recid=140485 stamp=763888107
output filename=/u01/app/oracle/diag/DBA/thread_1_seq_17531.24012.763888087 recid=140486 stamp=763888277
channel ORA_DISK_1: archivelog copy complete, elapsed time: 00:00:04
Finished backup at 07-OCT-11

Starting Control File and SPFILE Autobackup at 07-OCT-11
piece handle=+FLASH/elprd/autobackup/2011_10_07/s_763888317.20198.763888321 comment=NONE
Finished Control File and SPFILE Autobackup at 07-OCT-11

RMAN>


Now we can scp the archive log /u01/app/oracle/diag/DBA/thread_1_seq_17531.24012.763888087 and register the logfile
using alter database register logfile 'file_name' syntax.

Hope this will help someone.

Happy Sharing!!