RMAN failure after 11.2.0.3 BP#11 upgrade

Hi Everyone!!

Last week end we have upgraded our production Exadata databases to 11.2.0.3 along with Bundle Patch #11.   Patching went smoothly except few issue which happens always with any new code changes.
Oracle ACS engineers were onsite helping out with production and DR. After upgrade has been completed, we started seeing tape backup issues.

RMAN> # Backup all archive logs
2> run {
3> allocate channel t1 type 'SBT_TAPE' format '%U_%t';
4> allocate channel t2 type 'SBT_TAPE' format '%U_%t';
5> allocate channel t3 type 'SBT_TAPE' format '%U_%t';
6> allocate channel t4 type 'SBT_TAPE' format '%U_%t';
7> allocate channel t5 type 'SBT_TAPE' format '%U_%t';
8> send 'NB_ORA_POLICY=ORADB_XFINPRAC_HOT';
9> backup tag 'FINPROD  ARCHIVELOG Backup' filesperset 20 (archivelog all);
10> }
11>
12>
starting full resync of recovery catalog
full resync complete
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of allocate command on t1 channel at 01/23/2013 09:31:19
ORA-19554: error allocating device, device type: SBT_TAPE, device name:
ORA-27211: Failed to load Media Management Library
Additional information: 2

From the error it is clear that it is unable to create a channel.

RMAN> run{
2> allocate channel t1 type 'SBT_TAPE' format '%U_%t';
3>  }

using target database control file instead of recovery catalog
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of allocate command on t1 channel at 01/23/2013 09:31:18
ORA-19554: error allocating device, device type: SBT_TAPE, device name:
ORA-27211: Failed to load Media Management Library
Additional information: 2

RMAN>

I had to soft link MML client to fix the issue in both the RAC cluster RDBMS binaries.

$cd $ORACLE_HOME/lib
$ln -s /usr/openv/netbackup/bin/libobk.so64 libobk.so

Now I have tested the snippet. Errors vanished.
And we had to upgrade our RMAN catalog  db.

RMAN> run{
 allocate channel t1 type 'SBT_TAPE' format '%U_%t';
}2> 3>

using target database control file instead of recovery catalog
allocated channel: t1
channel t1: SID=3067 instance=finprod1 device type=SBT_TAPE
channel t1: Veritas NetBackup for Oracle - Release 7.0 (2010070800)
released channel: t1

RMAN> exit



Happy Sharing!!!