RDBMS patching in 11G RAC

Hello Everyone!!

Recently I happened to apply RDBMS interim patch in 11G RAC environment.

Here is the details how I have performed.

I did patching in rolling mode. Rolling mode can be explained by this pictorial diagram. You dont' require all the instances to be down.




Rac nodes in my environments are
Node1: rws60001tch
Node2: rws60002tch
Node3: rws60003tch

OPatch uses the following environment variables:
ORACLE_HOME — Oracle home location.
PATH — Path information.

I source the Node1's environment variables and set opatch executable in the path.

I brought down node1 and started the opatch from the patch directory. After patching done on node1, brought up node and brought down node2 and continued opatch, upon completion of node2 i started node2 DB and brought down node3 and started patching on node3. It was smooth patching. I have given the session details here for reference.

********************************************************************************
-bash-3.00$ opatch apply (started from node1)
Invoking OPatch 11.1.0.6.0

Oracle Interim Patch Installer version 11.1.0.6.0
Copyright (c) 2007, Oracle Corporation. All rights reserved.


Oracle Home : /slot01/tkr12r3m/db
Central Inventory : /SLOTS/slot01/tkr12r3m/oraInventory
from : /etc/oraInst.loc
OPatch version : 11.1.0.6.0
OUI version : 11.1.0.6.0
OUI location : /slot01/tkr12r3m/db/oui
Log file location : /slot01/tkr12r3m/db/cfgtoollogs/opatch/opatch2007-11-22_00-29-34AM.log

ApplySession applying interim patch '6417271' to OH '/slot01/tkr12r3m/db'

Running prerequisite checks...

OPatch detected the node list and the local node from the inventory. OPatch will patch the local system then propagate the patch to the remote nodes.


This node is part of an Oracle Real Application Cluster. (opatch identified node2 and node3 automatically)
Remote nodes: 'rws60002tch' 'rws60003tch'
Local node: 'rws60001tch'
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/slot01/tkr12r3m/db')


Is the local system ready for patching? [y|n] (opatch gets confirmation whether node 1 is down for patching)
y
User Responded with: Y
Backing up files and inventory (not for auto-rollback) for the Oracle Home
Backing up files affected by the patch '6417271' for restore. This might take a while...
Backing up files affected by the patch '6417271' for rollback. This might take a while...

Patching component oracle.rdbms, 11.1.0.6.0...
Updating archive file "/slot01/tkr12r3m/db/lib/libpls11.a" with "lib/libpls11.a/pkv.o"
Running make for target ioracle
Running make for target iextproc
Running make for target irman
Running make for target iwrap
Running make for target proc
ApplySession adding interim patch '6417271' to inventory

Verifying the update...
Inventory check OK: Patch ID 6417271 is registered in Oracle Home inventory with proper meta-data.
Files check OK: Files from Patch ID 6417271 are present in Oracle Home.

The local system has been patched. You can restart Oracle instances on it. (it says node1 patching done, i can start the instance on node1 now!!!)


Patching in rolling mode. (here is the info on patching type)

Remaining nodes to be patched:
'rws60002tch' 'rws60003tch'
What is the next node to be patched?
rws60002tch
You have selected 'rws60002tch' from 'rws60002tch' 'rws60003tch'

The node 'rws60002tch' will be patched next.


Please shutdown Oracle instances running out of this ORACLE_HOME on 'rws60002tch'.
(Oracle Home = '/slot01/tkr12r3m/db') - (here it asks me to shut down the node2)

Is the node ready for patching? [y|n]
y
User Responded with: Y
Updating nodes 'rws60002tch'
Apply-related files are:
FP = "/slot01/tkr12r3m/db/.patch_storage/6417271_Nov_19_2007_02_33_36/rac/copy_files.txt"
DP = "/slot01/tkr12r3m/db/.patch_storage/6417271_Nov_19_2007_02_33_36/rac/copy_dirs.txt"
MP = "/slot01/tkr12r3m/db/.patch_storage/6417271_Nov_19_2007_02_33_36/rac/make_cmds.txt"
RC = "/slot01/tkr12r3m/db/.patch_storage/6417271_Nov_19_2007_02_33_36/rac/remote_cmds.txt"

Instantiating the file "/slot01/tkr12r3m/db/.patch_storage/6417271_Nov_19_2007_02_33_36/rac/copy_files.txt.instantiated" by replacing $ORACLE_HOME in "/slot01/tkr12r3m/db/.patch_storage/6417271_Nov_19_2007_02_33_36/rac/copy_files.txt" with actual path.
Propagating files to remote nodes...
Instantiating the file "/slot01/tkr12r3m/db/.patch_storage/6417271_Nov_19_2007_02_33_36/rac/copy_dirs.txt.instantiated" by replacing $ORACLE_HOME in "/slot01/tkr12r3m/db/.patch_storage/6417271_Nov_19_2007_02_33_36/rac/copy_dirs.txt" with actual path.
Propagating directories to remote nodes...
Instantiating the file "/slot01/tkr12r3m/db/.patch_storage/6417271_Nov_19_2007_02_33_36/rac/make_cmds.txt.instantiated" by replacing $ORACLE_HOME in "/slot01/tkr12r3m/db/.patch_storage/6417271_Nov_19_2007_02_33_36/rac/make_cmds.txt" with actual path.
Running command on remote node 'rws60002tch':
cd /slot01/tkr12r3m/db/rdbms/lib; /usr/bin/make -f ins_rdbms.mk ioracle ORACLE_HOME=/slot01/tkr12r3m/db || echo REMOTE_MAKE_FAILED::>&2

Running command on remote node 'rws60002tch':
cd /slot01/tkr12r3m/db/rdbms/lib; /usr/bin/make -f ins_rdbms.mk iextproc ORACLE_HOME=/slot01/tkr12r3m/db || echo REMOTE_MAKE_FAILED::>&2

Running command on remote node 'rws60002tch':
cd /slot01/tkr12r3m/db/rdbms/lib; /usr/bin/make -f ins_rdbms.mk irman ORACLE_HOME=/slot01/tkr12r3m/db || echo REMOTE_MAKE_FAILED::>&2

Running command on remote node 'rws60002tch':
cd /slot01/tkr12r3m/db/plsql/lib; /usr/bin/make -f ins_plsql.mk iwrap ORACLE_HOME=/slot01/tkr12r3m/db || echo REMOTE_MAKE_FAILED::>&2

Running command on remote node 'rws60002tch':
cd /slot01/tkr12r3m/db/precomp/lib; /usr/bin/make -f ins_precomp.mk proc ORACLE_HOME=/slot01/tkr12r3m/db || echo REMOTE_MAKE_FAILED::>&2


The node 'rws60002tch' has been patched. You can restart Oracle instances on it.(patching done on node2, i have started node2 instance here, brougt down node3!!)


The node 'rws60003tch' will be patched next.


Please shutdown Oracle instances running out of this ORACLE_HOME on 'rws60003tch'.
(Oracle Home = '/slot01/tkr12r3m/db')

Is the node ready for patching? [y|n]
y
User Responded with: Y
Updating nodes 'rws60003tch'
Apply-related files are:
FP = "/slot01/tkr12r3m/db/.patch_storage/6417271_Nov_19_2007_02_33_36/rac/copy_files.txt"
DP = "/slot01/tkr12r3m/db/.patch_storage/6417271_Nov_19_2007_02_33_36/rac/copy_dirs.txt"
MP = "/slot01/tkr12r3m/db/.patch_storage/6417271_Nov_19_2007_02_33_36/rac/make_cmds.txt"
RC = "/slot01/tkr12r3m/db/.patch_storage/6417271_Nov_19_2007_02_33_36/rac/remote_cmds.txt"

Propagating files to remote nodes...
Propagating directories to remote nodes...
Instantiating the file "/slot01/tkr12r3m/db/.patch_storage/6417271_Nov_19_2007_02_33_36/rac/make_cmds.txt.instantiated" by replacing $ORACLE_HOME in "/slot01/tkr12r3m/db/.patch_storage/6417271_Nov_19_2007_02_33_36/rac/make_cmds.txt" with actual path.
Running command on remote node 'rws60003tch':
cd /slot01/tkr12r3m/db/rdbms/lib; /usr/bin/make -f ins_rdbms.mk ioracle ORACLE_HOME=/slot01/tkr12r3m/db || echo REMOTE_MAKE_FAILED::>&2

Running command on remote node 'rws60003tch':
cd /slot01/tkr12r3m/db/rdbms/lib; /usr/bin/make -f ins_rdbms.mk iextproc ORACLE_HOME=/slot01/tkr12r3m/db || echo REMOTE_MAKE_FAILED::>&2

Running command on remote node 'rws60003tch':
cd /slot01/tkr12r3m/db/rdbms/lib; /usr/bin/make -f ins_rdbms.mk irman ORACLE_HOME=/slot01/tkr12r3m/db || echo REMOTE_MAKE_FAILED::>&2

Running command on remote node 'rws60003tch':
cd /slot01/tkr12r3m/db/plsql/lib; /usr/bin/make -f ins_plsql.mk iwrap ORACLE_HOME=/slot01/tkr12r3m/db || echo REMOTE_MAKE_FAILED::>&2

Running command on remote node 'rws60003tch':
cd /slot01/tkr12r3m/db/precomp/lib; /usr/bin/make -f ins_precomp.mk proc ORACLE_HOME=/slot01/tkr12r3m/db || echo REMOTE_MAKE_FAILED::>&2


The node 'rws60003tch' has been patched. You can restart Oracle instances on it.

There were relinks on remote nodes. Remember to check the binary size and timestamp on the nodes 'rws60002tch' 'rws60003tch' .
The following make commands were invoked on remote nodes:
'cd /slot01/tkr12r3m/db/rdbms/lib; /usr/bin/make -f ins_rdbms.mk ioracle ORACLE_HOME=/slot01/tkr12r3m/db
cd /slot01/tkr12r3m/db/rdbms/lib; /usr/bin/make -f ins_rdbms.mk iextproc ORACLE_HOME=/slot01/tkr12r3m/db
cd /slot01/tkr12r3m/db/rdbms/lib; /usr/bin/make -f ins_rdbms.mk irman ORACLE_HOME=/slot01/tkr12r3m/db
cd /slot01/tkr12r3m/db/plsql/lib; /usr/bin/make -f ins_plsql.mk iwrap ORACLE_HOME=/slot01/tkr12r3m/db
cd /slot01/tkr12r3m/db/precomp/lib; /usr/bin/make -f ins_precomp.mk proc ORACLE_HOME=/slot01/tkr12r3m/db
'


OPatch succeeded.(node3 patching done, patching completed on all the three nodes)
-bash-3.00$


-bash-3.00$ opatch lsinventory ( verification of patch application)
Invoking OPatch 11.1.0.6.0

Oracle Interim Patch Installer version 11.1.0.6.0
Copyright (c) 2007, Oracle Corporation. All rights reserved.


Oracle Home : /slot01/tkr12r3m/db
Central Inventory : /SLOTS/slot01/tkr12r3m/oraInventory
from : /etc/oraInst.loc
OPatch version : 11.1.0.6.0
OUI version : 11.1.0.6.0
OUI location : /slot01/tkr12r3m/db/oui
Log file location : /slot01/tkr12r3m/db/cfgtoollogs/opatch/opatch2007-11-22_00-34-50AM.log

Lsinventory Output file location : /slot01/tkr12r3m/db/cfgtoollogs/opatch/lsinv/lsinventory2007-11-22_00-34-50AM.txt

--------------------------------------------------------------------------------
Installed Top-level Products (2):

Oracle Database 11g 11.1.0.6.0
Oracle Database 11g Examples 11.1.0.6.0
There are 2 products installed in this Oracle Home.


Interim patches (1) :

Patch 6417271 : applied on Thu Nov 22 00:30:56 PST 2007
Created on 19 Nov 2007, 02:33:36 hrs PST8PDT
Bugs fixed:
6417271



Rac system comprising of multiple nodes
Local node = rws60001tch
Remote node = rws60002tch
Remote node = rws60003tch

--------------------------------------------------------------------------------

OPatch succeeded.
-bash-3.00$
********************************************************************************

The best part of patching in RAC environment is you start the patching from node1, opatch copies the patch files automatically to the node2 and node3 and does the patching. You do not have to run opatch in node2 and node3 at all!!

Hope you got some good peice of info from this post... I planned some more posts from 11g as I have been working on 11g for a while.