R12 Service Management - OPMN

Hi Everyone!!

In this post, I will be covering the way OPMN manages the mid-tier services in R12.

you may would like to review the R12 installation post.

Once you have done the R12 installation, Mid-Tiers will be brought up automatically.

How Mid-Tier services are managed in R12??

R12 uses 10.1.3 fusion middle-ware AS stack, hence services in R12 is managed by OPMN(Oracle Process Manager and Notification Server). OPMN consists of two main components (Oracle Process Manager) and (Oracle Notification Server).

Oracle Process Manager is responsible for
1) starting
2) stopping
3) restarting
4) monitoring the services it manages (this includes death detection and automatic restart of the process)

Oracle Notification Server is the transport mechanism for failure, recovery, startup, and other related notifications between components in AS.

Single configuration file(opmn.xml) is used OPMN to manage the services. Config file location is given as $ORA_CONFIG_HOME/10.1.3/opmn/conf/opmn.xml

Services managed by opmn are (grep process-type opmn.xml)

1) HTTP_Server
2) oacore
3) forms
4) oafm

There are many benefits because of OPMN. To give an example,

Consider the scenario where one of your OC4J process has died. OPMN detects the death of the process which it manages and brings up in almost no time.

to elucidate this here is an example

-bash-2.05b$ ./adopmnctl.sh status

You are running adopmnctl.sh version 120.4

Checking status of OPMN managed processes...

Processes in Instance: r12test_ap6155rt.ap6155rt.us.oracle.com
-------------------+--------------------+---------+---------
ias-component | process-type | pid | status
-------------------+--------------------+---------+---------
OC4J | oafm | 30031 | Alive
OC4J | forms | 29934 | Alive
OC4J | oacore | 10444 | Alive
HTTP_Server| HTTP_Server| 29795 | Alive
ASG | ASG | N/A | Down


adopmnctl.sh: exiting with status 0

adopmnctl.sh: check the logfile /slot01/appmgr/R12/inst/apps/r12test_ap6155rt/logs/appl/admin/log/adopmnctl.txt for more information ...

-bash-2.05b$

All the OPMN managed processes are alive. Lets see, how opmn reacts to the death of the oacore OC4J process. I have killed the process with PID - 10444

-bash-2.05b$ kill -9 10444
-bash-2.05b$ ./adopmnctl.sh status

You are running adopmnctl.sh version 120.4

Checking status of OPMN managed processes...

Processes in Instance: r12test_ap6155rt.ap6155rt.us.oracle.com
-------------------+--------------------+---------+---------
ias-component | process-type | pid | status
-------------------+--------------------+---------+---------
OC4J | oafm | 30031 | Alive
OC4J | forms | 29934 | Alive
OC4J | oacore | 18813 | Init
HTTP_Server| HTTP_Server| 29795 | Alive
ASG | ASG | N/A | Down


adopmnctl.sh: exiting with status 0

adopmnctl.sh: check the logfile /slot01/appmgr/R12/inst/apps/r12test_ap6155rt/logs/appl/admin/log/adopmnctl.txt for more information ...

-bash-2.05b$

-bash-2.05b$ ./adopmnctl.sh status

You are running adopmnctl.sh version 120.4

Checking status of OPMN managed processes...

Processes in Instance: r12test_ap6155rt.ap6155rt.us.oracle.com
-------------------+--------------------+---------+---------
ias-component | process-type | pid | status
-------------------+--------------------+---------+---------
OC4J | oafm | 30031 | Alive
OC4J | forms | 29934 | Alive
OC4J | oacore | 18813 | Alive
HTTP_Server| HTTP_Server| 29795 | Alive
ASG | ASG | N/A | Down


adopmnctl.sh: exiting with status 0

adopmnctl.sh: check the logfile /slot01/appmgr/R12/inst/apps/r12test_ap6155rt/logs/appl/admin/log/adopmnctl.txt for more information ...

-bash-2.05b$

Within no time, OPMN detects the death and restarts the process it manages.(one or two seconds users will face the failures due to the process unavailability to serve the requests)

$LOG_HOME/ora/10.1.3/opmn/opmn.log is appened with the information ( it dumps the death detection and restart information)

08/01/21 07:09:45 [pm-process] Process Crashed: OC4J~oacore~default_group~1 (1238509304:10444) - Restarting
08/01/21 07:09:45 [pm-process] Starting Process: OC4J~oacore~default_group~1 (1238509305:0)
08/01/21 07:09:56 [pm-process] Process Alive: OC4J~oacore~default_group~1 (1238509305:18813)

This is one of greatest advantage you will get when you have R12, Even if oacore JVM crashes due to out of memory issue, opmn restarts after it detects OC4J has died.

What algorithm OPMN uses to detect the death?

1) OS process is checked by OPMN for every 2 seconds.
2) forward ping: periodically OPMN pings the process for every 20 seconds and expects response
3) reverse ping: every 20 seconds managed process sends OPMN a ping notification


So, as a administrator you may not find the issue that are existing in your environment till you review opmn logs as opmn restarts the process in no time, its rare people identifies the issue and raises support ticket.!!! :-)

I hope you find this article useful.

Leave your comments.!!

19 comments:

Atul Kumar said...

Nice Post Suresh.


Atul Kumar
http://onlineAppsDBA.com

Suresh Lakshmanan said...
This comment has been removed by the author.
Suresh Lakshmanan said...

Thank you so much Atul!

Suresh

Siva said...

Suresh,

Its an excellent enhancement by oracle in R12. And your explanation is beautiful.

Regards,
Shiva.

Suresh Lakshmanan said...

Hi Shiva,

Thanks for the encouragement!!

Regards
Sursh

Anonymous said...

Good,elborate and easy understandable post suresh..
We came across same senario in my client,where our's is an application server and 2 OC4J where soa and demantra deployed in each.

in our case Opmn restarting the 2 oc4j's some times,which is causing the users to logout.

We want to find out the reason of restarting in our case,it is an prod for us.

I want an info that how can find exacy cause for the issue.

Raghu Krishna

Suresh Lakshmanan said...

Hi Raghu,

Have you found anything interesting in the log files of opmn?

Thanks
Suresh

macdba said...

Hi Suresh,
Do you know a way where we can be informed via email whenever one of the components has died (example oc4j)? It could be a script or something?

Suresh Lakshmanan said...

Hi Macdba,

yes, grep the word "Process Crashed" in the oc4j log file and if you find hit then that has to be mailed as
mailx -s "oc4j crash" email@domain.com lt; content.txt

Thanks
Suresh

Raghu said...

Hi Suresh,


I am workin on R12.i have face d one issue recently.Oam Dash board Cm completed error.after that server is hang y it is happen every time.after my reseraching may problem with oacore or opmn.please guide me what i do


Regards

raghu

Suresh Lakshmanan said...

Hi Raghu,

What is the error message you are seeing in oacore/opmn log files?
I am online at meet.lsuresh at gmail.com /yahoo.co.in
during US timezone. If possible will help you over chat.

thanks
Suresh

ahmed said...

Hi,

Excellent post keep it up.

Thanks.

Akther Jaweed

Anonymous said...

Hi,

Excellent post.

Thanks & Regards

Akther Jaweed

Anonymous said...

God bless you Fantastic post my
doubt got clear
Shakeel Malik from kashmir

Sirisha said...

Excellent post which will clearly explain one of the advantages of R12 over 11i..good job

Sirisha

Anonymous said...

Awesome post Suresh sir !!

Amit

Anonymous said...

Awesome post Suresh Sir!!

Amit

Praveen Kasarla said...

Hi Suresh,

The way describing working of R12 services is excellent.
I have one query, what is the difference between adapcctl.sh and adoacorectl.sh. What action will performed when bounce of each service.

Thanks
Praveen K

Suresh Lakshmanan said...

Praveen,

apache techstack generally entry point where as oacore serves the business logic java codes.
if you shutdown oacore and have apache up then you would see static pages comes up without any issues.

Suresh