May 18, 2012

Resource temporarily unavailable

One server, when we tried to switch to oracle user, it failed with below message:
[root@cihcissdb758 ~]# su - oracle
su: cannot set user id: Resource temporarily unavailable

Trying connecting remotly also reported same error.
But when use any other unix id to login to that server, turned out to be successful.


It is obviously that some resouce reached max limit for oracle user.
Maybe max open files, max threads, max Semaphore etc...

In our case it is max processes limit reached.
oracle $ cat /etc/security/limits.conf|grep hard|grep proc
oracle hard nproc 26384

[root@cihcissdb758 security]# ps -eLf|grep oracle|wc -l
24714

After further checking, 90% of threads looks like below:
oracle 32708 1 0 05:13 pts/64 00:00:04 /stage/product/db/11.2.0/db/jdk/bin/java -DORACLE_HOME=/stage/product/db/11.2.0/db -classpath /stage/product/db/11.2.0/db/jlib/netcfg.jar:/stage/product/db/11.2.0/db/jdk/jre/lib/rt.jar:/stage/product/db/11.2.0/db/jdk/jre/lib/i18n.jar:/stage/product/db/11.2.0/db/jlib/srvm.jar:/stage/product/db/11.2.0/db/jlib/srvmhas.jar:/stage/product/db/11.2.0/db/jlib/srvmasm.jar:/stage/product/db/11.2.0/db/jlib/supercluster.jar:/stage/product/db/11.2.0/db/jlib/superclustecommon.jar:/stage/product/db/11.2.0/db/ons/lib/ons.jar:/stage/product/db/11.2.0/db/srvm/jlib/srvctl.jar:/stage/product/db/11.2.0/db/jlib/gns.jar -Djava.net.preferIPv4Stack=true -Djava.util.logging.config.file=/stage/grid/11.2.0/grid/srvm/admin/logging.properties oracle.ops.opsctl.OPSCTLDriver config database -S 1


[root@cihcissdb758 security]# ps -eLf|grep oracle|grep java|wc -l
21714

Killed all java processes which PPID=1:
[root@cihcissdb758 security]# ps -ef|grep oracle|grep java| awk '{if ($3==1) print $2;}' |xargs kill -9

Now check again:
[root@cihcissdb758 ~]# ps -eLf|grep oracle|grep java|wc -l
34

Back to normal. And now we can switch or login to oracle normally.
[root@cihcissdb758 ~]# su - oracle
[oracle@cihcissdb758 ~]#

2 Comments:

Anonymous said...

hi. what make you say it process limit max was exceeded? yours was 1600+ below what being set in limits.conf (if i read and understand correctly). and why you kill the java with ppid 1? what is special about java ppid=1 process? tqvm

Anonymous said...

The trick is the -L param to show the threads. Thanks for de nice post!

Post a Comment