June 27, 2012

LMS terminating the instance due to error 4031

One of our two node Rac DB encounter 04031 due to inbalance in mini sub heap.

After turned parameter for shared_pool, i decided to bounce instance one by one.
But when i was bouncing the instance on first node, the second node instance also crashed.

Below is in the logfile for second instance:

Wed Jun 27 04:57:28 2012
Reconfiguration started (old inc 4, new inc 6)
List of instances:
 2 (myinst: 2)
 Global Resource Directory frozen
 * dead instance detected - domain 0 invalid = TRUE
 Communication channels reestablished
 Master broadcasted resource hash value bitmaps
 Non-local Process blocks cleaned out
Wed Jun 27 04:57:28 2012
 LMS 1: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
Wed Jun 27 04:57:28 2012
 LMS 2: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
Wed Jun 27 04:57:28 2012
 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
 Set master node info
 Submitted all remote-enqueue requests
 Dwn-cvts replayed, VALBLKs dubious
 All grantable enqueues granted
 Post SMON to start 1st pass IR
Wed Jun 27 04:57:28 2012
Instance recovery: looking for dead threads
Beginning instance recovery of 1 threads
 parallel recovery started with 6 processes
Started redo scan
Completed redo scan
 read 202 KB redo, 74 data blocks need recovery
Wed Jun 27 04:57:32 2012
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /d001/product/ora_base/diag/rdbms/ttpoltd/ttpoltd2/trace/ttpoltd2_lms2_942.trc:
ORA-04031: unable to allocate 4136 bytes of shared memory ("shared pool","unknown object","sga heap(3,0)","gcs dynamic r")
LMS2 (ospid: 942): terminating the instance due to error 4031
Wed Jun 27 04:57:33 2012
System state dump requested by (instance=2, osid=942 (LMS2)), summary=[abnormal instance termination].
System State dumped to trace file /d001/product/ora_base/diag/rdbms/ttpoltd/ttpoltd2/trace/ttpoltd2_diag_920.trc
Wed Jun 27 04:57:34 2012
ORA-1092 : opitsk aborting process
Errors in file /d001/product/ora_base/diag/rdbms/ttpoltd/ttpoltd2/trace/ttpoltd2_diag_920.trc:
ORA-00601: cleanup lock conflict
Dumping diagnostic data in directory=[cdmp_20120627045733], requested by (instance=2, osid=942 (LMS2)), summary=[abnormal instance termination].
Wed Jun 27 04:57:35 2012
ORA-1092 : opitsk aborting process
Wed Jun 27 04:57:35 2012
License high water mark = 38
Instance terminated by LMS2, pid = 942
USER (ospid: 17522): terminating the instance
Instance terminated by USER, pid = 17522

From above log we can see during instance reconfigure progress, LMS hit 04031 and failed to progress with reconfigure operation, hence LMS terminated the instance.:

0 Comments:

Post a Comment