A three nodes Rac, the third node, when user connect, get below error:
ORA-02002: error while writing to audit trail
ORA-09925: Unable to create audit trail file
Linux-x86_64 Error: 28: No space left on device
The error seems the audit mount point is full.
Log on the server and check, the audit directory is:
/prod/product/corp_bac1/audit/trail
Let's check the audit directory mount point:
node3[oracle]_corp_bac3> df -h|grep /prod/product
34G 27G 5.1G 84% /prod/product
Wired, the mount point is not full:
node3[oracle]_corp_bac3> df -h|grep /prod/product
34G 27G 5.1G 84% /prod/product
Then why set get the error: "Linux-x86_64 Error: 28: No space left on device"?
Switch to audit directory and run ls -ltr, the screen flash with:
06:54 corp_bac3_ora_20624_2e.aud
06:54 corp_bac3_ora_20621_38.aud
06:54 corp_bac3_ora_20618_32.aud
06:54 corp_bac3_ora_20615_38.aud
06:54 corp_bac3_ora_20612_33.aud
06:54 corp_bac3_ora_20688_3c.aud
06:54 corp_bac3_ora_20685_34.aud
06:54 corp_bac3_ora_20682_3e.aud
06:54 corp_bac3_ora_20679_3e.aud
06:54 corp_bac3_ora_20676_3e.aud
06:54 corp_bac3_ora_20673_34.aud
06:54 corp_bac3_ora_20669_37.aud
06:54 corp_bac3_ora_20666_33.aud
06:54 corp_bac3_ora_20663_3c.aud
06:54 corp_bac3_ora_20660_32.aud
.........
.........
It showed hundreds of audit files generated in one minute.
I canceled the ls -ltr, and want to caculated the amount number of audit file under the directory:
[oracle@node3 trail]$ ls -F |grep /|wc
Above command is still running even after ten minutes, so i had to cancel it.
Till now it is already clear to me the issue is very possible caused by Inode resource used up:
$ df -h -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/rootvg-rootvol
15M 134K 15M 1% /
/dev/cciss/c0d0p1 65K 46 65K 1% /boot
tmpfs 16M 74 16M 1% /dev/shm
/dev/mapper/rootvg-lv_data
5.6M 4.4K 5.6M 1% /data
/dev/mapper/cihcispdb716_grid_vg-gridvol
2.2M 473K 1.7M 22% /prod/grid
/dev/mapper/cihcispdb716_product_vg-productvol
4.3M 4.3M 0K 100% /prod/product
3.24.148.15:/vol/database_backup
31M 37K 31M 1% /prod/backup
3.24.148.13:/vol/cis_orasoft
31M 5.5M 25M 19% /cis_orasoft
From above red part it shows Inodes used up. That's why we can't create any file even there are enough space left:
node3[oracle]_corp_bac3> df -h|grep /prod/product
34G 27G 5.1G 84% /prod/product
Cleaned some of the audit file and check:
[oracle@node3 11.2.0]$ df -h -i /prod/product
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/cihcispdb716_product_vg-productvol
4.3M 3.8M 444K 90% /prod/product
Connect the third node again, succeed.
GE, AGFA. experienced in 9i/10g/11g/12c, AdvanceReplication, Stream, GoldenGate, RAC, DataGuard, ODA, EXADATA, GridControl. In my career I have always been assigned to handle most complicated and critical oracle issues without solution online, and I hope to record some of them via this blog.
March 12, 2012
DB get an outage caused by INODE
Tags:
Inode
Subscribe to:
Post Comments (Atom)
0 Comments:
Post a Comment