DB get an outage caused by INODE

A three nodes Rac, the third node, when user connect, get below error:
ORA-02002: error while writing to audit trail
ORA-09925: Unable to create audit trail file
Linux-x86_64 Error: 28: No space left on device

The error seems the audit mount point is full.

Log on the server and check, the audit directory is:
/prod/product/corp_bac1/audit/trail

Let's check the audit directory mount point:
node3[oracle]_corp_bac3> df -h|grep /prod/product
                       34G   27G  5.1G  84% /prod/product

Wired, the mount point is not full:
node3[oracle]_corp_bac3> df -h|grep /prod/product
                       34G   27G  5.1G  84% /prod/product

Then why set get the error: "Linux-x86_64 Error: 28: No space left on device"？

Switch to audit directory and run ls -ltr, the screen flash with：
06:54 corp_bac3_ora_20624_2e.aud
06:54 corp_bac3_ora_20621_38.aud
06:54 corp_bac3_ora_20618_32.aud
06:54 corp_bac3_ora_20615_38.aud
06:54 corp_bac3_ora_20612_33.aud
06:54 corp_bac3_ora_20688_3c.aud
06:54 corp_bac3_ora_20685_34.aud
06:54 corp_bac3_ora_20682_3e.aud
06:54 corp_bac3_ora_20679_3e.aud
06:54 corp_bac3_ora_20676_3e.aud
06:54 corp_bac3_ora_20673_34.aud
06:54 corp_bac3_ora_20669_37.aud
06:54 corp_bac3_ora_20666_33.aud
06:54 corp_bac3_ora_20663_3c.aud
06:54 corp_bac3_ora_20660_32.aud
.........
.........
It showed hundreds of audit files generated in one minute.

I canceled the ls -ltr, and want to caculated the amount number of audit file under the directory:
[oracle@node3 trail]$ ls -F |grep /|wc

Above command is still running even after ten minutes, so i had to cancel it.

Till now it is already clear to me the issue is very possible caused by Inode resource used up:
$ df -h -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/mapper/rootvg-rootvol
                         15M    134K     15M    1% /
/dev/cciss/c0d0p1        65K      46     65K    1% /boot
tmpfs                    16M      74     16M    1% /dev/shm
/dev/mapper/rootvg-lv_data
                        5.6M    4.4K    5.6M    1% /data
/dev/mapper/cihcispdb716_grid_vg-gridvol
                        2.2M    473K    1.7M   22% /prod/grid
/dev/mapper/cihcispdb716_product_vg-productvol
                        4.3M    4.3M      0K  100% /prod/product
3.24.148.15:/vol/database_backup
                         31M     37K     31M    1% /prod/backup
3.24.148.13:/vol/cis_orasoft
                         31M    5.5M     25M   19% /cis_orasoft

From above red part it shows Inodes used up. That's why we can't create any file even there are enough space left:
node3[oracle]_corp_bac3> df -h|grep /prod/product
                       34G   27G  5.1G  84% /prod/product

Cleaned some of the audit file and check：
[oracle@node3 11.2.0]$ df -h -i /prod/product
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/mapper/cihcispdb716_product_vg-productvol
                        4.3M    3.8M    444K   90% /prod/product

Connect the third node again, succeed.


More......