Fix corruption blocks caused by Server Crash

A server turned into hang status, swap space used up, and 95% cpu consuming by sys.
After sysadmin team reboot the server, I re-brought up all databases.


Then things become crazy, many alerts flood into our mail box reporting corruption blocks on different Databases.

After checking found different type of corruption, including:
1. normal data/index block fractured corruption.
2. UNDO block corruption. ( Luckily not system undo)
3. SYSAUX table corruption: CSC(0x0b57.10b514f6) higher than block SCN(0x0000.00000000)
and.....
4. database core bootstrap object con$ block corruption.

After three hours work finally fixed most of them. Below is one:
/d333/oracle/cdixx/diag/diag/rdbms/cdixx/cdixx/trace/cdi1d_j000_6213.trc (incident=205700):
ORA-01578: ORACLE data block corrupted (file # 51, block # 173106)
ORA-01110: data file 51: '/d333/data01/oracle/cdixx/data/cdi1data04.dbf'

Let's check the file 51 and block 173106:
SYS @ cdi1d > select OWNER,SEGMENT_NAME,SEGMENT_TYPE,BLOCK_ID from dba_extents where FILE_ID=51 and 173106>=BLOCK_ID and 173106<=(BLOCK_ID+BLOCKS);
OWNER     SEGMENT_NAME    SEGMENT_TYPE   BLOCK_ID
--------- ------------ --------------- ----------
CDI1XXX01     V_WOXXXR           TABLE     172869

When query on the table:
SYS @ cdi1d > select /*+full(t)*/ count(*) from CDI1XXX01.V_WOXXXR t;
ERROR:
ORA-01578: ORACLE data block corrupted (file # 51, block # 173106)
ORA-01110: data file 1: '/d333/data01/oracle/cdixx/data/cdi1data04.dbf'

Now let's correct the block：
BBED> set file 1;
FILE# 1

BBED> dump block 173106;
File: /d333/data01/oracle/cdixx/data/cdi1data04.dbf (1)
Block: 173106 Offsets: 0 to 511 Dba:0x0042a432
------------------------------------------------------------------------
06c20000 32a4c20c 00000000 0000ff04 b50e0000 01000000 bacb0300 72642324
570b0000 03003200 45a3c20c 0f002200 1f590000 a8254106 34167400 00a0570b
0f7f2124 03001500 0e100400 6e0a4209 d7bb8200 0080570b 68332324 17002400
a0040000 c3a44006 86013100 1b001f2a 00000000 00000000 00000000 00014500
00009c00 6e081608 6b320000 45000100 02000300 04000500 06000700 08000900

BBED> modify /x 33 offset 14
File: /d333/data01/oracle/cdixx/data/cdi1data04.dbf (1)
Block: 173106 Offsets: 14 to 525 Dba:0x0042a432
------------------------------------------------------------------------
3304b50e 00000100 0000bacb 03007264 2324570b 00000300 320045a3 c20c0f00
22001f59 0000a825 41063416 740000a0 570b0f7f 21240300 15000e10 04006e0a
4209d7bb 82000080 570b6833 23241700 2400a004 0000c3a4 40068601 31001b00
1f2a0000 00000000 00000000 00000001 45000000 9c006e08 16086b32 00004500
01000200 03000400 05000600 07000800 09000a00 0b000c00 0d000e00 0f001000

BBED> modify /x 33 offset 16380
File: /d333/data01/oracle/cdixx/data/cdi1data04.dbf (1)
Block: 173106 Offsets: 16380 to 16383 Dba:0x0042a432
------------------------------------------------------------------------
33060000

BBED> sum apply
Check value for File 1, Block 173106:
current = 0x0eb5, required = 0x0eb5

BBED> verify
DBVERIFY - Verification starting
FILE = /d333/data01/oracle/cdixx/data/cdi1data04.dbf
BLOCK = 173106

Block Checking: DBA = 214082610, Block Type = KTB-managed data block
data header at 0x2ac48be2c27c
kdbchk: xaction header lock count mismatch
trans=3 ilk=27 nlo=26
Block 173106 failed with check code 6108

DBVERIFY - Verification complete
Total Blocks Examined : 1
Total Blocks Processed (Data) : 1
Total Blocks Failing (Data) : 1
Total Blocks Processed (Index): 0
Total Blocks Failing (Index): 0
Total Blocks Empty : 0
Total Blocks Marked Corrupt : 0
Total Blocks Influx : 0
Message 531 not found; product=RDBMS; facility=BBED


Now let's query the table again:
SYS @ cdi1d > select /*+full(t)*/ count(*) from CDI1XXX01.V_WOXXXR t;
COUNT(*)
----------
472070

Fixed.