Showing posts with label rman. Show all posts
Showing posts with label rman. Show all posts

November 5, 2012

DEBUG RMAN: "RMAN-20052: invalid datafile create SCN" due to PLUGIN_SCN out of SYNC

Below error persists for one database during SYNC operation:

RMAN> resync catalog;

starting full resync of recovery catalog
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of resync command on default channel at 11/05/2012 07:10:03
RMAN-20052: invalid datafile create SCN

Target DB version:
11.2.0.2.6

Catalog DB version:
11.2.0.2.6

Catalog schema version:
catalog > Select * from rcver ;
VERSION
------------
11.02.00.03

After compare between target and catalog, found the create_scn for all datafiles and tempfiles are SYNC in both DB:
CATALOG DB:
catalog > select file#,CREATION_CHANGE# from RC_DATAFILE where DB_NAME='ATPPSS' order by 1;
     FILE# CREATION_CHANGE#
---------- ----------------
         1                7
         2             2178
         3           969548
         4            18151
         5          1002153
         6          1053956
         7          1054333
         8          1501572
         9          1502015
        10          1506616
        11          1515561
        12          1516921
        13          5361004
        14          5779799
14 rows selected.

TARGET DB:
target > select file#,CREATION_CHANGE# from v$datafile;
     FILE# CREATION_CHANGE#
---------- ----------------
         1                7
         2             2178
         3           969548
         4            18151
         5          1002153
         6          1053956
         7          1054333
         8          1501572
         9          1502015
        10          1506616
        11          1515561
        12          1516921
        13          5361004
        14          5779799
14 rows selected.

After debug rman session, we can see the error report when doing DBMS_RCVCAT.CHECKDATAFILE on file# 13:
DBGRPC:        krmxrpc - channel default kpurpc2 err=0 db=rcvcat proc=RMANADMIN.DBMS_RCVCAT.CHECKDATAFILE excl: 0
   DBGRCVCAT: checkDatafile - size changed for file#: 12 from 131072 to 153600
   DBGRCVCAT: incResynActions - Not debugging
   DBGRCVCAT: setCloneName: file#=12, creation_fname=1516921, plugin_change#=0, old_clone_fname=, new_clone_fname=NONE
DBGRESYNC:     channel default: Resyncing datafile +DATA/atppss/datafile/example.3079.785280031 [23:38:41.110] (resync)
DBGRESYNC:     channel default: Calling checkDataFile for fileno 13 size 131072 [23:38:41.110] (resync)
DBGRPC:        krmxrpc - channel default kpurpc2 err=20052 db=rcvcat proc=RMANADMIN.DBMS_RCVCAT.CHECKDATAFILE excl: 97
DBGRPC:        krmxrpc - channel default kpurpc2 err=0 db=rcvcat proc=RMANADMIN.DBMS_RCVCAT.CANCELCKPT excl: 0
   DBGRCVCAT: cancelCkpt - rollback, released all locks
DBGRPC:        krmxrpc - channel default kpurpc2 err=0 db=target proc=SYS.DBMS_BACKUP_RESTORE.CFILEUSECURRENT excl: 0

Below is the defination for 20052 in DBMS_RCVCAT.CHECKDATAFILE:
PROCEDURE CHECKDATAFILE()
IF (plugged_readonly = 'NO' AND create_scn > this_ckp_scn) THEN              
   raise_application_error(-20052, 'Invalid datafile create SCN');           
ELSIF (plugged_readonly = 'YES' AND plugin_scn > this_ckp_scn) THEN          
   raise_application_error(-20055, 'Invalid datafile plugin SCN');           
END IF;                                                                      
...........
   IF (create_scn = dfRec.create_scn AND                                     
       plugin_scn = dfRec.plugin_scn) THEN                                   
       ...........
   ELSIF ((case when plugged_readonly = 'NO' then                            
           create_scn else plugin_scn end) >                                 
          (case when dfRec.plugged_readonly = 'NO' then                      
           dfRec.create_scn else dfRec.plugin_scn end)) THEN
        ...........                 
   ELSE -- (create_scn < dfRec.create_scn)                                   
      -- The client passed us a create SCN for this datafile that is         
      -- less than the SCN in the rcvcat.  I.e., the target database         
      -- controlfile now contains a previous incarnation of this datafile.   
      -- This can happen only if the user has some old controlfile, or       
      -- has done a resetlogs and not told us about it.                      
      IF (plugged_readonly = 'NO') THEN                                      
         raise_application_error(-20052, 'Invalid datafile create SCN');     
      ELSE                                                                   
         raise_application_error(-20055, 'Invalid datafile plugin SCN');     
      END IF;
    END IF;

All the judgement done by three argument: plugin_scn, plugged_readonly,create_scn. Let's check above three values in both target DB and catalog DB: 
CATALOG DB:
catalog > select CREATE_SCN,PLUGGED_READONLY,PLUGIN_SCN from df where DBINC_KEY='3199420' and FILE# =13;
CREATE_SCN PLU PLUGIN_SCN
---------- --- ----------
   5361004 NO      992188

TARGET DB:
target > select CREATION_CHANGE#,PLUGGED_READONLY,PLUGIN_CHANGE# from v$datafile where file#=13;
CREATION_CHANGE# PLU PLUGIN_CHANGE#
---------------- --- --------------
         5361004 NO               0

Now we can clearly see that red part's differ is the reason we got the error.
Since pluggable-DB is a new feature in 12c, and not published in any current version as yet.
So currently all databases’ all datafiles’ PLUGIN_CHANGE# should be 0.

The PLUGIN_SCN is 992188 in CATALOG DB is obviously not correct. Un-Register/Re-register the DB in catalog may or may not help to solve this issue.
More......

2 Comments
Tags： , , , , , , , , , ,

June 30, 2012

RMAN blocked by ARCH process, waiting “enq: WL - contention”

Today, we got a file system alert, one server's arch mount point is 90% full.

After checking found one DB's archivelog occupied 90% space.
And this DB's backup arch job was running since 10 days ago and still hang there, no progress.

After checking in the DB, found the rman job was blocked by below red part event:

      SSID    OSUSER     USERNAME          STATE      STATUS                    MODULE
---------- --------- ------------ -------------- ------------ -------------------------                                                  
BLOCKING_SESSION                   EVENT                          PROGRAM
---------------- ----------------------- --------------------------------
       244    oracle          SYS        WAITING       ACTIVE  rman@server621 (TNS V1-V3)
              58     enq: WL - contention       rman@server621 (TNS V1-V3)
SQL_TEXT
--------------------------------------
begin dbms_rcvman.setDatabase(upper(:dbname), :rlscn, :rltime, :fhdbi, :db_unique_name, TRUE); dbms_rcvman.setCanApplyAnyRedo(TRUE); dbms_rcvman.setCanConvertCf(TRUE); if (:canhandletts = 1) then   dbms_rcvman.setCanHandleTransportableTbs(TRUE); end if; end;

It is blocked by session 58, which is ARC0 process:
SID     SPID                        ROGRAM
------ -------- -------------------------- 
    58    10899    oracle@server621 (ARC1)

Killing ARC0 process, oracle will restart it automaticlly.
Then the rman backup job began to progress normally.
More......

1 Comments
Tags： , , ,