Tuesday, October 15, 2013

RMAN Backup Catalog issues after Database migrated to new Host.

Recently we has faced Rman Catalog issue . Even if backup went fine, but the catalog is not giving the correct info  and getting the below error..

RMAN> list backup;

starting full resync of recovery catalog
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of list command at 10/14/2013 02:05:57
RMAN-03014: implicit resync of recovery catalog failed
RMAN-03009: failure of full resync command on default channel at 10/14/2013 02:05:57
RMAN-20052: invalid datafile create SCN

Steps to reproduce :

1. We take rman backup from primary database to a proxy server and from that server it goes to tape.
2. When ever we run the backup , the catalog does not work properly in primary database.
3. we tried to unregister and re - register the catalog .
4. The catalog and resync works before the backup begins .
5.After the backup the catalog throws the error mentioned in the catalog .

Finally we raised the P1 SR 3-7947905791 with Oracle Support, they have suggested that it's a bug known defect in our software related to databases with plugged in tablespaces.

You can verify using the steps in the following note. 
RMAN-20052: invalid datafile create SCN During Resync or Backup using Recovery Catalog 11.2.0.3 (Doc ID 1411883.1)

1. Run RMAN command RESYNC CATALOG in debug mode:
DEBUG ON;
RESYNC CATALOG;
DEBUG OFF;

2. Find the failing filenumber.
In the tracefile, search for ORA--20052 which is the error. A few lines up you will see 
DBGRESYNC: channel default: Calling checkDataFile for fileno 15 size 655360 [09:09:16.974] (resync) =============> checkDataFile for fileno 15 when error occured

3. Run the queries in the note (TARGET database)
SQL> select file#, ts# from v$datafile where file# =<fileno from debug trace>;
Make a note of the TS# for use in step 4. 
SQL> select fetsn, fenum, feplus, fecrc_scn, fepdi from x$kccfe where fetsn in (select ts# from v$datafile where file# =<fileno>);
SQL> select foreign_dbid, plugin_change#, plugged_in from v$datafile where ts# in (select ts# from v$datafile where file# =<fileno>);

4. Run the queries in the note (CATALOG database)
select ts#, file#, PLUGIN_SCN from df where ts# = 13;

If the symptoms of the note continues to match, download and apply the patch for defect 13000553 - RMAN BACKUP FAILS WITH RMAN-20999 ERROR AT STANDBY DATABASE