Due to a series of tests performed in a development environment, we forced the cancellation of some operations that were being executed on ZDL. As a result, metadata became lost and inconsistent, with missing records and values in required fields in one of the internal views of the RASYS user.
The error below was encountered during any operation executed in RMAN, connected to the ZDL catalog for the database used in the test. It is worth noting that this issue occurred exclusively on this database, and all other databases remained intact.
starting full resync of recovery catalog
got ORA-01400: cannot insert NULL into ("RASYS"."BCF"."CKP_TIME")
ORA-06512: at "RASYS.DBMS_RCVCAT", line 11796
ORA-06512: at "RASYS.DBMS_RCVCAT", line 12488 during resync
retrying with snapshot controlfile
Creating and using snapshot control file for resync
starting full resync of recovery catalog
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of list command at 04/03/2024 12:31:40
RMAN-03014: implicit resync of recovery catalog failed
RMAN-03009: failure of full resync command on default channel at 04/03/2024 12:31:40
ORA-01400: cannot insert NULL into ("RASYS"."BCF"."CKP_TIME")
In order to identify the backup that was causing the inconsistency, it was necessary to connect to the database with RMAN, without connecting to the catalog, and then analyze the output of the existing backups.
export NLS_DATE_FORMAT='DD-MON-YYYY HH24:MI:SS'
export datenow=`date '+%Y%m%d%H%M%S'`
rman target / nocatalog
set echo on
show all;
list incarnation of database;
report schema;
list backup;
LIST BACKUP by file;
report obsolete;
restore database preview;
In the list backup and list backup by file commands, you will find the following information regarding the problematic backup.
list of Control File Backups
============================
CF Ckp SCN Ckp Time BS Key S #Pieces #Copies Compressed Tag
---------- -------------------- ------- - ------- ------- ---------- ---
323782562 03-APR-2024 13:19:38 20964 A 1 1 NO TAG20240403T132352
323762299 03-APR-2024 12:10:47 20954 X 1 1 NO TAG20240403T121047
323756203 03-APR-2024 11:55:47 20953 X 1 1 NO TAG20240403T115547
323748848 20955 A 1 1 NO TAG20240403T114723 << No checkpoint time
323746276 03-APR-2024 11:20:45 20952 X 1 1 NO TAG20240403T112045
323738516 03-APR-2024 10:58:35 20951 A 1 1 NO TAG20240403T105835
323713346 03-APR-2024 09:36:27 20939 A 1 1 NO TAG20240403T093627
323690376 03-APR-2024 08:20:04 20926 A 1 1 NO TAG20240403T082004
323348733 02-APR-2024 15:47:46 20956 A 1 1 NO TAG20240402T155158
323348728 02-APR-2024 15:47:44 20915 X 1 1 NO TAG20240402T154744
323228446 02-APR-2024 08:19:49 20914 A 1 1 NO TAG20240402T081949
322931033 01-APR-2024 11:46:45 20903 A 1 1 NO TAG20240401T114646
322926808 01-APR-2024 11:26:16 20889 A 1 1 NO TAG20240401T112616
322924494 01-APR-2024 11:17:18 20856 X 1 1 NO TAG20240401T111718
322886954 01-APR-2024 08:19:43 20855 A 1 1 NO TAG20240401T081943
322545602 31-MAR-2024 08:19:48 20844 A 1 1 NO TAG20240331T081948
322173486 30-MAR-2024 08:19:44 20833 A 1 1 NO TAG20240330T081944
321803664 29-MAR-2024 08:20:06 20822 A 1 1 NO TAG20240329T082004
321565437 28-MAR-2024 17:34:19 20810 X 1 1 NO TAG20240328T173411
321557668 28-MAR-2024 17:11:13 20808 A 1 1 NO TAG20240328T171112
321554974 28-MAR-2024 17:03:45 20798 A 1 1 NO TAG20240328T170334
321553322 28-MAR-2024 17:26:43 20809 X 1 1 NO TAG20240328T172640
321443722 28-MAR-2024 08:19:48 20788 A 1 1 NO TAG20240328T081946
As a solution, Oracle suggested deleting the controlfile backup and re-registering the database in the ZDL:
Due to this fact it can't sync the entry with the catalog.
Action Plan :
-----------------
1) As a solution, delete the bad backup of controlfile
rman target / nocatalog
DELETE BACKUP OF CONTROLFILE TAG 'TAG20240403T114723';
2) Register the database in zdlra now and let us know if you are still seeing the errorAfter performing the above procedure, all operations were successfully executed.
Some interesting references that were used in diagnosing the issue.
ORA-01400: cannot insert NULL into (“RMAN”.”CCF”.”CKP_TIME”) during RMAN RESYNC ( Doc ID 458933.1 )
Bug 34990084 – RMAN: Resync Catalog fails with ORA-1400: cannot insert null into (“OWNER”.”bcf”.”ckp_time”) ( Doc ID 34990084.8 )
RMAN Register Database Fails With ORA-01400: cannot insert NULL into CDF.CKP_TIME ( Doc ID 2820142.1 )
My Piece of Data 