On this article I am going to simulate how ZDLRA behave when a protected database becomes down for fail reason.
According to a document from Oracle, Zero Data Loss Recovery Appliance, break the streaming redo log shipping, create a parcial backup with redo data shipped until the crash. When protected database becomes up again, Oracle restart the online redo log apply, ZDLRA finish to create the backups with remains redo data and everything back to normal behavior.
Let’s test it.
First, force some switches to identify the sequence of each redo on each cluster node:
[oracle@node1 ~]$ sqlplus / as sysdba
SQL*Plus: Release 19.3.0.0.0 - Production on Wed Apr 15 15:03:13 2026
Version 19.30.0.0.0
Copyright (c) 1982, 2025, Oracle. All rights reserved.
Connected to:
Oracle Database 19c Enterprise Edition Release 19.3.0.0.0 - Production
Version 19.30.0.0.0
SQL> alter system switch logfile;
System altered.
SQL> alter system switch logfile;
System altered.
SQL>
[oracle@node2 ~]$ sqlplus / as sysdba
SQL*Plus: Release 19.3.0.0.0 - Production on Wed Apr 15 15:03:17 2026
Version 19.30.0.0.0
Copyright (c) 1982, 2025, Oracle. All rights reserved.
Connected to:
Oracle Database 19c Enterprise Edition Release 19.3.0.0.0 - Production
Version 19.30.0.0.0
SQL> alter system switch logfile;
System altered.
SQL> alter system switch logfile;
System altered.
SQL>
To verify if redo data was sent to Zero Data Loss Recovery Appliance, we need check ZDLRA dbnode alert log file. Check in both nodes, in my case, data was sent to database node 1, zdl02dbadm01. Pass protected database DBID to tail command to more accurate in your search:
[oracle@zdl02dbadm01 trace]$ tail -f alert_zdlra021.log |grep 1420112693
rfs (PID:28925): Opened log for T-1.S-420 dbid 1420112693 branch 1230214881
rfs (PID:28557): Opened log for T-2.S-416 dbid 1420112693 branch 1230214881
rfs (PID:28925): Opened log for T-1.S-421 dbid 1420112693 branch 1230214881
rfs (PID:28557): Opened log for T-2.S-417 dbid 1420112693 branch 1230214881
Now, let’s simulate a crash in protected database, killing pmon and smon processes.
[oracle@node1 rdbms]$ ps -ef |egrep 'pmon|smon'
oracle 85054 1 0 14:06 ? 00:00:00 ora_pmon_ORCL1
oracle 85400 1 0 14:06 ? 00:00:00 ora_smon_ORCL1
oracle 166702 3773914 0 15:05 pts/7 00:00:00 grep -E --color=auto pmon|smon
root 3089441 1 1 Feb10 ? 23:52:37
/u01/grid/19.3.0.0/bin/osysmond.bin
oracle 3818563 1 0 09:10 ? 00:00:01 asm_pmon_+ASM1
oracle 3818662 1 0 09:10 ? 00:00:00 asm_smon_+ASM1
oracle 3824180 1 0 09:12 ? 00:00:01 mdb_pmon_-MGMTDB
oracle 3824560 1 0 09:12 ? 00:00:00 mdb_smon_-MGMTDB
[oracle@node1 rdbms]$ kill -9 85400 85054
[oracle@node2 ~]$ ps -ef |egrep 'pmon|smon'
root 7598 1 1 Feb10 ? 21:38:53
/u01/grid/19.3.0.0/bin/osysmond.bin
oracle 1208179 1 0 09:12 ? 00:00:01 asm_pmon_+ASM2
oracle 1208831 1 0 09:12 ? 00:00:00 asm_smon_+ASM2
oracle 1620286 1 0 14:06 ? 00:00:00 ora_pmon_ORCL2
oracle 1620628 1 0 14:06 ? 00:00:00 ora_smon_ORCL2
oracle 1694970 1165529 0 15:05 pts/1 00:00:00 grep -E --color=auto pmon|smon
[oracle@node2 ~]$ kill -9 1620628 1620286
Now, let’s start the protected database in no mount, just to check how Recovery Appliance behaves:
[oracle@node1 rdbms]$ srvctl start database -d ORCL -o nomount
Checking generated backups in the crash moment:
[oracle@node1 rdbms]$ rman target / catalog /@zdlra02
Recovery Manager: Release 19.3.0.0.0 - Production on Wed Apr 15 15:07:20 2026
Version 19.30.0.0.0
Copyright (c) 1982, 2019, Oracle and/or its affiliates. All rights reserved.
connected to target database: ORCL (not mounted)
connected to recovery catalog database
recovery catalog schema version 23.04.00.23. is newer than RMAN version
RMAN> list backup of archivelog sequence 421 thread 1;
list backup of archivelog sequence 421 thread 1;
List of Backup Sets
===================
BS Key Size Device Type Elapsed Time Completion Time
------- ---------- ----------- ------------ ---------------
1074088883 157.50K SBT_TAPE 00:00:01 20260415
BP Key: 1074088884 Status: AVAILABLE Compressed: YES Tag: TAG20260415T150546
Handle: $RSCN_164062765_RTIM_1230214881_THRD_1_SEQ_421_CTKEY_1074088807_BACKUP
Media:
List of Archived Logs in backup set 1074088883
Thrd Seq Low SCN Low Time Next SCN Next Time
---- ------- ---------- -------- ---------- ---------
1 421 45952023980444 20260415 45952023980927 20260415
RMAN> list backup of archivelog sequence 420 thread 1;
list backup of archivelog sequence 420 thread 1;
List of Backup Sets
===================
BS Key Size Device Type Elapsed Time Completion Time
------- ---------- ----------- ------------ ---------------
1074088809 7.50K SBT_TAPE 00:00:00 20260415
BP Key: 1074088810 Status: AVAILABLE Compressed: YES Tag: TAG20260415T150341
Handle: $RSCN_164062765_RTIM_1230214881_THRD_1_SEQ_420_CTKEY_1074088796_BACKUP
Media:
List of Archived Logs in backup set 1074088809
Thrd Seq Low SCN Low Time Next SCN Next Time
---- ------- ---------- -------- ---------- ---------
1 420 45952023980422 20260415 45952023980444 20260415
RMAN> list backup of archivelog sequence 422 thread 1;
list backup of archivelog sequence 422 thread 1;
specification does not match any backup in the repository
RMAN> list backup of archivelog sequence 417 thread 2;
list backup of archivelog sequence 417 thread 2;
List of Backup Sets
===================
BS Key Size Device Type Elapsed Time Completion Time
------- ---------- ----------- ------------ ---------------
1074088879 199.50K SBT_TAPE 00:00:00 20260415
BP Key: 1074088880 Status: AVAILABLE Compressed: YES Tag: TAG20260415T150546
Handle: $RSCN_164062765_RTIM_1230214881_THRD_2_SEQ_417_CTKEY_1074088813_BACKUP
Media:
List of Archived Logs in backup set 1074088879
Thrd Seq Low SCN Low Time Next SCN Next Time
---- ------- ---------- -------- ---------- ---------
2 417 45952023980459 20260415 45952023980927 20260415
RMAN> list backup of archivelog sequence 416 thread 2;
list backup of archivelog sequence 416 thread 2;
List of Backup Sets
===================
BS Key Size Device Type Elapsed Time Completion Time
------- ---------- ----------- ------------ ---------------
1074088818 8.00K SBT_TAPE 00:00:00 20260415
BP Key: 1074088819 Status: AVAILABLE Compressed: YES Tag: TAG20260415T150343
Handle: $RSCN_164062765_RTIM_1230214881_THRD_2_SEQ_416_CTKEY_1074088798_BACKUP
Media:
List of Archived Logs in backup set 1074088818
Thrd Seq Low SCN Low Time Next SCN Next Time
---- ------- ---------- -------- ---------- ---------
2 416 45952023980434 20260415 45952023980459 20260415
RMAN> list backup of archivelog sequence 418 thread 2;
list backup of archivelog sequence 418 thread 2;
specification does not match any backup in the repository
RMAN>
As we can see, ZDLRA generated backups of all redo log data Opened to stream to Appliance before the crash. When the crash happen, Zero Data Loss Recovery Appliance, close the streaming and create the parcial redo data backup.
Now, let’s open the database and check what happen:
[oracle@node1 rdbms]$ srvctl stop database -d ORCL
[oracle@node1 rdbms]$ srvctl start database -d ORCL
Again, check in both ZDLRA nodes the redo log shipping. As we can see, now, each protected database node, is sending data for different dbnodes of Appliance. Also, as you can see, not only nexts but the lasts protected database sequences were opened to streaming:
[oracle@zdl02dbadm01 trace]$ tail -f alert_zdlra021.log |grep 1420112693
rfs (PID:28925): Opened log for T-1.S-420 dbid 1420112693 branch 1230214881
rfs (PID:28557): Opened log for T-2.S-416 dbid 1420112693 branch 1230214881
rfs (PID:28925): Opened log for T-1.S-421 dbid 1420112693 branch 1230214881
rfs (PID:28557): Opened log for T-2.S-417 dbid 1420112693 branch 1230214881
rfs (PID:181703): Opened log for T-2.S-418 dbid 1420112693 branch 1230214881
rfs (PID:181711): Opened log for T-2.S-417 dbid 1420112693 branch 1230214881
[oracle@zdl02dbadm02 trace]$ tail -f alert_zdlra022.log |grep 1420112693
rfs (PID:351889): Opened log for T-1.S-421 dbid 1420112693 branch 1230214881
rfs (PID:352174): Opened log for T-1.S-422 dbid 1420112693 branch 1230214881
rfs (PID:352172): Opened log for T-1.S-423 dbid 1420112693 branch 1230214881
Now, let’s check the backups:
[oracle@node1 rdbms]$ rman target / catalog /@zdlra02
Recovery Manager: Release 19.3.0.0.0 - Production on Wed Apr 15 15:12:21 2026
Version 19.30.0.0.0
Copyright (c) 1982, 2019, Oracle and/or its affiliates. All rights reserved.
connected to target database: ORCL (DBID=1420112693)
connected to recovery catalog database
recovery catalog schema version 23.04.00.23. is newer than RMAN version
RMAN> list backup of archivelog sequence 421 thread 1;
list backup of archivelog sequence 421 thread 1;
List of Backup Sets
===================
BS Key Size Device Type Elapsed Time Completion Time
------- ---------- ----------- ------------ ---------------
1074089132 157.50K SBT_TAPE 00:00:00 20260415
BP Key: 1074089133 Status: AVAILABLE Compressed: YES Tag: TAG20260415T151102
Handle: $RSCN_164062765_RTIM_1230214881_THRD_1_SEQ_421_CTKEY_1074089126_BACKUP
Media:
List of Archived Logs in backup set 1074089132
Thrd Seq Low SCN Low Time Next SCN Next Time
---- ------- ---------- -------- ---------- ---------
1 421 45952023980444 20260415 45952024080928 20260415
RMAN> list backup of archivelog sequence 422 thread 1;
list backup of archivelog sequence 422 thread 1;
List of Backup Sets
===================
BS Key Size Device Type Elapsed Time Completion Time
------- ---------- ----------- ------------ ---------------
1074089142 4.00K SBT_TAPE 00:00:00 20260415
BP Key: 1074089143 Status: AVAILABLE Compressed: YES Tag: TAG20260415T151108
Handle: $RSCN_164062765_RTIM_1230214881_THRD_1_SEQ_422_CTKEY_1074089136_BACKUP
Media:
List of Archived Logs in backup set 1074089142
Thrd Seq Low SCN Low Time Next SCN Next Time
---- ------- ---------- -------- ---------- ---------
1 422 45952024080928 20260415 45952024081417 20260415
RMAN> list backup of archivelog sequence 417 thread 2;
list backup of archivelog sequence 417 thread 2;
List of Backup Sets
===================
BS Key Size Device Type Elapsed Time Completion Time
------- ---------- ----------- ------------ ---------------
1074089128 199.50K SBT_TAPE 00:00:00 20260415
BP Key: 1074089129 Status: AVAILABLE Compressed: YES Tag: TAG20260415T151100
Handle: $RSCN_164062765_RTIM_1230214881_THRD_2_SEQ_417_CTKEY_1074089125_BACKUP
Media:
List of Archived Logs in backup set 1074089128
Thrd Seq Low SCN Low Time Next SCN Next Time
---- ------- ---------- -------- ---------- ---------
2 417 45952023980459 20260415 45952024080929 20260415
RMAN> list backup of archivelog sequence 418 thread 2;
list backup of archivelog sequence 418 thread 2;
specification does not match any backup in the repository
As we can see, during the recovery process when opening the protected databases, backups were created using the redo logs data switched and sent to ZDLRA.
Sequence 418 on thread 2 still opened on ZDLRA to build consistent data to sent to ZDLRA. Forcing a switch, this sequence was closed and also sent to ZDLRA create the backup.
[oracle@node2 ~]$ sqlplus / as sysdba
SQL*Plus: Release 19.3.0.0.0 - Production on Wed Apr 15 15:15:04 2026
Version 19.30.0.0.0
Copyright (c) 1982, 2025, Oracle. All rights reserved.
Connected to:
Oracle Database 19c Enterprise Edition Release 19.3.0.0.0 - Production
Version 19.30.0.0.0
SQL> alter system switch logfile;
System altered.
SQL>
Check on ZDLRA alert logs:
[oracle@zdl02dbadm01 trace]$ tail -f alert_zdlra021.log |grep 1420112693
rfs (PID:28925): Opened log for T-1.S-420 dbid 1420112693 branch 1230214881
rfs (PID:28557): Opened log for T-2.S-416 dbid 1420112693 branch 1230214881
rfs (PID:28925): Opened log for T-1.S-421 dbid 1420112693 branch 1230214881
rfs (PID:28557): Opened log for T-2.S-417 dbid 1420112693 branch 1230214881
rfs (PID:181703): Opened log for T-2.S-418 dbid 1420112693 branch 1230214881
rfs (PID:181711): Opened log for T-2.S-417 dbid 1420112693 branch 1230214881
rfs (PID:181703): Opened log for T-2.S-419 dbid 1420112693 branch 1230214881
Checking the backup:
RMAN> list backup of archivelog sequence 418 thread 2;
list backup of archivelog sequence 418 thread 2;
List of Backup Sets
===================
BS Key Size Device Type Elapsed Time Completion Time
------- ---------- ----------- ------------ ---------------
1074089502 77.00M SBT_TAPE 00:00:06 20260415
BP Key: 1074089503 Status: AVAILABLE Compressed: YES Tag: TAG20260415T151516
Handle: $RSCN_164062765_RTIM_1230214881_THRD_2_SEQ_418_CTKEY_1074089124_BACKUP
Media:
List of Archived Logs in backup set 1074089502
Thrd Seq Low SCN Low Time Next SCN Next Time
---- ------- ---------- -------- ---------- ---------
2 418 45952024080929 20260415 45952024154924 20260415
RMAN> list backup of archivelog sequence 419 thread 2;
list backup of archivelog sequence 419 thread 2;
specification does not match any backup in the repository
RMAN>
That’s good.
My Piece of Data 