Disclaimer

Showing posts with label Recovery. Show all posts
Showing posts with label Recovery. Show all posts

Saturday, 11 January 2025

How to recovery PDB when PDB database is dropped in Oracle

 

How to recovery PDB when PDB database is dropped :)

[oracle@rac01 ~]$ sqlplus '/as sysdba'

SQL*Plus: Release 21.0.0.0.0 - Production on Sat Jan 11 12:20:24 2025
Version 21.13.0.0.0

Copyright (c) 1982, 2023, Oracle.  All rights reserved.


Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.23.0.0.0

SQL>
SQL>
SQL> alter pluggable database PDB1 close;

Pluggable database altered.

SQL>
SQL>
SQL> drop pluggable database PDB1 including datafiles;

Pluggable database dropped.




SQL> show pdbs;

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         4 PDB2                           READ WRITE NO



============Alert log file contents======

2025-01-11T12:20:48.610884+01:00
alter pluggable database PDB1 close
2025-01-11T12:20:48.621225+01:00
PDB1(3):Pluggable database PDB1 closing
PDB1(3):JIT: pid 92739 requesting stop
PDB1(3):Closing sequence subsystem (204729984896).
PDB1(3):Buffer Cache flush started: 3
PDB1(3):Buffer Cache flush finished: 3
Pluggable database PDB1 closed
Completed: alter pluggable database PDB1 close
2025-01-11T12:21:10.490974+01:00
drop pluggable database PDB1 including datafiles
2025-01-11T12:21:10.596417+01:00
Deleted file +DATA/CDBAM/PDB1/undo_seed01.dbf
Deleted file +DATA/CDBAM/PDB1/users01.dbf
Deleted file +DATA/CDBAM/PDB1/temp01.dbf
Deleted file +DATA/CDBAM/PDB1/sysaux01.dbf
Deleted file +DATA/CDBAM/PDB1/system01.dbf
2025-01-11T12:21:10.640983+01:00
Stopped service pdb1
Completed: drop pluggable database PDB1 including datafiles




======Execute the below script to recover PDB1 ======

Note: Never use RESTORE command as per Oracle support suggestion

run
{
set newname for datafile 20 to '+DATA/CDBAM/PDB1/system01.dbf';
set newname for datafile 21 to '+DATA/CDBAM/PDB1/sysaux01.dbf';
set newname for datafile 22 to '+DATA/CDBAM/PDB1/users01.dbf';
set newname for datafile 23 to '+DATA/CDBAM/PDB1/undo_seed01.dbf';
recover pluggable database pdb1 until time "to_date('11-JAN-2025 12:21:00','DD-MON-YYYY HH24:MI:SS')" 
auxiliary destination '/applications/oracle/d19.3.0/CDBAM_BKP/AUX/';
}


log for recovery :-
[oracle@rac01 AUX]$ rman target /

Recovery Manager: Release 19.0.0.0.0 - Production on Sat Jan 11 14:07:44 2025
Version 19.23.0.0.0

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.

connected to target database: CDBAM (DBID=1604254345)

RMAN>

RMAN> run
{
set newname for datafile 20 to '+DATA/CDBAM/PDB1/system01.dbf';
set newname for datafile 21 to '+DATA/CDBAM/PDB1/sysaux01.dbf';
set newname for datafile 22 to '+DATA/CDBAM/PDB1/users01.dbf';
set newname for datafile 23 to '+DATA/CDBAM/PDB1/undo_seed01.dbf';
2> recover pluggable database pdb1 until time "to_date('11-JAN-2025 12:21:00','DD-MON-YYYY HH24:MI:SS')" auxiliary destination '/applications/oracle/d19.3.0/CDBAM_BKP/AUX/';
}3> 4> 5> 6> 7> 8>

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

Starting recover at 11-JAN-25
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=275 device type=DISK
RMAN-05026: warning: presuming following set of tablespaces applies to specified point-in-time

List of tablespaces expected to have UNDO segments
Tablespace SYSTEM
Tablespace UNDOTBS1

Creating automatic instance, with SID='aiAi'

initialization parameters used for automatic instance:
db_name=CDBAM
db_unique_name=aiAi_pitr_pdb1_CDBAM
compatible=19.0.0
db_block_size=8192
db_files=200
diagnostic_dest=/applications/oracle/d19.3.0/oracle_base
_pdb_name_case_sensitive=false
_system_trig_enabled=FALSE
sga_target=6144M
processes=200
db_create_file_dest=/applications/oracle/d19.3.0/CDBAM_BKP/AUX/
log_archive_dest_1='location=/applications/oracle/d19.3.0/CDBAM_BKP/AUX/'
enable_pluggable_database=true
_clone_one_pdb_recovery=true
#No auxiliary parameter file used


starting up automatic instance CDBAM

Oracle instance started

Total System Global Area    6442447480 bytes

Fixed Size                     9192056 bytes
Variable Size               1124073472 bytes
Database Buffers            5301600256 bytes
Redo Buffers                   7581696 bytes
Automatic instance created

contents of Memory Script:
{
# set requested point in time
set until  time "to_date('11-JAN-2025 12:21:00','DD-MON-YYYY HH24:MI:SS')";
# restore the controlfile
restore clone controlfile;

# mount the controlfile
sql clone 'alter database mount clone database';
}
executing Memory Script

executing command: SET until clause

Starting restore at 11-JAN-25
allocated channel: ORA_AUX_DISK_1
channel ORA_AUX_DISK_1: SID=211 device type=DISK

channel ORA_AUX_DISK_1: starting datafile backup set restore
channel ORA_AUX_DISK_1: restoring control file
channel ORA_AUX_DISK_1: reading from backup piece /applications/oracle/d19.3.0/CDBAM_BKP/c-1604254345-20250111-01
channel ORA_AUX_DISK_1: piece handle=/applications/oracle/d19.3.0/CDBAM_BKP/c-1604254345-20250111-01 tag=TAG20250111T121656
channel ORA_AUX_DISK_1: restored backup piece 1
channel ORA_AUX_DISK_1: restore complete, elapsed time: 00:00:01
output file name=/applications/oracle/d19.3.0/CDBAM_BKP/AUX/CDBAM/controlfile/o1_mf_mr4v9ltv_.ctl
Finished restore at 11-JAN-25

sql statement: alter database mount clone database

contents of Memory Script:
{
# set requested point in time
set until  time "to_date('11-JAN-2025 12:21:00','DD-MON-YYYY HH24:MI:SS')";
# set destinations for recovery set and auxiliary set datafiles
set newname for clone datafile  1 to new;
set newname for clone datafile  5 to new;
set newname for clone datafile  3 to new;
set newname for clone datafile  6 to new;
set newname for datafile  20 to new;
set newname for datafile  21 to new;
set newname for datafile  22 to new;
set newname for datafile  23 to new;
# restore the tablespaces in the recovery set and the auxiliary set
restore clone datafile  1, 5, 3, 6, 20, 21, 22, 23;

switch clone datafile all;
}
executing Memory Script

executing command: SET until clause

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

Starting restore at 11-JAN-25
using channel ORA_AUX_DISK_1

channel ORA_AUX_DISK_1: starting datafile backup set restore
channel ORA_AUX_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_AUX_DISK_1: restoring datafile 00001 to /applications/oracle/d19.3.0/CDBAM_BKP/AUX/CDBAM/datafile/o1_mf_system_%u_.dbf
channel ORA_AUX_DISK_1: restoring datafile 00005 to /applications/oracle/d19.3.0/CDBAM_BKP/AUX/CDBAM/datafile/o1_mf_undotbs1_%u_.dbf
channel ORA_AUX_DISK_1: restoring datafile 00003 to /applications/oracle/d19.3.0/CDBAM_BKP/AUX/CDBAM/datafile/o1_mf_sysaux_%u_.dbf
channel ORA_AUX_DISK_1: restoring datafile 00006 to /applications/oracle/d19.3.0/CDBAM_BKP/AUX/CDBAM/datafile/o1_mf_users_%u_.dbf
channel ORA_AUX_DISK_1: reading from backup piece /applications/oracle/d19.3.0/CDBAM_BKP/0i3evgde_18_1_1
channel ORA_AUX_DISK_1: piece handle=/applications/oracle/d19.3.0/CDBAM_BKP/0i3evgde_18_1_1 tag=TAG20250111T121646
channel ORA_AUX_DISK_1: restored backup piece 1
channel ORA_AUX_DISK_1: restore complete, elapsed time: 00:00:15
channel ORA_AUX_DISK_1: starting datafile backup set restore
channel ORA_AUX_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_AUX_DISK_1: restoring datafile 00020 to +DATA/CDBAM/PDB1/system01.dbf
channel ORA_AUX_DISK_1: restoring datafile 00021 to +DATA/CDBAM/PDB1/sysaux01.dbf
channel ORA_AUX_DISK_1: restoring datafile 00022 to +DATA/CDBAM/PDB1/users01.dbf
channel ORA_AUX_DISK_1: restoring datafile 00023 to +DATA/CDBAM/PDB1/undo_seed01.dbf
channel ORA_AUX_DISK_1: reading from backup piece /applications/oracle/d19.3.0/CDBAM_BKP/0k3evgdm_20_1_1
channel ORA_AUX_DISK_1: piece handle=/applications/oracle/d19.3.0/CDBAM_BKP/0k3evgdm_20_1_1 tag=TAG20250111T121646
channel ORA_AUX_DISK_1: restored backup piece 1
channel ORA_AUX_DISK_1: restore complete, elapsed time: 00:00:03
Finished restore at 11-JAN-25

datafile 1 switched to datafile copy
input datafile copy RECID=29 STAMP=1190124510 file name=/applications/oracle/d19.3.0/CDBAM_BKP/AUX/CDBAM/datafile/o1_mf_system_mr4v9vv6_.dbf
datafile 5 switched to datafile copy
input datafile copy RECID=30 STAMP=1190124510 file name=/applications/oracle/d19.3.0/CDBAM_BKP/AUX/CDBAM/datafile/o1_mf_undotbs1_mr4v9vvc_.dbf
datafile 3 switched to datafile copy
input datafile copy RECID=31 STAMP=1190124510 file name=/applications/oracle/d19.3.0/CDBAM_BKP/AUX/CDBAM/datafile/o1_mf_sysaux_mr4v9vv9_.dbf
datafile 6 switched to datafile copy
input datafile copy RECID=32 STAMP=1190124510 file name=/applications/oracle/d19.3.0/CDBAM_BKP/AUX/CDBAM/datafile/o1_mf_users_mr4v9vvg_.dbf

contents of Memory Script:
{
# set requested point in time
set until  time "to_date('11-JAN-2025 12:21:00','DD-MON-YYYY HH24:MI:SS')";
# online the datafiles restored or switched
sql clone "alter database datafile  1 online";
sql clone "alter database datafile  5 online";
sql clone "alter database datafile  3 online";
sql clone "alter database datafile  6 online";
sql clone 'PDB1' "alter database datafile
 20 online";
sql clone 'PDB1' "alter database datafile
 21 online";
sql clone 'PDB1' "alter database datafile
 22 online";
sql clone 'PDB1' "alter database datafile
 23 online";
#recover pdb
recover clone database tablespace  "SYSTEM", "UNDOTBS1", "SYSAUX", "USERS" pluggable database
 'PDB1'  delete archivelog;
#open in read write mode
sql clone 'alter database open resetlogs';
#unplug dropped pdb into temp file
sql clone "alter pluggable database PDB1 unplug into ''
/applications/oracle/d19.3.0/oracle_home/dbs/_rm_pdb_pitr_9_aiAi.xml''";
#create pdb using temp file of recovered pdb
sql "create pluggable database PDB1 using ''
/applications/oracle/d19.3.0/oracle_home/dbs/_rm_pdb_pitr_9_aiAi.xml'' nocopy tempfile reuse";
alter pluggable database PDB1 open;
}
executing Memory Script

executing command: SET until clause

sql statement: alter database datafile  1 online

sql statement: alter database datafile  5 online

sql statement: alter database datafile  3 online

sql statement: alter database datafile  6 online

sql statement: alter database datafile  20 online

sql statement: alter database datafile  21 online

sql statement: alter database datafile  22 online

sql statement: alter database datafile  23 online

Starting recover at 11-JAN-25
using channel ORA_AUX_DISK_1

Executing: alter database datafile 2, 4, 7, 15 offline
Executing: alter database datafile 16, 17, 18, 19 offline
starting media recovery

archived log for thread 1 with sequence 269 is already on disk as file +ARCH/cdbam/arch/1_269_1189847369.dbf
archived log for thread 1 with sequence 270 is already on disk as file +ARCH/cdbam/arch/1_270_1189847369.dbf
archived log file name=+ARCH/cdbam/arch/1_269_1189847369.dbf thread=1 sequence=269
archived log file name=+ARCH/cdbam/arch/1_270_1189847369.dbf thread=1 sequence=270
media recovery complete, elapsed time: 00:00:01
Finished recover at 11-JAN-25

sql statement: alter database open resetlogs

sql statement: alter pluggable database PDB1 unplug into ''/applications/oracle/d19.3.0/oracle_home/dbs/_rm_pdb_pitr_9_aiAi.xml''

sql statement: create pluggable database PDB1 using ''/applications/oracle/d19.3.0/oracle_home/dbs/_rm_pdb_pitr_9_aiAi.xml'' nocopy tempfile reuse

Statement processed

Removing automatic instance
shutting down automatic instance
Oracle instance shut down
Automatic instance removed
auxiliary instance file /applications/oracle/d19.3.0/oracle_home/dbs/_rm_pdb_pitr_9_aiAi.xml deleted
auxiliary instance file /applications/oracle/d19.3.0/CDBAM_BKP/AUX/CDBAM/datafile/o1_mf_sysaux_mr4v9vv9_.dbf deleted
auxiliary instance file /applications/oracle/d19.3.0/CDBAM_BKP/AUX/CDBAM/controlfile/o1_mf_mr4v9ltv_.ctl deleted
Finished recover at 11-JAN-25

RMAN>


PDB1 recovered successfully.
[oracle@rac01 ~]$ sqlplus '/as sysdba'

SQL*Plus: Release 21.0.0.0.0 - Production on Sat Jan 11 13:00:24 2025
Version 21.13.0.0.0

Copyright (c) 1982, 2023, Oracle.  All rights reserved.


Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.23.0.0.0


SQL> show pdbs;

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
	 3 PDB1                           READ WRITE NO
         4 PDB2                           READ WRITE NO










Tuesday, 17 December 2024

Solving ORA-00742 Log read detects lost write in Oracle

 


[oracle@rac201 ~]$ sqlplus '/as sysdba'

SQL*Plus: Release 19.0.0.0.0 - Production on Tue Dec 17 10:47:58 2024
Version 19.21.0.0.0

Copyright (c) 1982, 2022, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup;
ORACLE instance started.

Total System Global Area 2415917856 bytes
Fixed Size                  8928032 bytes
Variable Size            1006632960 bytes
Database Buffers         1392508928 bytes
Redo Buffers                7847936 bytes
Database mounted.
ORA-00742: Log read detects lost write in thread 1 sequence 170 block 72959
ORA-00312: online log 1 thread 1: '/data1/ORCL/redo01.log'



ORA-00742: Log read detects lost write in thread 1 sequence 170 block 72959




In this post we will see how to solve ORA-00742: Log read detects lost write-in thread 1 sequence, You can encounter this error if your Oracle Database machine was suddenly stopped without proper shutdown command due to some unavoidable problem like Power Failure, Power cable issue, etc.


STEP1: In this step, tried to login to the database, and below is an error.

SQL> startup;
ORACLE instance started.

Total System Global Area 2415917856 bytes
Fixed Size                  8928032 bytes
Variable Size            1006632960 bytes
Database Buffers         1392508928 bytes
Redo Buffers                7847936 bytes
Database mounted.
ORA-00742: Log read detects lost write in thread 1 sequence 170 block 72959
ORA-00312: online log 1 thread 1: '/data1/ORCL/redo01.log'



STEP2: Check the status of log file 1 in v$log & v$logfile and you can notice log group 1 is the current logfile.
set lines 300
set pages 3000
col MEMBER for a60
select * from v$Logfile;SQL> SQL> SQL>

    GROUP# STATUS  TYPE    MEMBER                                                       IS_     CON_ID
---------- ------- ------- ------------------------------------------------------------ --- ----------
         3         ONLINE  /data1/ORCL/redo03.log                                       NO           0
         2         ONLINE  /data1/ORCL/redo02.log                                       NO           0
         1         ONLINE  /data1/ORCL/redo01.log                                       NO           0
        11         STANDBY /data1/ORCL/stb_redo11.log                                   NO           0
        12         STANDBY /data1/ORCL/stb_redo12.log                                   NO           0
        13         STANDBY /data1/ORCL/stb_redo13.log                                   NO           0
        14         STANDBY /data1/ORCL/stb_redo14.log                                   NO           0

7 rows selected.

SQL> select * from v$log;

    GROUP#    THREAD#  SEQUENCE#      BYTES  BLOCKSIZE    MEMBERS ARC STATUS           FIRST_CHANGE# FIRST_TIM NEXT_CHANGE# NEXT_TIME     CON_ID
---------- ---------- ---------- ---------- ---------- ---------- --- ---------------- ------------- --------- ------------ --------- ----------
         1          1        170  209715200        512          1 NO  CURRENT                4974163 17-DEC-24   9.2954E+18                    0
         3          1        169  209715200        512          1 YES INACTIVE               4939898 14-DEC-24      4974163 17-DEC-24          0
         2          1        168  209715200        512          1 YES INACTIVE               4859292 08-DEC-24      4939898 14-DEC-24          0




STEP3: Try to clear the current log file most likely it will fail as group log 1 is the current logfile.
SQL> alter database clear unarchived logfile group 1;
alter database clear unarchived logfile group 1
*
ERROR at line 1:
ORA-01624: log 1 needed for crash recovery of instance orcl (thread 1)
ORA-00312: online log 1 thread 1: '/data1/ORCL/redo01.log'




STEP4: So we will try to perform manual recovery on the database, Please note since Group log 1 is current it will be not archived, and the requested archive log sequence 170 is the current redo logfile, and hence I have provided manually redo log 1 file location.

 But even after “Media recovery complete” While trying to open DB, I encounter an ORA-00600 error.


SQL> recover database until cancel;
ORA-00279: change 4974164 generated at 12/17/2024 10:08:15 needed for thread 1
ORA-00289: suggestion : /data1/ORCL/ARCH/1_170_1178543939.arc
ORA-00280: change 4974164 for thread 1 is in sequence #170


Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
/data1/ORCL/redo01.log
Log applied.
Media recovery complete.
SQL>
SQL>
SQL>
SQL>
SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-01589: must use RESETLOGS or NORESETLOGS option for database open


SQL>
SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-00600: internal error code, arguments: [krsi_al_hdr_update.invalid_nab_1], [4294967295], [], [], [], [], [], [], [], [], [], []



At this point, I tried to search in MOS and fortunately found below MOS note for the above issue


Alter Database Open Resetlogs returns error ORA-00600: [krsi_al_hdr_update.15], (Doc ID 2026541.1)


STEP 5: We followed the steps mention in MOS Note and again attempted fake recovery with the CANCEL option and was able to open the database. 

At the OS level move the current online redologs:



SQL> !mv /data1/ORCL/redo01.log /data1/ORCL/redo01_old.log

SQL>
SQL> recover database using backup controlfile until cancel;
ORA-00279: change 4977857 generated at 12/17/2024 10:23:53 needed for thread 1
ORA-00289: suggestion : /data1/ORCL/ARCH/1_170_1178543939.arc
ORA-00280: change 4977857 for thread 1 is in sequence #170


Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
CANCEL
Media recovery cancelled.
SQL>
SQL>
SQL>
SQL>
SQL> alter database open resetlogs;

Database altered.




SQL> select status from v$instance;

STATUS
------------
OPEN

SQL>
SQL>  set lines 300
set pages 3000
col MEMBER for a60
select * from v$Logfile;SQL> SQL> SQL>

    GROUP# STATUS  TYPE    MEMBER                                                       IS_     CON_ID
---------- ------- ------- ------------------------------------------------------------ --- ----------
         3         ONLINE  /data1/ORCL/redo03.log                                       NO           0
         2         ONLINE  /data1/ORCL/redo02.log                                       NO           0
         1         ONLINE  /data1/ORCL/redo01.log                                       NO           0
        11         STANDBY /data1/ORCL/stb_redo11.log                                   NO           0
        12         STANDBY /data1/ORCL/stb_redo12.log                                   NO           0
        13         STANDBY /data1/ORCL/stb_redo13.log                                   NO           0
        14         STANDBY /data1/ORCL/stb_redo14.log                                   NO           0

7 rows selected.







Sunday, 8 December 2024

Recover a corrupted Block - 19c

 

Recover a corrupted Block (Oracle 19c)

The steps to check for a corrupted blocks in the whole database as health monitor check and how to recover the corrupted block.


Steps: Recover the corrupted blocks

Step 1: Health Monitor the database:

RMAN> validate database;
 Starting validate at 11-MAY-24
 using target database control file instead of recovery catalog
 allocated channel: ORA_DISK_1
 channel ORA_DISK_1: SID=424 device type=DISK
 channel ORA_DISK_1: starting validation of datafile
 channel ORA_DISK_1: specifying datafile(s) for validation
 input datafile file number=00001 name=/data/ORCL/system01.dbf
 input datafile file number=00004 name=/data/ORCL/undotbs01.dbf
 input datafile file number=00003 name=/data/ORCL/sysaux01.dbf
 input datafile file number=00002 name=/data/ORCL/asyw_tbs01.dbf
 input datafile file number=00005 name=/data/ORCL/users02.dbf
 input datafile file number=00008 name=/data/ORCL/ORCL/datafile/o1_mf_wado_tbs_l5skfdoz_.dbf
 input datafile file number=00009 name=/data/ORCL/ORCL/datafile/o1_mf_wado_tbs_l5sldkgh_.dbf
 input datafile file number=00007 name=/data/ORCL/users01.dbf
 channel ORA_DISK_1: validation complete, elapsed time: 00:00:03
 List of Datafiles
 …..
 File Status Marked Corrupt Empty Blocks Blocks Examined High SCN
 
 3    FAILED 0              20612        84607           3571828
   File Name: /data/ORCL/sysaux01.dbf
   Block Type Blocks Failing Blocks Processed
 
   Data       0              9536
   Index      0              6038
   Other      2              48294
 ….
 validate found one or more corrupt blocks
 See trace file /u01/app/oracle/diag/rdbms/ORCL/ORCL/trace/ORCL_ora_2707.trc for details
 channel ORA_DISK_1: starting validation of datafile
 channel ORA_DISK_1: specifying datafile(s) for validation
 including current control file for validation
 Finished validate at 11-MAY-24


Note: We have two corrupted blocks with the tablespace SYSAUX.


Step 2: Check the alert file

Corrupt block relative dba: 0x00c0f945 (file 3, block 63813)
 Bad check value found during validation
 Data in bad block:
  type: 0 format: 2 rdba: 0x00c0f945
  last change scn: 0x0000.0000.00000000 seq: 0x1 flg: 0x05
  spare3: 0x0
  consistency value in tail: 0x00000001
  check value in block header: 0x5e85
  computed block checksum: 0xa00
 Reread of blocknum=63813, file=/data/ORCL/sysaux01.dbf. found same corrupt data
 Reread of blocknum=63813, file=/data/ORCL/sysaux01.dbf. found same corrupt data
 Reread of blocknum=63813, file=/data/ORCL/sysaux01.dbf. found same corrupt data
 Reread of blocknum=63813, file=/data/ORCL/sysaux01.dbf. found same corrupt data
 Reread of blocknum=63813, file=/data/ORCL/sysaux01.dbf. found same corrupt data
 Hex dump of (file 3, block 63814) in trace file /u01/app/oracle/diag/rdbms/ORCL/ORCL/trace/ORCL_ora_2707.trc
 Corrupt block relative dba: 0x00c0f946 (file 3, block 63814)
 Bad check value found during validation
 Data in bad block:
  type: 0 format: 2 rdba: 0x00c0f946
  last change scn: 0x0000.0000.00000000 seq: 0x1 flg: 0x05
  spare3: 0x0
  consistency value in tail: 0x00000001
  check value in block header: 0x5e86
  computed block checksum: 0xa00
 Reread of blocknum=63814, file=/data/ORCL/sysaux01.dbf. found same corrupt data
 Reread of blocknum=63814, file=/data/ORCL/sysaux01.dbf. found same corrupt data
 Reread of blocknum=63814, file=/data/ORCL/sysaux01.dbf. found same corrupt data
 Reread of blocknum=63814, file=/data/ORCL/sysaux01.dbf. found same corrupt data
 Reread of blocknum=63814, file=/data/ORCL/sysaux01.dbf. found same corrupt data


Step 3: Recover the corrupted blocks

RMAN> list failure;

 Database Role: PRIMARY
 List of Database Failures
 Failure ID Priority Status    Time Detected Summary
 
 801        HIGH     OPEN      11-MAY-24     Datafile 3: '/data/ORCL/sysaux01.dbf' contains one or more corrupt blocks

 RMAN> advise failure;
 
 Database Role: PRIMARY
 List of Database Failures
 Failure ID Priority Status    Time Detected Summary
 
 801        HIGH     OPEN      11-MAY-24     Datafile 3: '/data/ORCL/sysaux01.dbf' contains one or more corrupt blocks
 analyzing automatic repair options; this may take some time
 using channel ORA_DISK_1
 analyzing automatic repair options complete
 Mandatory Manual Actions
 no manual actions available
 Optional Manual Actions
 Shut down, mount the database and try flush redo using ALTER SYSTEM FLUSH REDO TO 'standby name' command.  Then perform a Data Guard role change (failover).  Available standbys: ORCLsb. 
 Automated Repair Options
 Option Repair Description
 
 1      Recover multiple corrupt blocks in datafile 3
   Strategy: The repair includes complete media recovery with no data loss
   Repair script: /u01/app/oracle/diag/rdbms/ORCL/ORCL/hm/reco_3225659642.hm

 
 RMAN> repair failure;
 
 Strategy: The repair includes complete media recovery with no data loss
 Repair script: /u01/app/oracle/diag/rdbms/ORCL/ORCL/hm/reco_3225659642.hm
 contents of repair script:
    # block media recovery for multiple blocks
    recover datafile 3 block 63813 to 63814;
 Do you really want to execute the above repair (enter YES or NO)? yes
 executing repair script
 Starting recover at 11-MAY-24
 using channel ORA_DISK_1
 finished standby search, restored 2 blocks
 starting media recovery
 media recovery complete, elapsed time: 00:00:03
 Finished recover at 11-MAY-24
 repair failure complete



Tuesday, 3 December 2024

How to restore or recreate the OLR if you don't have backup

 


The steps you shared are typically used for deconfiguring and reconfiguring Oracle Clusterware in an Oracle Real Application Clusters (RAC) environment, especially for recreating the Oracle Local Registry (OLR) file. 


Here's an explanation of the commands:

  1. Stop Clusterware

    # crsctl stop crs -f
    • This command stops the Oracle Clusterware stack on the local node forcefully.
    • The -f option ensures that all services are stopped even if there are dependencies.
  2. Deconfigure Clusterware

    # $GRID_HOME/crs/install/rootcrs.pl -deconfig -force
    • This Perl script is used to deconfigure Oracle Clusterware.
    • The -deconfig flag removes the cluster configuration from the node.
    • The -force flag ensures that the operation proceeds even if warnings are encountered.
  3. Recreate OLR

    # $GRID_HOME/root.sh
    • The root.sh script is executed to recreate the OLR and reinitialize the node with the Oracle Clusterware configuration.


Important Considerations

  • Always ensure you have a valid backup of the OLR and any other critical Clusterware files before proceeding with these steps.
  • Use these steps only under the guidance of Oracle Support or when you are certain about the procedure, as they involve low-level configuration changes.
  • If the RAC setup involves multiple nodes, ensure proper coordination across all nodes for any deconfiguration or reconfiguration.



How to restore OLR from Backup in Oracle RAC

 

Backup current OLR and all available OLR backups

How to take Manual backup of OLR?

# ocrconfig -local -manualbackup
rac01.samik.com     2024/11/30 11:23:36     /applications/oracle/g19.3.0/grid_base/crsdata/rac01.samik.com/olr/backup_20241201_122220.olr 2204791787


How to find the OLR auto backup?

[root@rac01.samik.com ~]# ocrconfig -local -showbackup auto

rac01.samik.com     2024/12/01 11:23:36     /applications/oracle/g19.3.0/grid_base/crsdata/rac01.samik.com/olr/autobackup_20241201_112336.olr     2204791795
rac01.samik.com     2024/11/30 11:23:36     /applications/oracle/g19.3.0/grid_base/crsdata/rac01.samik.com/olr/autobackup_20241130_112336.olr     2204791795



Verify available backups

# ocrconfig -local -showbackup

rac01.samik.com     2024/12/01 11:23:36     /applications/oracle/g19.3.0/grid_base/crsdata/rac01.samik.com/olr/autobackup_20241201_112336.olr     2204791795
rac01.samik.com     2024/11/30 11:23:36     /applications/oracle/g19.3.0/grid_base/crsdata/rac01.samik.com/olr/autobackup_20241130_112336.olr     2204791795
rac01.samik.com     2024/11/30 11:23:36     /applications/oracle/g19.3.0/grid_base/crsdata/rac01.samik.com/olr/backup_20241201_122220.olr 2204791787




Restore a  OLR from backup


Shutdown CRS and check that ohasd.bin process is not running and restore OLR

Step1:- Stop the cluster on the specific node
# crsctl stop crs


Step2:- Run the below command to restore OLR
# ocrconfig -local -restore  /applications/oracle/g19.3.0/grid_base/crsdata/rac01.samik.com/olr/backup_20241201_122220.olr


Step3:- Start the Cluster
# crsctl start crs





How to Restore ASM Based OCR After Complete Loss of the CRS Diskgroup on Linux/Unix Systems (Doc ID 1062983.1)


It is not possible to directly restore a manual or automatic OCR backup if the OCR is located in an ASM disk group. 

This is caused by the fact that the command 'ocrconfig -restore' requires ASM to be up & running in order to restore an OCR backup to an ASM disk group. However, for ASM to be available, the CRS stack must have been successfully started. 

For the restore to succeed, the OCR also must not be in use (r/w), i.e. no CRS daemon must be running while the OCR is being restored.

A description of the general procedure to restore the OCR can be found in the  documentation, this document explains how to recover from a complete loss of the ASM disk group that held the OCR and Voting files in a 11gR2 Grid environment.

Note:- From 12.2 onwards, OCR backups are located in ASM diskgroup. Refer Doc ID 2569847.1 having specific restoration steps involved when the OCR backup is located in ASM disk group.



When using an ASM disk group for CRS there are typically 3 different types of files located in the disk group that potentially need to be restored/recreated:

1) the Oracle Cluster Registry file (OCR)
2) the Voting file(s)
3) the shared SPFILE for the ASM instances


The following example assumes that the OCR was located in a single disk group used exclusively for CRS. The disk group has just one disk using external redundancy.

Since the CRS disk group has been lost the CRS stack will not be available on any node.

The following settings used in the example would need to be replaced according to the actual configuration:

GRID user:                       oragrid
GRID home:                       /u01/app/11.2.0/grid ($CRS_HOME)
ASM disk group name for OCR:     CRS
ASM/ASMLIB disk name:            ASMD40
Linux device name for ASM disk:  /dev/sdh1
Cluster name:                    rac_cluster1
Nodes:                           racnode1, racnode2



1. Locate the latest automatic OCR backup

When using a non-shared CRS home, automatic OCR backups can be located on any node of the cluster


[root@rac01.samik.com ~]# ocrconfig -showbackup

rac01.samik.com     2024/12/02 07:22:31     +OCRVD:/POCDB-SCAN-Cluster/OCRBACKUP/backup00.ocr.271.1186644149     2204791795
rac01.samik.com     2024/12/02 03:22:29     +OCRVD:/POCDB-SCAN-Cluster/OCRBACKUP/backup01.ocr.270.1186629747     2204791795
rac01.samik.com     2024/12/01 23:22:26     +OCRVD:/POCDB-SCAN-Cluster/OCRBACKUP/backup02.ocr.269.1186615345     2204791795
rac01.samik.com     2024/11/30 15:21:17     +OCRVD:/POCDB-SCAN-Cluster/OCRBACKUP/day.ocr.266.1186413679     2204791795
rac01.samik.com     2024/11/22 19:17:22     +OCRVD:/POCDB-SCAN-Cluster/OCRBACKUP/week.ocr.268.1185736643     220479179



2. Make sure the Grid Infrastructure is shutdown on all nodes

Given that the OCR diskgroup is missing, the GI stack will not be functional on any node, however there may still be various daemon processes running.  On each node shutdown the GI stack using the force (-f) option:

# $CRS_HOME/bin/crsctl stop crs -f



3. Start the CRS stack in exclusive mode

On the node that has the most recent OCR backup, log on as root and start CRS in exclusive mode, this mode will allow ASM to start & stay up without the presence of a Voting disk and without the CRS daemon process (crsd.bin) running.

11.2.0.2 and above:

# $CRS_HOME/bin/crsctl start crs -excl -nocrs
CRS-4123: Oracle High Availability Services has been started.


4. Label the CRS disk for ASMLIB use

If using ASMLIB the disk to be used for the CRS disk group needs to stamped first, as user root do:
# /usr/sbin/oracleasm createdisk ASMD40 /dev/sdh1
Writing disk header: done
Instantiating disk: done



5. Create the CRS diskgroup via sqlplus

The disk group can now be (re-)created via sqlplus from the grid user. The compatible.asm attribute must be set to 11.2 in order for the disk group to be used by CRS:

$ sqlplus / as sysasm
SQL*Plus: Release 11.2.0.1.0 Production on Tue Mar 30 11:47:24 2010
Copyright (c) 1982, 2009, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - Production
With the Real Application Clusters and Automatic Storage Management options

SQL> create diskgroup CRS external redundancy disk 'ORCL:ASMD40' attribute 'COMPATIBLE.ASM' = '11.2';

Diskgroup created.


6. Restore the latest OCR backup

Now that the CRS disk group is created and mounted the OCR can be restored - must be done as the root user:


Note:- Our backup is on ASM Disk group
# $CRS_HOME/bin/ocrconfig -restore backup00.ocr


7. Start the CRS daemon on the current node (11.2.0.1 only !)

Now that the OCR has been restored the CRS daemon can be started, this is needed to recreate the Voting file. Skip this step for 11.2.0.2.0.
# $CRS_HOME/bin/crsctl start res ora.crsd -init
CRS-2672: Attempting to start 'ora.crsd' on 'racnode1'
CRS-2676: Start of 'ora.crsd' on 'racnode1' succeeded



8. Recreate the Voting file

The Voting file needs to be initialized in the CRS disk group:

# $CRS_HOME/bin/crsctl replace votedisk +CRS
Successful addition of voting disk 00caa5b9c0f54f3abf5bd2a2609f09a9.
Successfully replaced voting disk group with +CRS.
CRS-4266: Voting file(s) successfully replaced


9. Recreate the SPFILE for ASM (optional)

Please note:
Starting with 11gR2 ASM can start without a PFILE or SPFILE, so if you are
- not using an  SPFILE for ASM
- not using a shared SPFILE for ASM
- using a shared SPFILE not stored in ASM (e.g. on cluster file system)
this step possibly should be skipped.

Also use extra care in regards to the asm_diskstring parameter as it impacts the discovery of the voting disks.

Please verify the previous settings using the ASM alert log.

Prepare a pfile (e.g. /tmp/asm_pfile.ora) with the ASM startup parameters - these may vary from the example below. If in doubt consult the ASM alert log  as the ASM instance startup should list all non-default parameter values. Please note the last startup of ASM (in step 2 via CRS start) will not have used an SPFILE, so a startup prior to the loss of the CRS disk group would need to be located.

*.asm_power_limit=1
*.diagnostic_dest='/u01/app/oragrid'
*.instance_type='asm'
*.large_pool_size=12M
*.remote_login_passwordfile='EXCLUSIVE'
Now the SPFILE can be created using this PFILE:

$ sqlplus / as sysasm
SQL*Plus: Release 11.2.0.1.0 Production on Tue Mar 30 11:52:39 2010
Copyright (c) 1982, 2009, Oracle. All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - Production
With the Real Application Clusters and Automatic Storage Management options

SQL> create spfile='+CRS' from pfile='/tmp/asm_pfile.ora';

File created.


10. Shutdown CRS

Since CRS is running in exclusive mode, it needs to be shutdown  to allow CRS to run on all nodes again. Use of the force (-f) option may be required:
# $CRS_HOME/bin/crsctl stop crs -f
...
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'auw2k3' has completed
CRS-4133: Oracle High Availability Services has been stopped.




11. Rescan ASM disks

If using ASMLIB rescan all ASM disks on each node as the root user:
# oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
Instantiating disk "ASMD40"


12. Start CRS

As the root user submit the CRS startup on all cluster nodes:
# $CRS_HOME/bin/crsctl start crs
CRS-4123: Oracle High Availability Services has been started.



13. Verify CRS

To verify that CRS is fully functional again:
# $CRS_HOME/bin/crsctl check cluster -all
**************************************************************
racnode1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
racnode2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************

# $CRS_HOME/bin/crsctl status resource -t













Friday, 29 November 2024

CURRENT redo log file delated at OS level - only one member in the group - Scenario Recap

 

Scenario Recap

  • You have 5 redo log groups.
  • Each group has only one member (no multiplexing).
  • The OS team accidentally deleted the "CURRENT" redo log file.

The redo log file labeled "CURRENT" is actively being written to by the database. It contains ongoing and uncommitted transactions. If it's deleted, the database will encounter issues during the next log switch or checkpoint.




1. What recovery do I need to perform?

You must recover the database because the "CURRENT" redo log file contains crucial data that hasn't been written to the data files. Since it is deleted, Oracle cannot proceed with normal operations.

You may need to:

  • Shut down the database.
  • Restore the missing redo log file or reset the redo logs.
  • Perform recovery to ensure no data corruption.

What recovery do I need to do?

When the "CURRENT" redo log file is deleted, Oracle will encounter a critical error when it tries to write to the missing file. The database will likely crash or hang. To recover:

  1. You need to stop the database (SHUTDOWN IMMEDIATE).
  2. Bring it to MOUNT mode (STARTUP MOUNT).
  3. Either clear the missing redo log (CLEAR UNARCHIVED LOGFILE) or restore it from a backup.



2. How will I lose transactions?

Let’s use an example:

Practical Setup:

  1. Create a table and insert some data:

    SQL> CREATE TABLE emp (id NUMBER, name VARCHAR2(50));
    SQL> INSERT INTO emp VALUES (1, 'John'); SQL> INSERT INTO emp VALUES (2, 'Jane'); SQL> COMMIT; -- These transactions are committed.
  2. Insert new data but don’t commit it:

    SQL> INSERT INTO emp VALUES (3, 'Alice'); -- Uncommitted transaction
    SQL> INSERT INTO emp VALUES (4, 'Bob'); -- Uncommitted transaction
  3. At this point, the "CURRENT" redo log contains:

    • Committed transactions (John and Jane).
    • Uncommitted transactions (Alice and Bob).

Impact:

  • If the "CURRENT" redo log is deleted:
    • Committed transactions are safe. They are already written to the data files.
    • Uncommitted transactions are lost. They were in the redo log and not yet written to the data files.

2. How will I lose my transactions?

You will lose:

  • Uncommitted transactions in the "CURRENT" redo log, as they were only in memory and the deleted log file.
  • Committed transactions in the "CURRENT" log if the log was not archived yet.

Practical Example:

  1. Create a table and commit a transaction:

    CREATE TABLE test_recovery (id NUMBER, name VARCHAR2(50));
    INSERT INTO test_recovery VALUES (1, 'Committed'); COMMIT;

    This transaction is safe, as it’s committed and likely written to previous redo logs.

  2. Insert another record without committing:

    INSERT INTO test_recovery VALUES (2, 'Uncommitted');

    If the redo log holding this change is deleted, this transaction is lost forever.

  3. Insert a committed transaction after the uncommitted one:

    INSERT INTO test_recovery VALUES (3, 'Committed After Loss');
    COMMIT;

    If the redo log containing this transaction was not archived, it will also be lost.




3. Do I really need to perform point-in-time recovery?

No, you generally don’t need to perform a point-in-time recovery unless:

  • There is corruption or loss of critical data.
  • The redo logs are unrecoverable.

You can usually reset the logs or perform standard recovery.


Do I need to perform point-in-time recovery?

Point-in-time recovery (PITR) is only required if:

  • You cannot clear the missing log (e.g., it contains unarchived transactions crucial to the database).

If the missing log is cleared:

  • PITR is unnecessary, but uncommitted and unarchived changes are lost.




4. Do I need to stop the database and start in mount stage?

Yes. You need to:

  1. Shut down the database:
    SQL> SHUTDOWN IMMEDIATE;
  2. Start it in mount mode:
    SQL> STARTUP MOUNT;

This is necessary because redo log management happens at the mount stage.


Do I need to stop the database and start in MOUNT stage?

  • Yes, for a "CURRENT" log, the database must be in MOUNT mode to clear or reset the redo log. This is because the log is actively used, and clearing it in an open database would disrupt transactions.



5. What does "CLEAR UNARCHIVED LOGFILE" mean?

This command clears the damaged redo log files, making them usable again, but data in the log may be lost.

Example:

  • If you clear an unarchived redo log file, any uncommitted transactions in that file will be lost.


What does "CLEAR UNARCHIVED LOGFILE" mean?

This command removes references to the deleted log, allowing Oracle to reuse the log group. However:

  • Unarchived transactions in the log are lost forever.



6. Do I need to "CLEAR UNARCHIVED LOGFILE" when the database is running or stopped?

You must clear the log file when the database is mounted but not open.


 Do I need to "CLEAR UNARCHIVED LOGFILE" while DB is running or after shutdown?

  • You must shutdown the database and bring it to MOUNT mode to execute this command.



7. What is the impact of "CLEAR UNARCHIVED LOGFILE"?

Example Impact:

  1. If you clear an unarchived redo log file:

    SQL> ALTER DATABASE CLEAR UNARCHIVED LOGFILE GROUP 1;
  2. Impact:
    • Committed transactions in other log files are safe.
    • Uncommitted transactions in the cleared redo log are lost.

Layman Explanation:

Think of the redo log as a diary. If you accidentally tear out a page before archiving it, you lose everything written on that page.


Impact of "CLEAR UNARCHIVED LOGFILE"

  • Impact: Any unarchived transactions in the missing log are lost.
  • Example:
    1. Missing redo log contains:
      • Committed transaction: INSERT INTO test_recovery VALUES (3, 'Committed After Loss'); COMMIT;
      • Uncommitted transaction: INSERT INTO test_recovery VALUES (2, 'Uncommitted');
    2. After clearing the log, both transactions are gone.




8. Do I need to restore the 10TB database?

No, you do not need to restore the entire 10TB database unless:

  • The database has corruption.
  • You lose all redo log files.

Strategy:

  • Fix the redo logs (e.g., recreate or clear them).
  • Perform recovery to ensure database consistency.

Do I need to restore the entire 10TB database?

  • No, you don’t need to restore the entire database unless the redo log loss has caused unrecoverable corruption.

Strategy:

  • If you have a backup of the redo logs or multiplexed copies, restore only the missing log.
  • If not, clear the log and accept the loss of unarchived changes.




9. Layman Explanation (Step-by-Step)

Imagine your redo log files are like flight recorders for a plane. If the recorder for a current flight (CURRENT redo log) is destroyed:

  1. You lose information about what happened on that flight (uncommitted transactions).
  2. Other flights (committed transactions) are safe because their information was already saved elsewhere (data files).

When a redo log is deleted:

  • Uncommitted transactions in that log are lost.
  • You must fix or reset the logs to let the database operate again.



10. Simplified Recovery Steps

  1. Check which redo log file is missing:

    SQL> SELECT GROUP#, STATUS, MEMBER FROM V$LOGFILE;
  2. Shut down the database:

    SQL> SHUTDOWN IMMEDIATE;
  3. Start in mount mode:

    SQL> STARTUP MOUNT;
  4. Clear the missing redo log file:

    SQL> ALTER DATABASE CLEAR UNARCHIVED LOGFILE GROUP 1;
  5. Open the database:

    SQL> ALTER DATABASE OPEN;
  6. If needed, recreate the redo log group:

    SQL> ALTER DATABASE ADD LOGFILE GROUP 1 '/path/to/new_redo01.log' SIZE 200M;



Conclusion

The key to avoiding such issues is:

  1. Always multiplex redo logs (at least 2 members per group).
  2. Ensure regular backups with archiving enabled.
  3. Understand the difference between committed and uncommitted transactions.

 Summary of Steps

If "CURRENT" Log is Deleted:

  1. Shutdown Database:
    SHUTDOWN IMMEDIATE;
  2. Start in MOUNT Mode:
    STARTUP MOUNT;
  3. Clear Missing Log:
    ALTER DATABASE CLEAR UNARCHIVED LOGFILE GROUP <group_number>;
  4. Open Database:
    ALTER DATABASE OPEN;
  5. Monitor Data Loss:
    • Check for missing transactions in application logs or DBA_DATA_FILES.




Eternal Lesson

  1. Multiplex redo logs to prevent data loss.
  2. Backup regularly to restore lost files without clearing.
  3. Use CLEAR UNARCHIVED LOGFILE cautiously, as it guarantees some data loss.




Undo Queries

  SQL> SQL> select sum(bytes /(1024*1024*1024)) from dba_undo_extents where status='EXPIRED'; select sum(bytes /(1024*1024*102...