Oracle ASM Corruption: How To Fix The ASM Disk HEADER_STATUS From FORMER To MEMBER

 

ASM Corruption: Case #1: How To Fix The ASM Disk HEADER_STATUS From FORMER or PROVISIONED To MEMBER.

 

This document explains, in detail, the steps required (with an example) to fix the ASM disk header from FORMER to MEMBER under the next scenarios:

 

 

A) Diskgroup was dropped by accident using the 

“SQL> drop <DG name> diskgroup;” statement.

 

B) Or under strange situations as described in the next bug:

 

Bug.13331814 ASM DISKS TURNED INTO FORMER WHILE DISKGROUP IS MOUNTED.

 

 

Example:

 

1) You created a diskgroup with the next disk member:

 

 

[grid@asmlnx1 ~]$ sqlplus "/as sysasm"

 

SQL*Plus: Release 12.1.0.1.0 Production on Sun May 18 16:16:57 2014

 

Copyright (c) 1982, 2013, Oracle. All rights reserved.

 

 

Connected to:

Oracle Database 12c Enterprise Edition Release 12.1.0.1.0 - 64bit Production With the Automatic Storage Management option

 

 

SQL> set linesize 100

SQL> column path format a50

SQL>   select   name,  path  , header_status   from   v$asm_disk  where  path like 'ORCL:SAN_A_DISK_1';

 

NAME PATH HEADER_STATU

 

ORCL:SAN_A_DISK_1 PROVISIONED

 

SQL> create diskgroup TESTDG external redundancy disk 'ORCL:SAN_A_DISK_1';

 

Diskgroup created.

 

 

SQL> select group_number, name, state from v$asm_diskgroup where name like 'TESTDG';

 

GROUP_NUMBER NAME STATE

 

 

1           TESTDG MOUNTED

 

 

SQL> set linesize 100
SQL> column path format a50
SQL> select name, path , header_status from v$asm_disk where group_number = 1;
 
NAME PATH HEADER_STATU
 
 
SAN_A_DISK_1 ORCL:SAN_A_DISK_1 MEMBER
 

 

 

 

2) Then later, the diskgroup is dropped by accident and cannot be mounted again:

 

SQL> drop diskgroup TESTDG;

 

Diskgroup dropped.

 

 

 

SQL> alter diskgroup TESTDG mount; alter diskgroup TESTDG mount

*

ERROR at line 1:

ORA-15032: not all alterations performed

ORA-15017: diskgroup "TESTDG" cannot be mounted

ORA-15063: ASM discovered an insufficient number of disks for diskgroup "TESTDG"

 

3) The original disk member is now shown as a FORMER disk:

 

 

SQL> set linesize 100

SQL> column path format a50

SQL>   select   name,  path  , header_status   from   v$asm_disk  where  path like 'ORCL:SAN_A_DISK_1';

 

 

 

 

4) Kfed also reports the FORMER status:

 

 

[grid@asmlnx1 ~]$ kfed read /dev/oracleasm/disks/SAN_A_DISK_1 | head -25

 

kfbh.block.obj: 2147483648 ; 0x008: disk=0

kfbh.check: 2862642817 ; 0x00c: 0xaaa07681

 

kfdhdb.driver.provstr:ORCLDISKSAN_A_DISK_1 ; 0x000: length=20

kfdhdb.driver.reserved[0]:   1598964051 ; 0x008: 0x5f4e4153

kfdhdb.driver.reserved[1]:   1229217601 ; 0x00c: 0x49445f41

kfdhdb.driver.reserved[2]: 828328787 ; 0x010: 0x315f4b53 kfdhdb.driver.reserved[3]: 0 ; 0x014: 0x00000000 kfdhdb.driver.reserved[4]: 0 ; 0x018: 0x00000000 kfdhdb.driver.reserved[5]: 0 ; 0x01c: 0x00000000 kfdhdb.compat: 168820736 ; 0x020: 0x0a100000

 

kfdhdb.dskname: SAN_A_DISK_1 ; 0x028: length=12

kfdhdb.grpname: TESTDG ; 0x048: length=6 kfdhdb.fgname: SAN_A_DISK_1 ; 0x068: length=12

 

5) The ASM disk header needs to be patched/updated with the HEADER_STATUS = “MEMBER” in order to mount the diskgroup again as follow:

 

 

5.1) Backup the first 10 MB of the associated disks using the dd command:

 

 

[grid@asmlnx1 ~]$ mkdir /tmp/asm

 

[grid@asmlnx1~]$

of=/tmp/SAN_A_DISK_1.dump bs=1048576 count=50 50+0 records in

50+0 records out

52428800 bytes (52 MB) copied, 2.0715 seconds, 25.3 MB/s 5.2) Generate the template patch from the FORMER disk:

 

/tmp/asm/patch_SAN_A_DISK_1.txt

 

 

[grid@asmlnx1~]$ ls -l /tmp/asm/patch_SAN_A_DISK_1.txt

-rw-r--r-- 1 grid oinstall 6594 Apr 10 16:48 /tmp/asm/patch_SAN_A_DISK_1.txt

 

 

5.3) Edit the file (/tmp/asm/patch_SAN_A_DISK_1.txt) and change the code # from 4 to 3 and the status from FORMER to MEMBER in the next line (make sure you save the changes):

 

 

Original line:

 

  

New line:

 

 

 

Note: #4 stands for “FORMER” & #3 stands for “MEMBER”

 

5.4) Verify the new values in the patch file:

 

 

[grid@asmlnx1~]$ grep kfdhdb.hdrsts /tmp/asm/patch_SAN_A_DISK_1.txt

 

 

 

5.5) Now you can patch the FORMER disk using the “/tmp/asm/patch_SAN_A_DISK_1.txt” patch file:

 

 

text=/tmp/asm/patch_SAN_A_DISK_1.txt

 

 

5.6) Verify the new values in the real disk (ASM disk header was fixed with the HEADER_STATUS= “MEMBER”):

 

 

[grid@asmlnx1~]$ kfed read /dev/oracleasm/disks/SAN_A_DISK_1 | grep kfdhdb.hdrsts

 

 

 

 

6) Now you can mount the diskgroup:

 

 

[grid@asmlnx1 ~]$ sqlplus "/as sysasm"

 

SQL*Plus: Release 12.1.0.1.0 Production on Sun May 18 16:31:25 2014

 

Copyright (c) 1982, 2013, Oracle. All rights reserved.

 

 

Connected to:

Oracle Database 12c Enterprise Edition Release 12.1.0.1.0 - 64bit Production With the Automatic Storage Management option

 

SQL> alter diskgroup TESTDG mount;

 

Diskgroup altered.

 

SQL> set linesize 100

SQL> column path format a50

SQL>   select   name,  path  , header_status   from   v$asm_disk  where  path like 'ORCL:SAN_A_DISK_1';

 

 

SQL> alter diskgroup TESTDG mount;

 

 

Diskgroup altered.

 

 

 

Annex: Syntax to create the ASMLIB disk:

 

 

[root@asmlnx1 ~]# /etc/init.d/oracleasm querydisk -p SAN_A_DISK_1 Disk "SAN_A_DISK_1" is a valid ASM disk

/dev/asmdisk1_4k_udev_p1: LABEL="SAN_A_DISK_1" TYPE="oracleasm"

 

 

 

[root@asmlnx1 ~]# /etc/init.d/oracleasm createdisk SAN_A_DISK_1

/dev/asmdisk1_4k_udev_p1

Marking disk "SAN_A_DISK_1" as an ASM disk: [ OK ]