Adding a disk to an ASM diskgroup can be seen simple. But if you will do this in a production database, all steps in this article should be done very carefully.
When adding new disks to a diskgroup, it is advisable to add all new disks in the same command. This allows ASM to rebalance once for all new disks, instead of repeatedly as each new disk is added.
If disks need to be removed from an ASM disk group, then again all disks should be removed in a single command to minimize the rebalance operations.
Lastly, if you add and remove disks to ASM disk group in the same time, you should prepare single command to minimize the rebalance operations.
ADDING DISK TO ASM DISK GROUP
CREATE DISK AND MAP TO DATABASE SERVERS
This step is done by your linux/storage administrators.
!!! DISKS IN THE DISK GROUP WE WILL CREATE FOR ASM SHOULD BE THE SAME SIZE !!!
!!! FURTHER, DISKS MUST BE UNFORMATTED AND UNUSED !!!
ENABLE MULTIPATH SERVICE ON BOTH NODES
This step is also done by your linux administrators.
1. Create /etc/ multipath.conf With Needed Storage Info
2. Start multipathd service
service multipathd start
3. To Start multipath Service Automatically After Reboot
chkconfig –add multipathd
chkconfig multipathd on
4. Show The Current Multipath Topology From All Available Information
multipath - ll
DISK PARTITION PROCESS IS DONE ONLY NODE1
!!! APPLY BELOW STEPS ONLY ON NODE1 !!! NO OPERATION MUST BE DONE ON NODE2 !!!
!!! HOW MANY DISKS WILL BE USED, THE FOLLOWING OPERATIONS ARE DONE FOR ALL DISKS !!!
You can take your device name (/dev/sdf1) from linux/storage admins.
[root@racnode1 ~]# parted /dev/sdf
GNU Parted 3.1
Using /dev/sdf
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) p
Model: QEMU QEMU HARDDISK (scsi)
Disk /dev/sdf: 4398GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:
Number Start End Size File system Name Flags
(parted) mkpart primary 0.00GB 4096GB
(parted) print
Model: QEMU QEMU HARDDISK (scsi)
Disk /dev/sdf: 4398GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:
Number Start End Size File system Name Flags
1 1049kB 4096GB 4096GB primary
(parted) q
Information: You may need to update /etc/fstab.
[root@racnode1 ~]# ll /dev/sdf*
brw-rw---- 1 root disk 8, 80 Nov 19 00:16 /dev/sdf
brw-rw---- 1 root disk 8, 81 Nov 19 00:16 /dev/sdf1CHECK ASM DISK NAMES&DISK GROUP NAMES AND DECIDE ASM DISK NAME
I decided to use ASMDISK04 as name after below checks.
[root@racnode1 ~]# ll /dev/oracleasm/disks/
brw-rw---- 1 grid asmadmin 8, 33 Nov 19 01:42 ASMDISK01
brw-rw---- 1 grid asmadmin 8, 49 Nov 19 00:44 ASMDISK02
brw-rw---- 1 grid asmadmin 8, 17 Nov 19 00:42 ASMDISK03
[root@racnode1 ~]#
[root@racnode1 ~]# oracleasm listdisks
ASMDISK01
ASMDISK02
ASMDISK03
[root@racnode1 ~]#
SQL> SET linesize 999;
SET pagesize 999;
COL asmdisk_name FOR a15
COL mount_status FOR a15
COL header_status FOR a15
COL mode_status FOR a15
COL state FOR a15
COL total_mb FOR a15
COL free_mb FOR a15
COL path FOR a35
COL label FOR a15
COL diskgroup_name FOR a15
SELECT SUBSTR (d.name, 1, 16) AS asmdisk_name,
d.mount_status,
d.header_status,
d.mode_status,
d.state,
TO_CHAR (d.total_mb) total_mb,
TO_CHAR (d.free_mb) free_mb,
d.PATH,
d.label,
dg.name AS diskgroup_name
FROM V$ASM_DISKGROUP dg RIGHT OUTER JOIN V$ASM_DISK d USING (group_number);STAMP THE DISKS. DO THIS OPERATION ONLY ON NODE1
!!! HOW MANY DISKS ARE GIVEN, THE FOLLOWING OPERATIONS ARE DONE FOR ALL DISKS !!!
The name of the disk “ASMDISK04” can be of your choice and preferably should denote the purpose of the disk.
You can take your device name (/dev/sdf1) from linux/storage admins.
[root@racnode1 /]# oracleasm createdisk ASMDISK04 /dev/sdf1
Writing disk header: done
Instantiating disk: done
[root@racnode1 /]#CHECK NEWLY ADDED ASM DISK
[root@racnode1 ~]# ll /dev/oracleasm/disks/
brw-rw---- 1 grid asmadmin 8, 33 Nov 19 01:42 ASMDISK01
brw-rw---- 1 grid asmadmin 8, 49 Nov 19 00:44 ASMDISK02
brw-rw---- 1 grid asmadmin 8, 17 Nov 19 00:42 ASMDISK03
brw-rw---- 1 grid asmadmin 8, 17 Nov 19 00:42 ASMDISK04
[root@racnode1 ~]#
[root@racnode1 ~]# oracleasm listdisks
ASMDISK01
ASMDISK02
ASMDISK03
ASMDISK04
[root@racnode1 ~]#Run the below command as grid user on each of the new disks discovered from the “listdisks” command.
Syntax
oracleasm querydisk <diskname_output_of_oracleasm listdisks>
Example
[root@racnode1 ~]# oracleasm querydisk ASMDISK004
Disk "ASMDISK004" is a valid ASM disk
[root@racnode1 ~]#DO SCANDISK ON THE ALL THE NODES (IF RAC)
[root@racnode2 ~]# oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks…
Scanning system for ASM disks…
Instantiating disk "ASMDISK004"
[root@racnode2 ~]#CHECK NEWLY ADDED ASM DISK
[root@racnode2 ~]# ll /dev/oracleasm/disks/
brw-rw---- 1 grid asmadmin 8, 33 Nov 19 01:42 ASMDISK01
brw-rw---- 1 grid asmadmin 8, 49 Nov 19 00:44 ASMDISK02
brw-rw---- 1 grid asmadmin 8, 17 Nov 19 00:42 ASMDISK03
brw-rw---- 1 grid asmadmin 8, 17 Nov 19 00:42 ASMDISK04
[root@racnode2 ~]#
[root@racnode2 ~]# oracleasm listdisks
ASMDISK01
ASMDISK02
ASMDISK03
ASMDISK04
[root@racnode2 ~]#Run the below command as grid user on each of the new disks discovered from the “listdisks” command.
Syntax
oracleasm querydisk <diskname_output_of_oracleasm listdisks>
Example
[root@racnode1 ~]# oracleasm querydisk ASMDISK004
Disk "ASMDISK004" is a valid ASM disk
[root@racnode1 ~]#CHECK THE VISIBILITY OF THE DISK AT OS LEVEL:( AS GRID OWNER FROM ALL THE NODES)
Syntax
$dd if=[raw/character or block partition device in question] of=/dev/null count=100 bs=8192
Example : Raw Disk
$dd if=/dev/sdf1 of=/dev/null count=100 bs=8192
Example : ASMLIB device
$dd if=/dev/oracleasm/disks/ASMDISK04 of=/dev/null count=100 bs=8192USE KFED TO READ THE DISKS:(AS GRID OWNER FROM ALL THE NODES)
Verify that kfed can read on the candidate disk(s) at OS level (as oracle or grid OS user) from each node
Syntax
$kfed read [raw/character or block partition device in question]
Example : Raw Disk
$kfed read /dev/sdf1
Example : ASMLIB device:
$kfed read /dev/oracleasm/disks/ASMDISK04If you are getting an error like kfbh.type : 0 ; 0x002: KFBTYP_INVALID, then you can ignore it safely.
NOW CREATE A DISKGROUP FOR TESTING PURPOSE
It is always a good practice to create a temporary disk group using the new disk(s) to verify everything works fine without touching the production disk groups. Worst case scenario the test diskgroup would fail, which is fine.
Syntax
SQL> CREATE DISKGROUP TEST EXTERNAL REDUNDANCY DISK '[YOUR CANDIDATE DISK]';
Example
CREATE DISKGROUP TESTDG EXTERNAL REDUNDANCY DISK '/dev/oracleasm/disks/ASMDISK04';CHECK IF THE DISKGROUP IS CREATED AND MOUNTED
SQL> SELECT STATE, NAME FROM V$ASM_DISKGROUP;IF IT IS NOT MOUNTED, MOUNT IT MANUALLY USING THE BELOW COMMAND
Make sure you try mounting the TEST disk group from each ASM instance in case of a RAC configuration.
SQL> ALTER DISKGROUP TEST MOUNT;
SQL> SELECT STATE, NAME FROM V$ASM_DISKGROUP;CREATE A TABLESPACE ON THIS DISKGROUP
SQL>Create tablespace TEST_TS datafile '+TESTDG' size 5G;
Tablespace createdAs tablespace created, we can confirm that the disk is healthy. We can clean the diskgroup.
LET'S DROP THE TABLESPACE AND THE DISKGROUP
SQL> DROP TABLESPACE TEST_TS;From All The ASM Instances, Except From One
SQL> ALTER DISKGROUP TESTDG DISMOUNT; From The ASM Instance, Which The Diskgroup Is Still Mounted
SQL> DROP DISKGROUP TESTDG; If you try to drop the diskgroup, without dismounting the diskgroup from all instance, except one, then you will hit the error.
ORA-15073: diskgroup TESTDG is mounted by another ASM instance
Now we have cleaned up diskgroup, after confirming that disk is perfectly healthy. So let's proceed and add that disk to your required existing diskgroup.
ADD DISK TO REAL ASM DISK GROUP
!!! WHEN ADDING NEW DISKS TO A DISKGROUP, IT IS ADVISABLE TO ADD ALL NEW DISKS IN THE SAME COMMAND. THIS ALLOWS ASM TO REBALANCE ONCE FOR ALL NEW DISKS, INSTEAD OF REPEATEDLY AS EACH NEW DISK IS ADDED !!!
Finally, after rigorously validating the new disks, we can add them to the desired disk group. Add the candidate disk to the disk group using the command below.
$sqlplus / as sysasm
Syntax
SQL> ALTER DISKGROUP [my production diskgroup] ADD DISK '[your candidate disk]';
Example
SQL> ALTER DISKGROUP DATA ADD DISK '/dev/oracleasm/disks/ASMDISK04' NAME DATA_001 rebalance power 1024;
In cases where the power value is not given, this value will be determined according to the asm_power_limit parameter of the ASM.
SQL> show parameter limit
NAME TYPE VALUE
------------------- ----------- ------
asm_power_limit integer 1CHECK THE REBALANCE STATUS
$sqlplus / as sysasm
SQL> select group_number, operation, state, est_minutes from v$asm_operation;If no rows returned, then the rebalance is compeleted.
CHECK THE NEWLY ADDED DISK IN ASM DISKGROUP
SQL> SET linesize 999;
SET pagesize 999;
COL asmdisk_name FOR a15
COL mount_status FOR a15
COL header_status FOR a15
COL mode_status FOR a15
COL state FOR a15
COL total_mb FOR a15
COL free_mb FOR a15
COL path FOR a35
COL label FOR a15
COL diskgroup_name FOR a15
SELECT SUBSTR (d.name, 1, 16) AS asmdisk_name,
d.mount_status,
d.header_status,
d.mode_status,
d.state,
TO_CHAR (d.total_mb) total_mb,
TO_CHAR (d.free_mb) free_mb,
d.PATH,
d.label,
dg.name AS diskgroup_name
FROM V$ASM_DISKGROUP dg RIGHT OUTER JOIN V$ASM_DISK d USING (group_number);REMOVING DISK FROM ASM DISK GROUP
!!! IF DISKS NEED TO BE REMOVED FROM AN ASM DISK GROUP, THEN AGAIN ALL DISKS SHOULD BE REMOVED IN A SINGLE COMMAND TO MINIMIZE THE REBALANCE OPERATIONS !!!
Example
SQL> alter diskgroup DATA drop disk DATA_001 rebalance power 1024;In cases where the power value is not given, this value will be determined according to the asm_power_limit parameter of the ASM.
SQL> show parameter limit
NAME TYPE VALUE
------------------- ----------- ------
asm_power_limit integer 1CHECK THE REBALANCE STATUS
$sqlplus / as sysasm
SQL> select group_number, operation, state, est_minutes from v$asm_operation;If no rows returned, then the rebalance is compeleted.
CHANGING REBALANCE POWER IF NEEDED
It is possible to change the speed after starting the Rebalance process. You can do this as follows.
Example
SQL> alter diskgroup DATA rebalance power [0-1024];ADDING AND REMOVING DISKS FROM ASM DISK GROUP IN THE SAME TIME
How to ADD/DROP ASM DISK in SINGLE COMMAND (Doc ID 1910831.1)
When a disk is dropped, the disk group is rebalanced by moving all of the file extents from the dropped disk to other disks in the disk group. A drop disk operation might fail if not enough space is available on the other disks. The best approach is to perform both the add and drop operation with the same ALTER DISKGROUP statement. This has the benefit of rebalancing data extents only one time and ensuring that there is enough space for the rebalance operation to succeed.
Use below syntax to add and drop disks in a single command.
Note:In the 'drop disk' clause, v$ asm_disk.name should be used. Also multiple disks can be added and dropped and ensure that you need add same size of the disks that are going to be dropped.
Login into ASM instance as sysasm.
$sqlplus / as sysasm
Syntax
SQL> alter diskgroup <DG_NAME_1> add disk '<disk path>' <DISK_NAME1> drop disk <DISK_NAME_TO_BE_DROPPED>;
Example
SQL> ALTER DISKGROUP DATA ADD DISK '/dev/oracleasm/disks/ASMDISK04' NAME DATA_001 drop disk DATA_000 rebalance power [0-1024];HOW TO DELETE ASM DISK ON MULTIPATH DEVICE IN CENTOS/RHEL
The post outlines steps to remove ASM disk on multipath or local device (Labeled as ASMDISK01 as an example in this post) from server.
1. First make sure that ASM disk is no longer used by any disk group, otherwise, you might risk data integrity. The DBA can check this by using asmcmd or by logging into the ASM database.
SQL> SET linesize 999;
SET pagesize 999;
COL asmdisk_name FOR a15
COL mount_status FOR a15
COL header_status FOR a15
COL mode_status FOR a15
COL state FOR a15
COL total_mb FOR a15
COL free_mb FOR a15
COL path FOR a35
COL label FOR a15
COL diskgroup_name FOR a15
SELECT SUBSTR (d.name, 1, 16) AS asmdisk_name,
d.mount_status,
d.header_status,
d.mode_status,
d.state,
TO_CHAR (d.total_mb) total_mb,
TO_CHAR (d.free_mb) free_mb,
d.PATH,
d.label,
dg.name AS diskgroup_name
FROM V$ASM_DISKGROUP dg RIGHT OUTER JOIN V$ASM_DISK d USING (group_number);2. Find out ASM disk mapping
[root@racnode1 ~]# oracleasm listdisks
DISK00
...
[root@racnode1 ~]# oracleasm querydisk -p DISK00
Disk "DISK00" is a valid ASM disk
/dev/sdah1: LABEL="DISK00" TYPE="oracleasm" PARTLABEL="primary" PARTUUID="63805bfa-8a2c-46cf-a3b9-03e0f2626f5e"
[root@racnode1 ~]#
[root@racnode1 ~]# oracleasm querydisk -d DISK00
Disk "DISK00" is a valid ASM disk on device [66,17]
[root@racnode1 ~]#3. Identify multipath or local device used for DISK00 disk from ASM:
For Multipath
[root@racnode1 ~]# multipath -ll
For Local DISK
[root@racnode1 ~]# fdisk -l 4 . After you have confirmed the disk to be deleted as well as its corresponding disk in device multipath or local, you can go ahead and delete it from ASM first.
[root@racnode1 ~]# oracleasm deletedisk DISK00
Removing ASM disk "DISK00": [ OK ]5. Next step is to clean up the device multipath or local for the deleted disk. You can use any of the commands below to delete the multipath or local disk.
For Multipath
[root@racnode1 ~]# multipath -ll
mpath1 dm-4 IBM DS
size=10G features='0' hwhandler='0' wp=rw
...
[root@racnode1 ~]# multipath –f mpath1
For Local DISK
[root@racnode1 ~]# echo 1 > /sys/block/sdah1/device/delete6. LUN could be removed from SAN Storage Server now by your linux/storage administrators.
7. Make sure multipath or local is removed from server
For Multipath
[root@racnode1 ~]# multipath -ll
For Local DISK
[root@racnode1 ~]# fdisk -l SCRIPT TO CHECK WHICH ASM DISKS ARE MAPPED TO WHICH PHYSICAL DISK PATHS
[+ASM1 racnode1:/tmp>] more asm_mapped_ph_disks.sh #!/bin/bash ## ASMLIB_DISK -- disk name in ASMLIB ## ASM_DISK -- disk name in ASM ## DEVICE -- physical disk name GRID_HOME=`cat /etc/oratab | grep ^+ASM | awk -F":" '{print $2}'` for ASMLIB_DISK in `ls /dev/oracleasm/disks/*` do ASM_DISK=`$ORACLE_HOME/bin/kfed read $ASMLIB_DISK | grep dskname | tr -s ' '| cut -f2 -d' '` majorminor=`ls -l $ASMLIB_DISK | tr -s ' ' | cut -f5,6 -d' '` device=`ls -l /dev/ | tr -s ' ' | grep -w "$majorminor" | cut -f10 -d' '` echo "ASMLIB disk name : $ASMLIB_DISK" echo "ASM_DISK name : $ASM_DISK" echo "Physical disk device : /dev/$device" done [+ASM1 racnode1:/tmp>]
No comments:
Post a Comment