Expanding an ASM disk

One of the major advantages of ASM is the ability to reconfigure the storage online. In theory you should be able to add disks, remove disks, and resize disks all the while your ASM and RDBMS instances just keep humming along.

However, I don’t think it is actually possible to expand a lun without downtime if you are using ASMLib. Part of the problem seems to be that with ASMLib you have to create a partition and certainly on RedHat 4 Update 3, using kernel 2.6.9-34.ELsmp, to change a partition table required that ASM was not using the disk that the partition table was residing on.

Recently I found this out the hard way when I attempted to increase the size of a lun that was being used by ASM. Expanding the lun on the storage was fairly straightforward on the EMC Clariion on which the data was residing.

I’m not really sure if this is the best way of mapping OS device -> ASM disk:

[jason@bdb ~]$ sudo /etc/init.d/oracleasm querydisk VOL4
Disk “VOL4″ is a valid ASM disk on device [8, 1]


I believe this to be the major and minor number of the device, so you can look in /dev to see what device this corresponds to:

[jason@bdb ~]$ ls -l /dev/sda1
brw-rw—- 1 root disk 8, 1 Apr 3 16:29 /dev/sda1


Or indeed thanks to Charles Kim you can run the querydisk the opposite way round:

[jason@bdb ~]$ sudo /etc/init.d/oracleasm querydisk /dev/sda1
Disk “/dev/sda1″ is marked an ASM disk with the label “VOL4″


I cannot see in any V$ASM view where this mapping from asm disk -> OS device is exposed, perhaps ASMLib is getting in the way here. What can say is that ASM disk VOL4 maps to /dev/sda1 which is a partiton on /dev/sda. I then increased the size of the lun that this device was created from.

Then comes the scary part, getting the OS to see the increased lun. This was running on rhel 4 update 3, and after a reboot I could see the following via fdisk:

[jason@bdb ~]$ sudo /sbin/fdisk /dev/sda

Command (m for help): p

Disk /dev/sda: 429.4 GB, 429496729600 bytes
255 heads, 63 sectors/track, 52216 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 1 32635 262140606 83 Linux


So the OS can see that the device /dev/sda now has 52216 cylinders but only 32635 (which was the original lun size) have been allocated to the partition /dev/sda1. Now you can actually just delete the partition and recreate it without losing any data:

Command (m for help): d
Selected partition 1


This has deleted the partition

Command (m for help): p

Disk /dev/sda: 429.4 GB, 429496729600 bytes
255 heads, 63 sectors/track, 52216 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System


Now you have to recreate it:

Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-52216, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-52216, default 52216):
Using default value 52216

Now we can see the /dev/sda1 partition is up to the full capacity of the underlying lun:

Command (m for help): p

Disk /dev/sda: 429.4 GB, 429496729600 bytes
255 heads, 63 sectors/track, 52216 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 1 52216 419424988+ 83 Linux

Don’t forget to write the changes out:


Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

I actually had ASM shut down at this point because fdisk had previously stated the device was busy (when trying to write the new partition table) and that the kernel would still use the old partition:


WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table.
The new table will be used at the next reboot.
Syncing disks.


Once ASM (and obviously the RDBMS instance relying on this ASM instance) was down, I was able to write the partition table. Without changing the partition table, ASM would not recognise that the luns had been increased. After this partition table was written, getting ASM to increase what it thought was the size of disk was quite simple:


SQL> select group_number, name, TOTAL_MB, FREE_MB
from V$asm_disk_stat; 2

GROUP_NUMBER NAME TOTAL_MB FREE_MB
———— —————————— ———- ———-
1 VOL1 61439 61187
2 VOL2 61439 61164
3 VOL3 61439 61164
4 VOL4 255996 157374
4 VOL5 153597 95230

SQL> alter diskgroup DATA4 resize all rebalance power 4;

Diskgroup altered.


SQL> select group_number, name, TOTAL_MB, FREE_MB
from V$asm_disk_stat; 2

GROUP_NUMBER NAME TOTAL_MB FREE_MB
———— —————————— ———- ———-
1 VOL1 61439 61187
2 VOL2 61439 61164
3 VOL3 61439 61164
4 VOL4 409594 310962
4 VOL5 153597 95240


So what am I saying, well it seems that with ASMLib it is hard to resize a disk completely online with ASM, but due to the fact that the Linux partition table cannot be re-written while ASM has open the device.

Perhaps, this is a disadvantage of ASMLib compared to running without out, though for my money ASMLib seems to be the favoured Oracle solution, certainly I think it is pushed in the documentation.