ASM Disk Discovery

There is a bit of confusion around about how you can present devices for ASM to use. ASM disk discovery is essentially the process by which ASM discovers device names that it is allowed access to. The easiest mechanism to get ASM to see devices is to use the device name as discovered by the O/S in conjunction with the ASM_DISKSTRING parameter.

This parameter allows the ASM instance to find all devices within the location that the parameter has been set to, and wildcards are allowed. For ASM to see the device you want, it must be located within the path you have specified in the ASM_DISKSTRING parameter AND the permissions must be such that whatever user your ASM instance runs as can read/write to the device you want ASM to find.

The ASM_diskstring has default values that are operating system dependent. On Linux the default of ASM_DISKSTRING is /dev/raw/*, I’d change this to /dev/sd* assuming your devices follow this nomenclature.

While on Solaris the default is /dev/rsdk/* It is important to note on Solaris you have to use the devices under the rdsk tree NOT the /dev/dsk tree!

Once a new device is added to allow ASM to see it, a select on either the V$ASM_DISK, or V$ASM_DISKGROUP views kicks of a discovery process.

It is the RBAL background process that is responsible for discovering the devices. Running an strace on the rbal background process after adding a new device shows the following:

14472 access("/dev/loop4", R_OK|W_OK)   = 0
14472 stat64("/dev/loop4", {st_mode=S_IFBLK|0660, st_rdev=makedev(7, 4), ...}) = 0
14472 open("/dev/loop4", O_RDONLY|O_LARGEFILE) = 25
14472 ioctl(25, BLKGETSIZE, 0xbfd90a70) = 0
14472 close(25)

In the above I had ASM_DISKSTRING set to /dev/loop* and therefore RBAL discovers all devices, including ones ASM already knows about (this is not shown in the above strace output) beginning with loop within the /dev/ directory. This then presented me (I already had some devices within a diskgroup) with the following in V$ASM_DISK:

sys@+ASM> select group_number, disk_number, mount_status, header_status, path from v$asm_disk; 
GROUP_NUMBER DISK_NUMBER MOUNT_S HEADER_STATU PATH
------------ ----------- ------- ------------ --------
           0           0 CLOSED  CANDIDATE    /dev/loop4
           1           0 CACHED  MEMBER       /dev/loop0
           1           2 CACHED  MEMBER       /dev/loop2
           1           1 CACHED  MEMBER       /dev/loop1

A mount_status of closed means ASM can see the disk but it is not using it. A header_status of candidate means the disk is not part of a disk group (as opposed to MEMBER which means it is part of a diskgroup) but can be added to one.

It’s not all that scary, but I guess it might be outside some DBAs comfort zones!

Advertisements

4 thoughts on “ASM Disk Discovery

  1. I’m wondering if you’ve run into the issue that we have on Solaris. We chown the ASM assigned devices to oracle:dba so that it can be used. However, whenever a reconfiguration reboot occurs, the ownership of the entire tree get set back to root:sys.

    We’ve created a small start-up script read a newly invented file /etc/asmtab and to chown these devices back but I was wondering if there is a more elegant solution to this problem.

  2. Hi Tim,

    Interesting. I have not used ASM on solaris. So, have not experienced this issue.

    However, I have seen recommendations to use mknod to create a new device that points to the actual /dev/rdsk device, you then chown (oracle:db) the device you have created with mknod and then set ASM_DISKSTRING appropriately to use this newly created device, rather than the /dev/rdsk one (though of course they are pointing to the same bit of tin).

    Drop me a note if you need further details.

    It would be interesting to hear if this procedure works.

    jason.

  3. Jason,
    Thank you for your thoughts on ASM. I have a question maybe you can shed light on. I have a customer who is using ASM. One day, they were standing at the storage rack and saw that a disk had a light on that they thought indicated some sort of failure. They pulled out the disk and re-seated it. Light went away. It turns out, this disk was part of the ASM configuration. The DISKGROUP that this disk was a part of has two failgroups, each with 5 disks. As you might expect, ASM detected the disk being removed, and rebalanced all the data that was on it to the other 4 remaining disks in the failgroup. ASM also closed the disk (offlined it?). Now the header_status shows UNKNOWN and I can’t seem to add it back without getting an error. The physical check of the disk using OS utilities seems fine. This is on Oracle Enterprise Linux 5 Using Oracle RDBMS 11.1.0.7. If you have any thoughts, thank you in advance.
    Chris Ruel..
    Indianapolis, IN

  4. I have a configuration where full device-path-names were used to create ASM diskgroups (Oracle DB on Solaris 10).
    Recently the LUN IDs were changed on the SAN storage and so the disk-device-names changed (on the Solaris 10 host) and so the ASM could not find the disks for the diskgroups.
    I have tried setting the asm_diskstring parameter to a wildcard but that didn’t work. Though it just struck me now that what if the ownership had changed like Tim mentioned. I have also seen the recommendation about creating ones own device files somewhere else in the filesystem e.g., /racdevs and then using those as wildcards in the asm_disktring. Assuming asm_diskstring is not set, the question is does ASM actually search for the member disks in a diskgroup that was created using full disk-device-paths or does ASM just only looks for those devices exactly as they are specified? If ASM searches every time it’s started, then setting the asm_disktring should probably work. In which case, I will investigate the permissions on the devices and the possibility of creating my own special device files. Thanks

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s