Keep Disks in your Diskgroup the same size

In my production instances, I have only ever used ASM with external redundancy. Hey, I’m paying expensive fees for fancy hardware RAID, I might as well use it. I therefore tend to present to ASM a LUN which is normally a RAID 10 stripe set. As far as ASM is concerned this is one large disk and it does not have to worry about failure groups.

Well, that is the ideal, but as we know we don’t live in a perfect world. On one of the production instances, due to using “hand me down” hardware, I have created 2 LUNS, and these are of differing sizes. Everything was running along happily until, one of the LUNS ran out of space.

I had thought ASM was meant to distribute extents based on the size of the disks in the diskgroup, so for example, if your diskgroup was made up of 2 disks and one was say 60GB and one was 120GB the 120GB would contain twice as many extents (Allocation Units) as the 60GB disk. This would ensure that one disk in the diskgroup was never filled up while the other one still had plenty of space. Well, it seems that this does not necessarily work perfectly in practice.

So I have a diskgroup lets, call it DATA4 and it is made up of two disks VOL4 and VOL5 and when you look at V$ASM_DISKGROUP this diskgroup has lots of lovely free space:

 SQL> select group_number, name, total_mb, free_mb
 from V$ASM_DISKGROUP;

GROUP_NUMBER NAME		 TOTAL_MB	FREE_MB
------------ ------------------ ----------     ----------
4            DATA4		 220391	       24501

So If you just looked at that view you would be hard pushed to explain why you could not allocate space in your diskgroup. However if you diskgroup is made up of multiple disks take a look at the following view:

SQL> select group_number, name, TOTAL_MB, FREE_MB
from V$asm_disk_stat;
GROUP_NUMBER NAME			      TOTAL_MB	  FREE_MB
------------ ------------------------------ ---------- ----------
	   4 VOL4				124660	    24501
	   4 VOL5				 95731	        0

Oh great! All the available space in the diskgroup is on one of the disks in the diskgroup. ASM is not clever enough to just then allocate new extents to this disk in the diskgroup, it will just keep on doing it’s effective round robin distribution of extents, which means you will get an ORA-15041 error saying the diskgroup space is exhausted. And you’ll be convinced that it ain’t so if you just look at V$ASM_DISKGROUP.

Thankfully, there is help at hand to fix this in the rebalance process. I had thought a rebalance was only required when the storage had physically changed, i.e. adding a new disk, but a rebalance basically evened out where the data was stored:

SQL> alter diskgroup DATA4 rebalance;

You can set a variable level of speed to the rebalance using the power syntax. After the rebalance completed and it took 41 minutes at power 1. I saw the following in V$ASM_DISK_STAT:

SQL> select group_number, name, TOTAL_MB, FREE_MB
from V$asm_disk_stat;
GROUP_NUMBER NAME			      TOTAL_MB	  FREE_MB
------------ ------------------------------ ---------- ----------
	   4 VOL4				124660	    13859
	   4 VOL5				 95731	    10642

Bingo! I can now allocate new extents in my diskgroup and I have not increased the storage available by 1 byte.

Definitely, it will save you pain if you keep all disks in your diskgroup the same size.

Book Review: Oracle Automatic Storage Management

I have been using ASM for 2 years now on both RAC and single instance servers. I think I have a good handle on how to use ASM, but the documentation is extremely light on how ASM actually works and interacts with the RDBMS instance it is supporting. Things are improved a little with the 11g documentation, in that there is a new storage administrators guide book, but still it is not exactly clear on how ASM actually works.

I was therefore keen to get my hands on Oracle Automatic Storage Management which would hopefully give a better insight into how ASM works than is available via the documentation alone. Interesting to note that two of the authors are also two authors on the oracle documentation itself.

I would say it’s a bit hit and miss. The book starts with an introduction from the architect of ASM, Bill Bridge. This was fascinating, in how long it took to go from initial idea to released product, 7 years! I also found it interesting that part of the initial design goal was that there would not be a directory tree, and files would not have names nor be accessible via the standard OS utilities. It now makes sense as to why ASMCMD feels like a bolted on piece of crud. I wonder how much this lack of filesystem familiarity has hurt ASM take up? Certainly when first released I was skeptical, partly as I was so used to being able to “see” my database datafiles - it’s a bit of a mindset shift moving from a cooked filesystem. I’m still not all that convinced it brings much to a single instance, but I think ASM with RAC makes a lot of sense. The changes in name of the product over the years also give an amusing insight into marketing at Oracle.

Chapters 10, & 11 really give you the added value over and above the online Oracle documentation, in that they give more detail on how ASM actually works in terms of the metadata and how the RDBMS actually ineracts with ASM held datafiles. I found the myth-busting in these chapters very worthwhile. While Chapter 5 has a little on Allocation Units and extents that is also of use. Unfortunately, I would say quite a few of the Chapters have a bit of filler in them, and there are many screenshots. I’m not all that keen on screenshots and I personally find they take up a lot of space for little added value. I also felt a few of the chapters did not give much added value over the documentation, which to be fair does tell you what you need to get up and running with ASM.

So all in all, there are some nuggets in this title that I found interesting/useful to know, but I suspect you could bump along happily without it and if Oracle improved their documentation just ever so slightly, you could save yourself a good 30 pounds sterling.

Thoughts on the X4500

We have recently obtained a Sun Fire X4500 storage array at work. This has an astonishing 24TB of storage in a tiny footprint of 4U. The price per Terabyte is utterly compeling, coming in at under £700 per Terabyte. Or put another way less than £0.70/GB. This storage is delivered via 48 drives. Ok so you don’t get hardware RAID with this so I guess a direct comparison is unfair, but we also have a lot of these Clariion Cx300’s but the price per GB comes in at around £4/GB so in terms of raw storage space you are talking going on x 6 the cost, and yes you should not buy your storage in terms of space capacity but I/O capacity, but these numbers make things very interesting indeed.

But the X4500 is not just a storage array, it is also a 4 way (2-dual core cpu’s) opteron server and the plans are to run Oracle on this, while you can install Linux (or even dare I say Windows) on here I think it’s going to be Solaris and we will then use ZFS to provide some data protection, possibly straight RAID 1/0 but maybe RAID-Z. It will also be interesting to see what kind of I/O we can get out of ZFS on here, though the theoretical maximum seems to be 2GB/s from disk to memory - we will see.