ASM Rebalance – I/O Saturation?

I really think that the ability to perform online storage reconfigurations is one of the killer features of ASM. Not only is it possible online, it also is relatively trivial to modify the configuration of a diskgroup with only a simple alter diskgroup command.

It does however beg the question: is it practical to do an online reconfiguration? Features are all very nice in theory, but can you afford the additional I/O that a rebalance will necessitate. Sure you can have a very low asm_power_limit, to try and minimise that I/O and you can even set this to 0 which will ensure that no automatic rebalance occurs, and then you as the DBA can decide when is the best for a rebalance to occur by doing a manual rebalance. Does this become a trade off between a low impact on your “normal” workload versus taking a bit longer to do the actual rebalance?

I see someone within oracle has a sense of humour as the asm_power_limit takes it all the way to 11.

I was doing an (automatic) rebalance and I thought I’d take a look at what kind of I/O load I was doing:

SQL> select * from v$asm_operation;

GROUP_NUMBER OPERA STAT   POWER  ACTUAL	SOFAR    EST_WORK EST_RATE EST_MINUTES
---------    ----- ---- ------- ------- --------   -------   -------    --------
4            REBAL  RUN    1	    1	  19504     83593    428	149

You can see this rebalance is running with an asm_power_limit of just 1, which is the default value.

A sample of iostat output shows the effect this is having on the I/O subsystem:

First thing to be aware is, this is an idle system – it’s just one of my test RAC clusters. So the only workload going on is the asm rebalance. So there are 2 devices involved in this diskgroup that is undergoing the rebalance, sdr, which was the original (only) member of the diskgroup and sdz which is the new member. You can see that sdr is having it’s extents read and some of these are being transferred to the other device within the diskgroup – just what you’d expect.

What maybe you would not expect, even with this less than optimised I/O environment is that we see that the utilisation of the sdr device goes through the roof. This is with power 1, I’d hate to see what (if anything) a higher power limit did, and I’d hate to see what a rebalance would do to a system that was undergoing a real-world workload – particularly one where you were trying to add more disk spindles due to a burdened I/O subsystem. I’m pretty sure a rebalance does not work out how busy the device is and then throttle up or down, the speed should only be determined by the asm_power_limit.

Looks like the best option for doing a rebalance is to find a “less busy” time and perform a manual rebalance, rather than have the automatic rebalance done when the diskgroups are reconfigured, at least you can control when to take the I/O hit.

---

Possibly related posts: (automatically generated)

• ASM and disk “hot spots”