Creating ASM diskgroups on Exadata with ASMCA

I recently had the chance to create some diskgroups on an Exadata box outside the standard installation procedure, while this is not necessarily Exadata specific, I thought the technique of using ASMCA silently on the command line to create the diskgroups was sufficiently novel for a short blog posting. If for nothing else but to remind myself on how to do this in future.

This example uses the disks presented from a quarter rack exadata system and creates a diskgroup called DATA01:

asmca -silent -createDiskGroup -diskGroupName 'DATA01' -diskList o/192.168.10.14/DATA01_CD_00_cel01,o/192.168.10.14/DATA01_CD_01_cel01,
o/192.168.10.14/DATA01_CD_02_cel01,o/192.168.10.14/DATA01_CD_03_cel01,
o/192.168.10.14/DATA01_CD_04_cel01,o/192.168.10.14/DATA01_CD_05_cel01,
o/192.168.10.14/DATA01_CD_06_cel01,o/192.168.10.14/DATA01_CD_07_cel01,
o/192.168.10.14/DATA01_CD_08_cel01,o/192.168.10.14/DATA01_CD_09_cel01,
o/192.168.10.14/DATA01_CD_10_cel01,o/192.168.10.14/DATA01_CD_11_cel01,
o/192.168.10.15/DATA01_CD_00_cel02,o/192.168.10.15/DATA01_CD_01_cel02,
o/192.168.10.15/DATA01_CD_02_cel02,o/192.168.10.15/DATA01_CD_03_cel02,
o/192.168.10.15/DATA01_CD_04_cel02,o/192.168.10.15/DATA01_CD_05_cel02,
o/192.168.10.15/DATA01_CD_06_cel02,o/192.168.10.15/DATA01_CD_07_cel02,
o/192.168.10.15/DATA01_CD_08_cel02,o/192.168.10.15/DATA01_CD_09_cel02,
o/192.168.10.15/DATA01_CD_10_cel02,o/192.168.10.15/DATA01_CD_11_cel02,
o/192.168.10.16/DATA01_CD_00_cel03,o/192.168.10.16/DATA01_CD_01_cel03,
o/192.168.10.16/DATA01_CD_02_cel03,o/192.168.10.16/DATA01_CD_03_cel03,
o/192.168.10.16/DATA01_CD_04_cel03,o/192.168.10.16/DATA01_CD_05_cel03,
o/192.168.10.16/DATA01_CD_06_cel03,o/192.168.10.16/DATA01_CD_07_cel03,
o/192.168.10.16/DATA01_CD_08_cel03,o/192.168.10.16/DATA01_CD_09_cel03,
o/192.168.10.16/DATA01_CD_10_cel03,o/192.168.10.16/DATA01_CD_11_cel03 -redundancy NORMAL -compatible.asm 11.2.0.0 -compatible.rdbms 11.2.0.0 -sysAsmPassword welcome1 -silent

Note I’ve edited the above to have returns after every couple of disks for readability. Of course you can specify your required redundancy level and differing compatibile parameters. You can see the 3 cells here that make up an Exadata 1/4 rack. You can also see the Infiniband IP addresses of each of these cells in the path to each griddisk.

You can see however the nice mapping between each of the griddisks names and the storage cells upon which they reside (e.g. DATA01_CD_00_cel01 – DATA01 griddisk created on Celldisk 00 of storage cel01) – I really like this naming feature of Exadata it makes life that little bit more straightforward for the administrator.

Only other thing to be aware of, when creating the DBFS_DG diskgroup all those CD_00, and CD_01 griddisks don’t exist as this is the storage on each cell is used for the systems disks.

About these ads
Previous Post
Leave a comment

6 Comments

  1. Chris

     /  May 22, 2012

    Jason, very timely post. I’m working on a silent 11gR2 RAC rollout and this was spot-on. Many thanks!

    Reply
  2. Bal

     /  May 29, 2012

    Hi jason, I am working on a exadata X2 2 1/4 machine. I see there are 36 disks across 3 exadata storage cells each of size 423 GB. There are 3 failure groups defined across 3 storage cells. I want to make a DATA disk group using these 36 disks but with NORMAL redundency. My concern is whether I need to move 12 disks of 423 GB each to other two failure group so that I will get all 18 disk space in normal redundancy.

    Reply
    • jarneil

       /  May 29, 2012

      Hi,

      I think you are misunderstanding how NORMAL redundancy works. Each failure group is defined for a self-contained set of disks that could fail together.

      Hence in your 1/4 rack you have 3 failure groups – 1 for each storage cell. You do not need to have 2 failure groups for normal redundancy, but can have more than this. ASM will ensure under normal redundancy that each primary extent and secondary extent are stored on different failure groups.

      On your 1/4 rack ASM will put primary extents on ALL 3 failure groups and will put secondary extents on ALL 3 failure groups but fore EACH extent and normal redundancy there will only be ONE primary extent and ONE secondary extent.

      You do not need to attempt to move disks into different failure groups – this is not a good idea.

      jason.

      Reply
      • Bal Sharma

         /  May 29, 2012

        Thanks jason, for correcting me. It is a nice explanation. But I have a doubt- Initially my Exadata 1/4 RAC was configured with HIGH redundency for DATA, RECO and SYSTEMDG. So on a 3 storage cell I got somewhere around 4.5 TB space which is as per expectation.
        But now I want it to convert to NORMAL redundency for DATA and RECO so that I can get around 9 TB usable space. I tried to drop the ASM disk groups using ASMCA as I dropped all the databases before dropping the DATA and RECO(There are no production DB’s running on this as of now).
        But even after selecting NORMAL redundency I was not able to see expected disk space.

        I was just enquiring with few folks and heard that as initial allocation was with HIGH redundency – In order to change we should be creating all the disk groups after dropping and recreating from Storage cell. Or may be One Click installer will have certain script in order to do all these.

        It would be great if you can guide me in this situation.

        Thanks & Regards,

        bal

      • jarneil

         /  May 31, 2012

        Hi Bal,

        The redundancy level of the diskgroup is set when you create it. It is not a setting at the storage cell level, so you should not have to recreate anything on there.

        I wonder if you are looking at the wrong information to see how much space you have? You should look at free_mb from V$ASM_DISKGROUP. If you look at usable_file_mb then you will see a lot less than you expect as it shows how much space you can use and still rebalance AFTER the failure of an entire failure group.

        jason.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 58 other followers

%d bloggers like this: