Kfed and Exadata ASM disks

I’ve written in the past on the usefulness of kfed. As Martin Berger requested seeing some output from kfed with Exadata disks I thought I would oblige.

So What is kfed

kfed is the so called Kernel Files Editor, Miladin Modrakovic has written quite nicely about this. It can be used to read and modify ASM disk headers. The ASM Support guy, Bane Radulović also has a nice write up.

Before you can run kfed you need to have disk to point it to, and this is where it gets interesting on Exadata as compared to traditional, say Fibre Channel SAN attached storage. On there you typically have devices like /dev/sdX that are your luns from the SAN.

Exadata is Different

I think we are all understanding that things are a little different on Exadata and the way disks are presented is certainly unusual. Exadata uses a protocol called iDB to communicate to the Storage Servers. It is a network protocol.

This is one of the first things that really surprised me in working with Exadata. I was so used to running iostat on a database server to see how busy the storage was. Well you can forget doing that on Exadata, as it will only show you the local compute node disks – not really where the action is!

kfod Disk Discovery

So to run kfed, we need to find something to run it against, and here kfod (Kernel Files Oracle Storage Manager Discovery Tool) is your friend:

db01: oracle$ kfod di=all 
-------------------------------------------------------------------------------- 
 Disk          Size Path                                     User     Group 
================================================================================ 
   1:    1501184 Mb o/192.168.10.3/DATA01_CD_00_cel01 <unknown> <unknown> 
   2:    1501184 Mb o/192.168.10.3/DATA01_CD_01_cel01 <unknown> <unknown> 
   3:    1501184 Mb o/192.168.10.3/DATA01_CD_02_cel01 <unknown> <unknown> 
   4:    1501184 Mb o/192.168.10.3/DATA01_CD_03_cel01 <unknown> <unknown> 
. 
. 
. 
-------------------------------------------------------------------------------- 
ORACLE_SID ORACLE_HOME 
================================================================================ 
     +ASM1 /u01/app/ora/product/11.2.0.2/grid_1 
     +ASM2 /u01/app/ora/product/11.2.0.2/grid_1

Output above has been edited to prevent tedium. This is basically scanning for valid devices and spitting out the size and path to those devices. So now you can feed something like o/192.168.10.3/DATA01_CD_03_cel01 into kfed:

db01: oracle$ kfed read o/192.168.10.3/DATA01_CD_03_cel01 
kfbh.endian:                          1 ; 0x000: 0x01 
kfbh.hard:                          130 ; 0x001: 0x82 
kfbh.type:                            1 ; 0x002: KFBTYP_DISKHEAD 
kfbh.datfmt:                          1 ; 0x003: 0x01 
kfbh.block.blk:                       0 ; 0x004: T=0 NUMB=0x0 
kfbh.block.obj:              2147483651 ; 0x008: TYPE=0x8 NUMB=0x3 
kfbh.check:                   828339576 ; 0x00c: 0x315f7578 
kfbh.fcn.base:                        0 ; 0x010: 0x00000000 
kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000 
kfbh.spare1:                          0 ; 0x018: 0x00000000 
kfbh.spare2:                          0 ; 0x01c: 0x00000000 
kfdhdb.driver.provstr:         ORCLDISK ; 0x000: length=8 
kfdhdb.driver.reserved[0]:            0 ; 0x008: 0x00000000 
kfdhdb.driver.reserved[1]:            0 ; 0x00c: 0x00000000 
kfdhdb.driver.reserved[2]:            0 ; 0x010: 0x00000000 
kfdhdb.driver.reserved[3]:            0 ; 0x014: 0x00000000 
kfdhdb.driver.reserved[4]:            0 ; 0x018: 0x00000000 
kfdhdb.driver.reserved[5]:            0 ; 0x01c: 0x00000000 
kfdhdb.compat:                186646528 ; 0x020: 0x0b200000 
kfdhdb.dsknum:                        3 ; 0x024: 0x0003 
kfdhdb.grptyp:                        2 ; 0x026: KFDGTP_NORMAL 
kfdhdb.hdrsts:                        3 ; 0x027: KFDHDR_MEMBER 
kfdhdb.dskname:      DATA01_CD_03_CEL01 ; 0x028: length=18 
kfdhdb.grpname:                  DATA01 ; 0x048: length=6 
kfdhdb.fgname:                    CEL01 ; 0x068: length=4 
kfdhdb.capname:                         ; 0x088: length=0 
kfdhdb.crestmp.hi:             32965963 ; 0x0a8: HOUR=0xb DAYS=0xa MNTH=0x1 YEAR=0x7dc 
kfdhdb.crestmp.lo:            180889600 ; 0x0ac: USEC=0x0 MSEC=0x20a SECS=0x2c MINS=0x2 
kfdhdb.mntstmp.hi:             32967149 ; 0x0b0: HOUR=0xd DAYS=0xf MNTH=0x2 YEAR=0x7dc 
kfdhdb.mntstmp.lo:           2947617792 ; 0x0b4: USEC=0x0 MSEC=0x45 SECS=0x3b MINS=0x2b 
kfdhdb.secsize:                     512 ; 0x0b8: 0x0200 
kfdhdb.blksize:                    4096 ; 0x0ba: 0x1000 
kfdhdb.ausize:                  4194304 ; 0x0bc: 0x00400000 
kfdhdb.mfact:                    454272 ; 0x0c0: 0x0006ee80 
kfdhdb.dsksize:                  375296 ; 0x0c4: 0x0005ba00 
kfdhdb.pmcnt:                         2 ; 0x0c8: 0x00000002 
kfdhdb.fstlocn:                       1 ; 0x0cc: 0x00000001 
kfdhdb.altlocn:                       2 ; 0x0d0: 0x00000002 
kfdhdb.f1b1locn:                      0 ; 0x0d4: 0x00000000 
kfdhdb.redomirrors[0]:                0 ; 0x0d8: 0x0000 
kfdhdb.redomirrors[1]:                0 ; 0x0da: 0x0000 
kfdhdb.redomirrors[2]:                0 ; 0x0dc: 0x0000 
kfdhdb.redomirrors[3]:                0 ; 0x0de: 0x0000 
kfdhdb.dbcompat:              186646528 ; 0x0e0: 0x0b200000 
kfdhdb.grpstmp.hi:             32965963 ; 0x0e4: HOUR=0xb DAYS=0xa MNTH=0x1 YEAR=0x7dc 
kfdhdb.grpstmp.lo:            177854464 ; 0x0e8: USEC=0x0 MSEC=0x276 SECS=0x29 MINS=0x2 
kfdhdb.vfstart:                       0 ; 0x0ec: 0x00000000 
kfdhdb.vfend:                         0 ; 0x0f0: 0x00000000 
kfdhdb.spfile:                        0 ; 0x0f4: 0x00000000 
kfdhdb.spfflg:                        0 ; 0x0f8: 0x00000000 
kfdhdb.ub4spare[0]:                   0 ; 0x0fc: 0x00000000 
kfdhdb.ub4spare[1]:                   0 ; 0x100: 0x00000000 
kfdhdb.ub4spare[2]:                   0 ; 0x104: 0x00000000 
kfdhdb.ub4spare[3]:                   0 ; 0x108: 0x00000000 
kfdhdb.ub4spare[4]:                   0 ; 0x10c: 0x00000000 
kfdhdb.ub4spare[5]:                   0 ; 0x110: 0x00000000 
kfdhdb.ub4spare[6]:                   0 ; 0x114: 0x00000000 
kfdhdb.ub4spare[7]:                   0 ; 0x118: 0x00000000 
kfdhdb.ub4spare[8]:                   0 ; 0x11c: 0x00000000 
kfdhdb.ub4spare[9]:                   0 ; 0x120: 0x00000000 
kfdhdb.ub4spare[10]:                  0 ; 0x124: 0x00000000 
kfdhdb.ub4spare[11]:                  0 ; 0x128: 0x00000000 
kfdhdb.ub4spare[12]:                  0 ; 0x12c: 0x00000000 
kfdhdb.ub4spare[13]:                  0 ; 0x130: 0x00000000 
kfdhdb.ub4spare[14]:                  0 ; 0x134: 0x00000000 
kfdhdb.ub4spare[15]:                  0 ; 0x138: 0x00000000 
kfdhdb.ub4spare[16]:                  0 ; 0x13c: 0x00000000 
kfdhdb.ub4spare[17]:                  0 ; 0x140: 0x00000000 
kfdhdb.ub4spare[18]:                  0 ; 0x144: 0x00000000 
kfdhdb.ub4spare[19]:                  0 ; 0x148: 0x00000000 
kfdhdb.ub4spare[20]:                  0 ; 0x14c: 0x00000000 
kfdhdb.ub4spare[21]:                  0 ; 0x150: 0x00000000 
kfdhdb.ub4spare[22]:                  0 ; 0x154: 0x00000000 
kfdhdb.ub4spare[23]:                  0 ; 0x158: 0x00000000 
kfdhdb.ub4spare[24]:                  0 ; 0x15c: 0x00000000 
kfdhdb.ub4spare[25]:                  0 ; 0x160: 0x00000000 
kfdhdb.ub4spare[26]:                  0 ; 0x164: 0x00000000 
kfdhdb.ub4spare[27]:                  0 ; 0x168: 0x00000000 
kfdhdb.ub4spare[28]:                  0 ; 0x16c: 0x00000000 
kfdhdb.ub4spare[29]:                  0 ; 0x170: 0x00000000 
kfdhdb.ub4spare[30]:                  0 ; 0x174: 0x00000000 
kfdhdb.ub4spare[31]:                  0 ; 0x178: 0x00000000 
kfdhdb.ub4spare[32]:                  0 ; 0x17c: 0x00000000 
kfdhdb.ub4spare[33]:                  0 ; 0x180: 0x00000000 
kfdhdb.ub4spare[34]:                  0 ; 0x184: 0x00000000 
kfdhdb.ub4spare[35]:                  0 ; 0x188: 0x00000000 
kfdhdb.ub4spare[36]:                  0 ; 0x18c: 0x00000000 
kfdhdb.ub4spare[37]:                  0 ; 0x190: 0x00000000 
kfdhdb.ub4spare[38]:                  0 ; 0x194: 0x00000000 
kfdhdb.ub4spare[39]:                  0 ; 0x198: 0x00000000 
kfdhdb.ub4spare[40]:                  0 ; 0x19c: 0x00000000 
kfdhdb.ub4spare[41]:                  0 ; 0x1a0: 0x00000000 
kfdhdb.ub4spare[42]:                  0 ; 0x1a4: 0x00000000 
kfdhdb.ub4spare[43]:                  0 ; 0x1a8: 0x00000000 
kfdhdb.ub4spare[44]:                  0 ; 0x1ac: 0x00000000 
kfdhdb.ub4spare[45]:                  0 ; 0x1b0: 0x00000000 
kfdhdb.ub4spare[46]:                  0 ; 0x1b4: 0x00000000 
kfdhdb.ub4spare[47]:                  0 ; 0x1b8: 0x00000000 
kfdhdb.ub4spare[48]:                  0 ; 0x1bc: 0x00000000 
kfdhdb.ub4spare[49]:                  0 ; 0x1c0: 0x00000000 
kfdhdb.ub4spare[50]:                  0 ; 0x1c4: 0x00000000 
kfdhdb.ub4spare[51]:                  0 ; 0x1c8: 0x00000000 
kfdhdb.ub4spare[52]:                  0 ; 0x1cc: 0x00000000 
kfdhdb.ub4spare[53]:                  0 ; 0x1d0: 0x00000000 
kfdhdb.acdb.aba.seq:                  0 ; 0x1d4: 0x00000000 
kfdhdb.acdb.aba.blk:                  0 ; 0x1d8: 0x00000000 
kfdhdb.acdb.ents:                     0 ; 0x1dc: 0x0000 
kfdhdb.acdb.ub2spare:                 0 ; 0x1de: 0x0000

Lots of spares! It’s worth noting that kfed can also be used for editing your header if it gets corrupted.

About these ads
Leave a comment

2 Comments

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 51 other followers

%d bloggers like this: