I’ve written in the past on the usefulness of kfed. As Martin Berger requested seeing some output from kfed with Exadata disks I thought I would oblige.
So What is kfed
kfed is the so called Kernel Files Editor, Miladin Modrakovic has written quite nicely about this. It can be used to read and modify ASM disk headers. The ASM Support guy, Bane Radulović also has a nice write up.
Before you can run kfed you need to have disk to point it to, and this is where it gets interesting on Exadata as compared to traditional, say Fibre Channel SAN attached storage. On there you typically have devices like /dev/sdX that are your luns from the SAN.
Exadata is Different
I think we are all understanding that things are a little different on Exadata and the way disks are presented is certainly unusual. Exadata uses a protocol called iDB to communicate to the Storage Servers. It is a network protocol.
This is one of the first things that really surprised me in working with Exadata. I was so used to running iostat on a database server to see how busy the storage was. Well you can forget doing that on Exadata, as it will only show you the local compute node disks – not really where the action is!
kfod Disk Discovery
So to run kfed, we need to find something to run it against, and here kfod (Kernel Files Oracle Storage Manager Discovery Tool) is your friend:
db01: oracle$ kfod di=all
--------------------------------------------------------------------------------
Disk Size Path User Group
================================================================================
1: 1501184 Mb o/192.168.10.3/DATA01_CD_00_cel01 <unknown> <unknown>
2: 1501184 Mb o/192.168.10.3/DATA01_CD_01_cel01 <unknown> <unknown>
3: 1501184 Mb o/192.168.10.3/DATA01_CD_02_cel01 <unknown> <unknown>
4: 1501184 Mb o/192.168.10.3/DATA01_CD_03_cel01 <unknown> <unknown>
.
.
.
--------------------------------------------------------------------------------
ORACLE_SID ORACLE_HOME
================================================================================
+ASM1 /u01/app/ora/product/11.2.0.2/grid_1
+ASM2 /u01/app/ora/product/11.2.0.2/grid_1
Output above has been edited to prevent tedium. This is basically scanning for valid devices and spitting out the size and path to those devices. So now you can feed something like o/192.168.10.3/DATA01_CD_03_cel01 into kfed:
db01: oracle$ kfed read o/192.168.10.3/DATA01_CD_03_cel01 kfbh.endian: 1 ; 0x000: 0x01 kfbh.hard: 130 ; 0x001: 0x82 kfbh.type: 1 ; 0x002: KFBTYP_DISKHEAD kfbh.datfmt: 1 ; 0x003: 0x01 kfbh.block.blk: 0 ; 0x004: T=0 NUMB=0x0 kfbh.block.obj: 2147483651 ; 0x008: TYPE=0x8 NUMB=0x3 kfbh.check: 828339576 ; 0x00c: 0x315f7578 kfbh.fcn.base: 0 ; 0x010: 0x00000000 kfbh.fcn.wrap: 0 ; 0x014: 0x00000000 kfbh.spare1: 0 ; 0x018: 0x00000000 kfbh.spare2: 0 ; 0x01c: 0x00000000 kfdhdb.driver.provstr: ORCLDISK ; 0x000: length=8 kfdhdb.driver.reserved[0]: 0 ; 0x008: 0x00000000 kfdhdb.driver.reserved[1]: 0 ; 0x00c: 0x00000000 kfdhdb.driver.reserved[2]: 0 ; 0x010: 0x00000000 kfdhdb.driver.reserved[3]: 0 ; 0x014: 0x00000000 kfdhdb.driver.reserved[4]: 0 ; 0x018: 0x00000000 kfdhdb.driver.reserved[5]: 0 ; 0x01c: 0x00000000 kfdhdb.compat: 186646528 ; 0x020: 0x0b200000 kfdhdb.dsknum: 3 ; 0x024: 0x0003 kfdhdb.grptyp: 2 ; 0x026: KFDGTP_NORMAL kfdhdb.hdrsts: 3 ; 0x027: KFDHDR_MEMBER kfdhdb.dskname: DATA01_CD_03_CEL01 ; 0x028: length=18 kfdhdb.grpname: DATA01 ; 0x048: length=6 kfdhdb.fgname: CEL01 ; 0x068: length=4 kfdhdb.capname: ; 0x088: length=0 kfdhdb.crestmp.hi: 32965963 ; 0x0a8: HOUR=0xb DAYS=0xa MNTH=0x1 YEAR=0x7dc kfdhdb.crestmp.lo: 180889600 ; 0x0ac: USEC=0x0 MSEC=0x20a SECS=0x2c MINS=0x2 kfdhdb.mntstmp.hi: 32967149 ; 0x0b0: HOUR=0xd DAYS=0xf MNTH=0x2 YEAR=0x7dc kfdhdb.mntstmp.lo: 2947617792 ; 0x0b4: USEC=0x0 MSEC=0x45 SECS=0x3b MINS=0x2b kfdhdb.secsize: 512 ; 0x0b8: 0x0200 kfdhdb.blksize: 4096 ; 0x0ba: 0x1000 kfdhdb.ausize: 4194304 ; 0x0bc: 0x00400000 kfdhdb.mfact: 454272 ; 0x0c0: 0x0006ee80 kfdhdb.dsksize: 375296 ; 0x0c4: 0x0005ba00 kfdhdb.pmcnt: 2 ; 0x0c8: 0x00000002 kfdhdb.fstlocn: 1 ; 0x0cc: 0x00000001 kfdhdb.altlocn: 2 ; 0x0d0: 0x00000002 kfdhdb.f1b1locn: 0 ; 0x0d4: 0x00000000 kfdhdb.redomirrors[0]: 0 ; 0x0d8: 0x0000 kfdhdb.redomirrors[1]: 0 ; 0x0da: 0x0000 kfdhdb.redomirrors[2]: 0 ; 0x0dc: 0x0000 kfdhdb.redomirrors[3]: 0 ; 0x0de: 0x0000 kfdhdb.dbcompat: 186646528 ; 0x0e0: 0x0b200000 kfdhdb.grpstmp.hi: 32965963 ; 0x0e4: HOUR=0xb DAYS=0xa MNTH=0x1 YEAR=0x7dc kfdhdb.grpstmp.lo: 177854464 ; 0x0e8: USEC=0x0 MSEC=0x276 SECS=0x29 MINS=0x2 kfdhdb.vfstart: 0 ; 0x0ec: 0x00000000 kfdhdb.vfend: 0 ; 0x0f0: 0x00000000 kfdhdb.spfile: 0 ; 0x0f4: 0x00000000 kfdhdb.spfflg: 0 ; 0x0f8: 0x00000000 kfdhdb.ub4spare[0]: 0 ; 0x0fc: 0x00000000 kfdhdb.ub4spare[1]: 0 ; 0x100: 0x00000000 kfdhdb.ub4spare[2]: 0 ; 0x104: 0x00000000 kfdhdb.ub4spare[3]: 0 ; 0x108: 0x00000000 kfdhdb.ub4spare[4]: 0 ; 0x10c: 0x00000000 kfdhdb.ub4spare[5]: 0 ; 0x110: 0x00000000 kfdhdb.ub4spare[6]: 0 ; 0x114: 0x00000000 kfdhdb.ub4spare[7]: 0 ; 0x118: 0x00000000 kfdhdb.ub4spare[8]: 0 ; 0x11c: 0x00000000 kfdhdb.ub4spare[9]: 0 ; 0x120: 0x00000000 kfdhdb.ub4spare[10]: 0 ; 0x124: 0x00000000 kfdhdb.ub4spare[11]: 0 ; 0x128: 0x00000000 kfdhdb.ub4spare[12]: 0 ; 0x12c: 0x00000000 kfdhdb.ub4spare[13]: 0 ; 0x130: 0x00000000 kfdhdb.ub4spare[14]: 0 ; 0x134: 0x00000000 kfdhdb.ub4spare[15]: 0 ; 0x138: 0x00000000 kfdhdb.ub4spare[16]: 0 ; 0x13c: 0x00000000 kfdhdb.ub4spare[17]: 0 ; 0x140: 0x00000000 kfdhdb.ub4spare[18]: 0 ; 0x144: 0x00000000 kfdhdb.ub4spare[19]: 0 ; 0x148: 0x00000000 kfdhdb.ub4spare[20]: 0 ; 0x14c: 0x00000000 kfdhdb.ub4spare[21]: 0 ; 0x150: 0x00000000 kfdhdb.ub4spare[22]: 0 ; 0x154: 0x00000000 kfdhdb.ub4spare[23]: 0 ; 0x158: 0x00000000 kfdhdb.ub4spare[24]: 0 ; 0x15c: 0x00000000 kfdhdb.ub4spare[25]: 0 ; 0x160: 0x00000000 kfdhdb.ub4spare[26]: 0 ; 0x164: 0x00000000 kfdhdb.ub4spare[27]: 0 ; 0x168: 0x00000000 kfdhdb.ub4spare[28]: 0 ; 0x16c: 0x00000000 kfdhdb.ub4spare[29]: 0 ; 0x170: 0x00000000 kfdhdb.ub4spare[30]: 0 ; 0x174: 0x00000000 kfdhdb.ub4spare[31]: 0 ; 0x178: 0x00000000 kfdhdb.ub4spare[32]: 0 ; 0x17c: 0x00000000 kfdhdb.ub4spare[33]: 0 ; 0x180: 0x00000000 kfdhdb.ub4spare[34]: 0 ; 0x184: 0x00000000 kfdhdb.ub4spare[35]: 0 ; 0x188: 0x00000000 kfdhdb.ub4spare[36]: 0 ; 0x18c: 0x00000000 kfdhdb.ub4spare[37]: 0 ; 0x190: 0x00000000 kfdhdb.ub4spare[38]: 0 ; 0x194: 0x00000000 kfdhdb.ub4spare[39]: 0 ; 0x198: 0x00000000 kfdhdb.ub4spare[40]: 0 ; 0x19c: 0x00000000 kfdhdb.ub4spare[41]: 0 ; 0x1a0: 0x00000000 kfdhdb.ub4spare[42]: 0 ; 0x1a4: 0x00000000 kfdhdb.ub4spare[43]: 0 ; 0x1a8: 0x00000000 kfdhdb.ub4spare[44]: 0 ; 0x1ac: 0x00000000 kfdhdb.ub4spare[45]: 0 ; 0x1b0: 0x00000000 kfdhdb.ub4spare[46]: 0 ; 0x1b4: 0x00000000 kfdhdb.ub4spare[47]: 0 ; 0x1b8: 0x00000000 kfdhdb.ub4spare[48]: 0 ; 0x1bc: 0x00000000 kfdhdb.ub4spare[49]: 0 ; 0x1c0: 0x00000000 kfdhdb.ub4spare[50]: 0 ; 0x1c4: 0x00000000 kfdhdb.ub4spare[51]: 0 ; 0x1c8: 0x00000000 kfdhdb.ub4spare[52]: 0 ; 0x1cc: 0x00000000 kfdhdb.ub4spare[53]: 0 ; 0x1d0: 0x00000000 kfdhdb.acdb.aba.seq: 0 ; 0x1d4: 0x00000000 kfdhdb.acdb.aba.blk: 0 ; 0x1d8: 0x00000000 kfdhdb.acdb.ents: 0 ; 0x1dc: 0x0000 kfdhdb.acdb.ub2spare: 0 ; 0x1de: 0x0000
Lots of spares! It’s worth noting that kfed can also be used for editing your header if it gets corrupted.
kevinclosson
/ February 23, 2012Hi Jason,
I’d like to leave references to some fundamental Exadata FAQ posts…kfod is mentioned in one so it might be of interest to your readers…if you’ll allow:
http://kevinclosson.wordpress.com/2008/09/26/oracle-exadata-storage-server-part-iii-a-faq-is-born/
http://kevinclosson.wordpress.com/2008/09/26/oracle-exadata-storage-server-frequently-asked-questions-part-ii/
http://kevinclosson.wordpress.com/2008/10/01/oracle-exadata-storage-server-frequently-asked-questions-part-iii/
jarneil
/ February 23, 2012Kevin,
Gladly, Sir!
Jason.