Exadata ASM Disk Headers

This post is more for academic interest, but I have had a bit of a look at ASM disk headers on Exadata, under various conditions.

First up we see the disks of a newly racked Exadata that has not been configured, apart from some networking. None of the disks on the cell have been touched:

[root@cel01] od -c /dev/sdd --read-bytes 1056300 
0000000  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
* 
4017040  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
4017054

So it’s pretty much a blank slate to begin with.

Next I looked at the header after the celldisk and griddisks had been created:

[root@cel01 ~]# od -c /dev/sdd |head -100 
0000000                   o   r   a   c   l   e       s   a   g   e   d 
0000020   i   s   k  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
0000040  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
* 
0000100   .   !   o   R  \0  \0  \0 006  \0 002  \0  \0 004  \0  \0  \0 
0000120 001  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
0000140  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
* 
0001000   *   c   d   I   n   f   o   * 030   N 334 230   9   ) 230 210 
0001020 271   3 246 347   ^ 367   m 241 354 177   T 334  \0  \0  \0  \0 
0001040 005  \0  \0  \0 006  \0  \0  \0  \0 200  \0  \0  \0 200  \0  \0 
0001060  \0 200 266 350  \0  \0  \0  \0 005  \0  \0  \0  \0  \0  \0  \0 
0001100 001  \0  \0  \0 005  \0  \0  \0 342 177  \0  \0  \0  \0  \0  \0 
0001120 233 001  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
0001140  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
* 
0002000   C   D   _   0   3   _   c   e   l   0   1  \0  \0  \0  \0  \0
0002020  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
0002040  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
* 
0005000   S   M   s   c   t   r  \a  \0 345 256   u   c 001  \0  \0  \0 
0005020  \0  \0  \0  \0 001  \0  \0  \0  \0  \0  \0  \0 002  \0  \0  \0 
0005040  \0 005  \0  \0 001  \0  \0  \0  \0  \0  \0  \0 003  \0  \0  \0 
0005060 001  \0  \0  \0 001  \0  \0  \0  \0  \0  \0  \0 005  \0  \0  \0 
0005100   & 312 001  \0   G  \a  \0  \0  \0  \0  \0  \0 006  \0  \0  \0 
0005120 002  \0  \0  \0 376 004  \0  \0  \0  \0  \0  \0 006  \0  \0  \0 
0005140 001 005  \0  \0 202   i 001  \0 376 004  \0  \0  \a  \0  \0  \0 
0005160 203   n 001  \0 243   [  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
0005200  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
* 
77742000   G   D   t   a   b   l   e   ! 355   I 211   !   ' 314 334   u 
77742020 211   h 305 310 330 221   q 224 243   [  \0  \0 001  \0  \0  \0 
77742040  \a  \0  \0  \0 314 034 355 022  \0  \0  \0  \0  \0  \0  \0  \0 
77742060  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
77742100  \0  \0  \0  \0  \0  \0  \0  \0 222 001  \0  \0  \0  \0  \0  \0 
77742120  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
* 
77743000   R   E   C   O   0   1   _   C   D   _   0   3   _   c   e   l 
77743020   0   1  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
77743040  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
* 
77747000   G   D   t   a   b   l   e   !   Z   A   D 332   r 312 370 003 
77747020 351 027 270   E   p 034 215 227 200   n 001  \0 002  \0  \0  \0 
77747040 006  \0  \0  \0 321 321 223 037  \0  \0  \0  \0  \0  \0  \0  \0 
77747060  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
77747100  \0  \0  \0  \0  \0  \0  \0  \0 222 001  \0  \0  \0  \0  \0  \0 
77747120  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
* 
77750000   D   A   T   A   0   1   _   C   D   _   0   3   _   c   e   l 
77750020   0   1  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
77750040  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
* 
77754000   G   D   t   a   b   l   e   ! 363   O   K 231 317   ,   ] 251 
77754020 370 375 310 273 277 263 322 224   G  \a  \0  \0 001  \0  \0  \0 
77754040 005  \0  \0  \0 201   "   z   R  \0  \0  \0  \0  \0  \0  \0  \0 
77754060  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
77754100  \0  \0  \0  \0  \0  \0  \0  \0 222 001  \0  \0  \0  \0  \0  \0 
77754120  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
* 
77755000   D   B   F   S   _   D   G   _   C   D   _   0   3   _   c   e 
77755020   l   0   1  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
77755040  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
* 
77761000   G   D   t   a   b   l   e   !  \0  \0  \0  \0  \0  \0  \0  \0 
77761020  \0  \0  \0  \0  \0  \0  \0  \0 001  \0  \0  \0 001  \0  \0  \0 
77761040 001  \0  \0  \0   y   l 001   R  \0  \0  \0  \0  \0  \0  \0  \0 
77761060  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
77761100  \0  \0  \0  \0  \0  \0  \0  \0 001  \0  \0  \0  \0  \0  \0  \0 
77761120  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
* 
77762000   P   R   I   M   A   R   Y   _   M   D  \0  \0  \0  \0  \0  \0 
77762020  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
* 
77766000   G   D   t   a   b   l   e   !  \0  \0  \0  \0  \0  \0  \0  \0 
77766020  \0  \0  \0  \0  \0  \0  \0  \0 001  \0  \0  \0 001  \0  \0  \0 
77766040 002  \0  \0  \0   b   v   ^ 031  \0  \0  \0  \0  \0  \0  \0  \0 
77766060  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
77766100  \0  \0  \0  \0  \0  \0  \0  \0 001  \0  \0  \0  \0  \0  \0  \0 
77766120  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
* 
77767000   S   E   C   O   N   D   A   R   Y   _   M   D  \0  \0  \0  \0 
77767020  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
* 
77773000   G   D   t   a   b   l   e   !  \0  \0  \0  \0  \0  \0  \0  \0 
77773020  \0  \0  \0  \0  \0  \0  \0  \0 001  \0  \0  \0 001  \0  \0  \0 
77773040 003  \0  \0  \0   l   y   T   N  \0  \0  \0  \0  \0  \0  \0  \0 
77773060  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
77773100  \0  \0  \0  \0  \0  \0  \0  \0 001  \0  \0  \0  \0  \0  \0  \0 
77773120  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
* 
77774000   U   N   D   O   A   R   E   A   _   M   D  \0  \0  \0  \0  \0 
77774020  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
* 
100000000                   o   r   a   c   l   e       s   a   g   e   d 
100000020   i   s   k  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
100000040  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
* 
100000100   .   !   o   R  \0  \0  \0 006  \0 002  \0  \0 004  \0  \0  \0 
100000120 001  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
100000140  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0

First up you can see that this is labelled as an “oracle sagedisk” note the code name SAGE – “Storage Appliance for Grid Environments”

Next you can see the label for this particular celldisk – “C D _ 0 3 _ c e l 0 1″

Curious, at least to me is the next part – “S M s c t r” – something about SMart SCan”?

Finally you can see the Griddisks created on this celldisk, the DATA01, RECO01 and the DBFS_DG.

Last header dump I have is of a dropped celldisk:

[root@cel01 ~]# od -c /dev/sdd |head -100 
0000000   d   r   o   p   p   e   d       c   e   l   l   d   i   s   k 
0000020  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
* 
0000100 026 031 024   Q  \0  \0  \0 006  \0 002  \0  \0 004  \0  \0  \0 
0000120 001  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
0000140  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
* 
0001000   *   c   d   I   n   f   o   * 201   K  \v 036 362 326 340   G 
0001020 024 245   l 022 031   . 322 212   N 312 216   K  \0  \0  \0  \0 
0001040  \t  \0  \0  \0 003  \0  \0  \0  \0 200  \0  \0  \0 200  \0  \0 
0001060  \0 200 266 350  \0  \0  \0  \0 005  \0  \0  \0  \0  \0  \0  \0 
0001100 001  \0  \0  \0 005  \0  \0  \0 361 177  \0  \0  \0  \0  \0  \0 
0001120 233 001  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
0001140  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
* 
0002000   C   D   _   0   3   _   c   e   l   0   1  \0  \0  \0  \0  \0 
0002020  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
0002040  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
* 
0005000   S   M   s   c   t   r 003  \0   '   :   p   c 001  \0  \0  \0 
0005020  \0  \0  \0  \0 001  \0  \0  \0  \0  \0  \0  \0 002  \0  \0  \0 
0005040  \0 005  \0  \0 001  \0  \0  \0  \0  \0  \0  \0 003  \0  \0  \0 
0005060 001  \0  \0  \0 001  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
0005100  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
* 
77761000   G   D   t   a   b   l   e   !  \0  \0  \0  \0  \0  \0  \0  \0 
77761020  \0  \0  \0  \0  \0  \0  \0  \0 001  \0  \0  \0 001  \0  \0  \0 
77761040 001  \0  \0  \0   y   l 001   R  \0  \0  \0  \0  \0  \0  \0  \0 
77761060  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
77761100  \0  \0  \0  \0  \0  \0  \0  \0 001  \0  \0  \0  \0  \0  \0  \0 
77761120  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
* 
77762000   P   R   I   M   A   R   Y   _   M   D  \0  \0  \0  \0  \0  \0 
77762020  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
* 
77766000   G   D   t   a   b   l   e   !  \0  \0  \0  \0  \0  \0  \0  \0 
77766020  \0  \0  \0  \0  \0  \0  \0  \0 001  \0  \0  \0 001  \0  \0  \0 
77766040 002  \0  \0  \0   b   v   ^ 031  \0  \0  \0  \0  \0  \0  \0  \0 
77766060  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
77766100  \0  \0  \0  \0  \0  \0  \0  \0 001  \0  \0  \0  \0  \0  \0  \0 
77766120  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
* 
77767000   S   E   C   O   N   D   A   R   Y   _   M   D  \0  \0  \0  \0 
77767020  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
* 
77773000   G   D   t   a   b   l   e   !  \0  \0  \0  \0  \0  \0  \0  \0 
77773020  \0  \0  \0  \0  \0  \0  \0  \0 001  \0  \0  \0 001  \0  \0  \0 
77773040 003  \0  \0  \0   l   y   T   N  \0  \0  \0  \0  \0  \0  \0  \0 
77773060  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
77773100  \0  \0  \0  \0  \0  \0  \0  \0 001  \0  \0  \0  \0  \0  \0  \0 
77773120  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
* 
77774000   U   N   D   O   A   R   E   A   _   M   D  \0  \0  \0  \0  \0 
77774020  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
* 
100000000                   o   r   a   c   l   e       s   a   g   e   d 
100000020   i   s   k  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
100000040  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
* 
100000100   .   !   o   R  \0  \0  \0 006  \0 002  \0  \0 004  \0  \0  \0 
100000120 001  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
100000140  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
* 
100001000   *   c   d   I   n   f   o   * 201   K  \v 036 362 326 340   G 
100001020 024 245   l 022 031   . 322 212   O 312 216   K  \0  \0  \0  \0 
100001040  \b  \0  \0  \0 003  \0  \0  \0  \0 200  \0  \0  \0 200  \0  \0 
100001060  \0 200 266 350  \0  \0  \0  \0 005  \0  \0  \0  \0  \0  \0  \0 
100001100 001  \0  \0  \0 005  \0  \0  \0 361 177  \0  \0  \0  \0  \0  \0 
100001120 233 001  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
100001140  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
* 
100002000   C   D   _   0   3   _   c   e   l   0   1  \0  \0  \0  \0  \0 
100002020  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
100002040  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
* 
100005000   S   M   s   c   t   r 003  \0   '   :   p   c 001  \0  \0  \0 
100005020  \0  \0  \0  \0 001  \0  \0  \0  \0  \0  \0  \0 002  \0  \0  \0 
100005040  \0 005  \0  \0 001  \0  \0  \0  \0  \0  \0  \0 003  \0  \0  \0 
100005060 001  \0  \0  \0 001  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
100005100  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
* 
177742000   G   D   t   a   b   l   e   ! 274   A   F 274 206 211 030   % 
177742020 347 035 221 252   R   : 032 207 243   [  \0  \0 001  \0  \0  \0 
177742040  \a  \0  \0  \0 330 217 331 256  \0  \0  \0  \0  \0  \0  \0  \0 
177742060  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
177742100  \0  \0  \0  \0  \0  \0  \0  \0 222 001  \0  \0  \0  \0  \0  \0 
177742120  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
* 
177743000   R   E   C   O   0   1   _   C   D   _   0   3   _   c   e   l 
177743020   0   1  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
177743040  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
* 
177747000   G   D   t   a   b   l   e   ! 027   I 320 224 210   T 033 210 
177747020   5   \ 353   g 035   o 371 253 200   n 001  \0 002  \0  \0  \0 
177747040 006  \0  \0  \0 327 177 303 304  \0  \0  \0  \0  \0  \0  \0  \0 
177747060  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
177747100  \0  \0  \0  \0  \0  \0  \0  \0 222 001  \0  \0  \0  \0  \0  \0 
177747120  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 
* 
177750000   D   A   T   A   0   1   _   C   D   _   0   3   _   c   e   l

You can see straight away that the disk gets marked with the “d r o p p e d c e l l d i s k” But with a simple drop the rest of information on the drive is still in place. If you really want to clean your disks you can always try the following:

cellcli> drop celldisk all ERASE=1pass

This should clear up your disks quite nicely. Goes without saying, don’t try this in production!

About these ads
Leave a comment

4 Comments

  1. Great info, especially for those like me which have no access to an Exadata.
    Can you please provide infos how kfed sees these disks? Especially the “S M s c t r” might be useful for other things ;-)

    Reply
  2. Martin,

    On Exadata, there’s a file (/etc/oracle/cell/network-config/cellip.ora) that contains a list of the Infiniband IPs of the Exadata storage servers. If your ASM_DISKSTRING parameter includes “o/*/*” then ASM will check these IPs for any griddisks that are available.

    As for the “S M s c t r” in the headers, it could be that is how ASM knows that the disk is ok to enable the cell.smart_scan_capable attribute for the diskgroup.

    Reply
    • Andy, thank you for the infos.
      Unfortunately I was somehow unclear in my question.
      I’m interested in a simple
      kfed read /dev/sdd
      probably this “S M s c t r” is part of any kfdhdb.* information.
      I’d like to see if I can write this to ‘ordinary’ (non Exadata) disks and if this changes the instances behavior.

      Reply
      • jarneil

         /  February 23, 2012

        Will do a follow up very shortly – though I don’t think it shows anything particularly interesting I’m afraid!

        jason.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 52 other followers

%d bloggers like this: