Building an Exadata Compute Node USB imaging drive

I’ve been having a lot of fun recently with doing a bare metal restore of an Exadata compute node. This process is reasonably well documented in MOS: 1084360.1. However one area where it falls down on, is how to actually image the node.

So here is my guide to building an Exadata Compute Node USB imaging drive. This will use Virtualbox on OS X. Yes, you could build it on an existing Exadata node.

Obtaining Exadata Software

First thing you need to do is download the correct version you are after. Yes, you can grab it from Oracle edelivery, as long as the version you are after is not too old. There are choices for the various versions and separate downloads for compute nodes and storage nodes. You will download a zip file. When unzipped, assuming it is a compute node you are building you will find a computeImageMake_VERSION.tar.zip. You need to uncompress and then untar this as root

You will now have a dl360 directory. Just happens that dl360 is a model of HP server. If you are building a storage node, it will be dl180, also a type of HP Proliant server.

Descending to this newly created directory you will see a readme: README_FOR_FACTORY.txt, which explains how to build the image. You will also see makeImageMedia.sh which actually does the building of the image.

Building the Image

Using a Linux VM and VirtualBox. present the USB stick to the VM. You need at least 2GB in size.

[root@localhost ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                       14G   11G  2.3G  83% /
/dev/sda1              99M   13M   82M  14% /boot
tmpfs                 502M     0  502M   0% /dev/shm
Downloads             297G  210G   88G  71% /media/sf_Downloads
/dev/hdc               43M   43M     0 100% /media/VBOXADDITIONS_4.1.6_74713
/dev/sdb1             3.8G  888K  3.8G   1% /media/UNTITLED

Unmount the device:

[root@localhost ~]# umount /dev/sdb1

Use fdisk to remove the FAT32 formatting and create a partition:

[root@localhost ~]# fdisk /dev/sdb

Command (m for help): p

Disk /dev/sdb: 4041 MB, 4041211904 bytes
255 heads, 63 sectors/track, 491 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1         492     3946495    b  W95 FAT32
Partition 1 has different physical/logical beginnings (non-Linux?):
     phys=(1023, 254, 63) logical=(0, 0, 3)
Partition 1 has different physical/logical endings:
     phys=(1023, 254, 63) logical=(491, 80, 37)

Command (m for help): d
Selected partition 1

Command (m for help): p

Disk /dev/sdb: 4041 MB, 4041211904 bytes
255 heads, 63 sectors/track, 491 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-491, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-491, default 491):
Using default value 491

Command (m for help): p

Disk /dev/sdb: 4041 MB, 4041211904 bytes
255 heads, 63 sectors/track, 491 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1         491     3943926   83  Linux

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

This is the crucial part, the build will fail if it is formatted with FAT when you try and create the image.

Now you are ready to run the makeImageMedia.sh script:

[root@localhost dl360]# ./makeImageMedia.sh

Please wait. Calculating md5 checksums for cellbits ...
Please wait. Making initrd ...
180027 blocks
Please wait. Calculating md5 checksums for boot ...

Choose listed USB devices to set up the Oracle CELL installer

sdb   Approximate capacity 3946 MB
Enter the comma separated (no spaces) list of devices or word 'ALL' for to select all: ALL
sdb will be used as the Oracle CELL installer

All data on sdb will be erased. Proceed [y/n]? y

Command (m for help): Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): Command action
   e   extended
   p   primary partition (1-4)
Partition number (1-4): First cylinder (1-491, default 1): Last cylinder or +size or +sizeM or +sizeK (1-491, default 491):
Command (m for help): The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
umount2: Invalid argument
umount: /dev/sdb1: not mounted
mke2fs 1.39 (29-May-2006)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
492032 inodes, 983973 blocks
49198 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=1010827264
31 block groups
32768 blocks per group, 32768 fragments per group
15872 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736

Writing inode tables: done                           
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 25 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.
Copying files... will take several minutes


    GNU GRUB  version 0.97  (640K lower / 3072K upper memory)

 [ Minimal BASH-like line editing is supported.  For the first word, TAB
   lists possible command completions.  Anywhere else TAB lists the possible
   completions of a device/filename.]
grub> root (hd0,0)
 Filesystem type is ext2fs, partition type 0x83
grub> setup (hd0)
 Checking if "/boot/grub/stage1" exists... no
 Checking if "/grub/stage1" exists... yes
 Checking if "/grub/stage2" exists... yes
 Checking if "/grub/e2fs_stage1_5" exists... yes
 Running "embed /grub/e2fs_stage1_5 (hd0)"...  15 sectors are embedded.
succeeded
 Running "install /grub/stage1 (hd0) (hd0)1+15 p (hd0,0)/grub/stage2 /grub/grub.conf"... succeeded
Done.
grub> Done creation of installation USB for DL360

This will now have built your image on the USB drive. It is now a good idea to test that it really has worked and contains the software you require.

Testing your Image

There are a couple of ways of testing you have the image you want. Thanks to my e-dba colleague, Matthew Walden, for pointing these out.

First off, present again the USB drive to your VM.

On OS X when inserting an Exadata USB image, it is unrecognised by the OS and you see this dialogue box:

You need to choose eject. Then you can attach the USB drive to the VirtualBox VM. Once this is done, the following folder will appear:

You can look inside the image.id file to check what the image was built with:

You can clearly see this is a Compute node and is built with the 11.2.1.2.3 software version.

You can also test that your image will boot using VirtualBox and the plop bootloader. This allows you to select a USB device to boot from in VirtualBox. It downloads to an iso file which you boot a VM from. You then see the following menu:

After presenting your USB drive containing the Exadata image to the VM, you can select the USB device to boot from. Eventually you will see the following splash screen:

The above steps should give you some confidence that the USB image you have created will actually boot and Exadata node.

Note the above is only intended to for bare metal restore or reimaging of an Exadata node. In later software versions mounting an iso image via virtual cdrom on the iLOM works fine and would be my preferred solution.