Upgrading to Oracle 11g Clusterware

I have just done a couple of 10g to 11g Oracle Clusterware upgrades on a pair of 2 node RAC clusters. These are now happily running 11g Clusterware with 10g ASM and database instances.

First off, I have found the documentation a little bit on the sparse side in terms on how to actually do a clusterware upgrade. It took a little while for me to realise that it is very possible to perform a rolling upgrade when upgrading your clusterware, know I knew this was possible when patching from 10.2.0.X to 10.2.0.Y but it took a little longer for me to understand that this can be done when going up to 11g.

The best place in the online documentation for information about this is Appendix B of the Oracle Clusterware Installation Guide. Another useful thing to look at is metalink note 338706.1 which tells you about the prerequisites you need to fulfill before you can upgrade your clusterware to 11g. Of course it is only with hindsight that I have seen the information there in the Clusterware Installation Guide. Here is what I did to upgrade, you are far better of, unlike myself, running the preupdate.sh script as recommended – but hey this what testing is all about ;-)

From the unziped clusterware directory run the cluster verification utility to check your system is ready to upgrade:

runcluvfy.sh stage -pre crsinst -n node1,node2 -verbose

make sure you upgrade any rpm’s needing changed.

Bring down the database and ASM instances on the first node you want to upgrade and then stop crs:

/opt/oracle/product/crs/bin/crsctl stop crs

If you run the preupdate.sh script that is in the clusterware/upgrade directory you don’t need to stop crs yourself or indeed perform the next step in changing permissions of the crs directory as it’s taken care for you.

The permissions on my crs directory were incorrect and the directory was owned by root. I changed them with:

chown -R oracle:oinstall crs/

run the installer and it will detect your CRS_HOME and offer to upgrade it, you want to make sure that on the Specify Hardware Cluster Installation Mode screen you select just the node you want to upgrade, assuming you are doing it rolling:

clusterware install

Once the upgrade has done it’s thing you are prompted to run the rootupgrade script:

[root@linuxrac2 install]# ./rootupgrade
Checking to see if Oracle CRS stack is already up...


copying ONS config file to 11.1 CRS home
/bin/cp: `/opt/oracle/product/crs/opmn/conf/ons.config' and `/opt/oracle/product/crs/opmn/conf/ons.config' are the same file
/opt/oracle/product/crs/opmn/conf/ons.config was copied successfully to /opt/oracle/product/crs/opmn/conf/ons.config
WARNING: directory '/opt/oracle/product' is not owned by root
Oracle Cluster Registry configuration upgraded successfully
Adding daemons to inittab

Attempting to start CRS stack
The CRS stack will be started shortly
Oracle CRS stack has failed to start. Check the file /var/adm/messages or the crsd, cssdd, and evmd logs in
/opt/oracle/product/crs/log/linuxrac2 directory for more details

You don’t need to worry when it says CRS stack has failed to start, because after a few moments CRS is running happily! Your database and/or ASM isntance will now be automatically restarted as well.

It is also worth pointing out that the active CRS version only becomes the 11.1.0.6.0 version after all nodes are upgraded:

[root@linuxrac2 crsd]# crsctl query crs softwareversion
CRS software version on node linuxrac2 is 11.1.0.6.0
[root@linuxrac2 crsd]# crsctl query crs activeversion
CRS active version on the cluster is 10.2.0.3.0

You now basically proceed to perform the same on the other nodes in your cluster, and there you have it, a rolling clusterware upgrade from 10g to 11g. I was actually well impressed with how smooth and painless the upgrade was and there really were no brown trouser moments.

It remains to be seen how stable the new 11g clusterware is but I’m sure it’s just a coincidence that about 12 hours after the upgrade one of the nodes on one of the clusters had a kernel panic and froze!

About these ads
Leave a comment

12 Comments

  1. Ravi Kosaraju

     /  February 7, 2008

    Hi Jason

    Thanks much for the tips. I was struggling with the Clusterware installation on a single node and was getting “file not found” errors. Your suggestion of setting the permissions for the CRS_HOME directory were exactly what I needed to get the installation/upgrade going.

    I do have an issue with rolling upgrades for the database though. The checkboxes for the Node Name selection are all grayed out and its forcing me to select all the nodes on the cluster to be upgraded. Is there any reason why it won’t let me perform a rolling upgrade?

    Thanks
    – Ravi

    Reply
  2. jarneil

     /  February 7, 2008

    Hi Ravi!

    Glad to be of help!

    You say you have a problem with rolling upgrades for the database, if you really mean the RDBMS, well you can’t do a 10 -> 11 rolling upgrade for that.

    jason.

    Reply
  3. jarneil

     /  February 8, 2008

    Of course I should have said that when installing the RDBMS software, you should get the option of installing it on the local node or doing a cluster installation with checkboxes for each node. I’m literally installing 11g rdbms sofware onto a 10g RAC cluster now, and I have the option to tick the boxes.

    It sounds as though it has not detected your clusterware – though if it’s offering you the ability to install on all nodes that is puzzling,

    Reply
  4. Adrian

     /  March 16, 2008

    Useful tips, thanks.
    I wonder what’s the case with the licenses.
    What licenses one needs to run Oracle 10G database with Oracle 11g Clusterware and Oracle 11G ASM instances.

    I was not able to find it in the docs or on the web, so I will have to ask Oracle maybe.

    Do you know how it is?

    Thanks in advance

    Reply
  5. jarneil

     /  March 17, 2008

    Hi Adrian,

    Thanks for dropping by. I would say that as far as I know, as long as you have a valid support contract then you can upgrade your clusterware with your normal license that you have for your 10g RDBMS instance. Oracle don’t particularly care, as long as you have the support contract to enable you to upgrade to the new verson.

    cheers,

    jason.

    Reply
  6. Hi Jason,

    I feel (have felt) your pain. I too have gone through what you went through. Oracle’s documentation on upgrading from 10g RAC to 11g RAC is as usual very confusing. Oracle seems to have covered every piece you need to upgrade a 10g RAC/CRS/ASM system but these instructions are spread out through 7 different upgrade “guides” and often hidden in appendix’s as you have pointed out.

    Like yourself I took a test 2 node 10g cluster and just started playing around with different upgrade paths, restoring the cluster after every “try”.

    I eventually put together a 37 page step by step guide broken into four sections:

    Section 1 – Ugrade the Clusterware
    Section 2 – Installing the RDBMS
    Section 3 – Upgrade the ASM & LISTENER
    Section 4 – Upgrade Your Database(s)

    I then found a good book on the internet written by Sam Alapat & Charles Kim which confirmed your blog findings and the way I concluded was the “best practices” way to upgrade my 10g clusters.

    I just wish Oracle would have put out a step by step “recommended” upgrade guide for RAC/CRS/ASM clusters.

    I can email you the guide if you want to make it available here but you seemed to cover most of what I already discovered.

    Reply
    • Gene

       /  August 7, 2009

      Hi Gary,
      Looks like you have what I was looking on one place. I will check the book too, but I would appreciate if can e-mail to me your guide document.
      Thanks,
      Gene

      Reply
    • Suresh

       /  August 20, 2009

      Neil/Gary/Jason,
      Could you publish a step-by-step procedure for upgrading 2 node 10g cluster from 10.2.0.3 to 10.2.0.4 please. Or alternately send it to my email id please.

      Regards
      Suresh

      Reply
  7. Jay Jayarajan

     /  December 3, 2010

    I am in the process of reviewing documents related to upgarde of Oracle10g RAC environment to Oracle11g RAC environment which involves upgrade of CRS, ASM and RDBMS and database.

    Is it possible to get a copy of the step-by-step upgrade documentation you have mentioned above.

    Many Thanks,

    Jay Jayarajan

    Reply
  8. jarneil

     /  December 3, 2010

    Hi Jay,

    This is referring to 11gR1. My recommendation is to take a look at: http://jarneil.files.wordpress.com/2010/11/upgrade-war-story1.pdf which is gives a good guide to 11gR2 RAC and database upgrades.

    jason.

    Reply
  1. Log Buffer #83: a Carnival of the Vanities for DBAs
  2. Rolling Oracle Clusterware Upgrade to 10.2.0.4 « jarneil

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 52 other followers

%d bloggers like this: