In Enkitec We Trust

The 2nd of March 2015 was my first day as part of the Accenture Enkitec Group.

When I first started using Exadata back in 2011, the one thing I relied on more than anything else to get me up to speed was the original Expert Oracle Exadata book by Kerry Osborne, Randy Johnson, and Tanel Poder. I am equally sure the 2nd Edition will prove just as valuable.

e-dba years

I have thoroughly enjoyed my past 3 1/2 years with e-dba. Both myself and e-dba as a company have grown enormously in this time, and it has been a really positive experience being with a growing company.

At e-dba I had all the exposure to Exadata I could have wanted and they have many Exadata customers and a large number of exa racks under their care.

It was a wrench to leave.

Feeling gravity’s pull

Over the past couple of years I have come to know and appreciate the talents of several of the members of the Accenture Enkitec Group. Kerry expressed this well in that oak table world talk at OpenWorld 2014 as an effect like a “gravitational pull” when recruiting people.

It is certainly something I felt when weighing up my options. The prospect of working with such an outstanding collection of Oracle talent was too hard to ignore.

I would always have regretted not haven taken the chance to work in this team.

I can’t wait to get started.

New Beginnings

Today I resigned from a job that I have loved. In many ways it felt like cheating on a girlfriend. The illicit calls to the other party behind the first ones back, it really felt in some ways like betrayal. I was not looking forwards to informing my current employer that I was leaving them for another. Though to be fair to them they have been fantastically supportive.

I have given 10 years of service and have shepharded the companies databases and indeed the companies data throughout this time. From the hazy beginnings on a scattering of Oracle 8i single instances to the heady heights of MAA architecture and 11gR2 today. It has been quite a journey and I have learned a lot on the way, particularly in regards to many tasks that are not normally viewed as being with the standard purview of a DBA.

Total control over the OS? Check.

Complete control over storage configuration? Check.

Ability to change any network settings? Check.

For a DBA it’s total control to have things set up in just the way you want them. I would not hesitate recommending experiencing this level of control to another DBA. No finger pointing, no waiting for someone else to do their piece of work. Success or failure is down to you.

And yet, I’m still leaving!

For some time I’ve been looking for more. More varied systems to work on, and new things to learn. I’m really hoping the new opening is going to enable this. I’m leaving to become a consultant with e-dba.

e-dba are forming a new team to look after their Exadata customers and I’m going to be part of that. I’m also really hoping for a varied range of experiences with them.

Indeed I’m really looking forward to be getting my hands on some Exadata work! In fact so much so, that I’m reading up on the fantastic Expert Oracle Exadata book already.

In Praise of Evernote

I don’t consider myself to be the most organised individual. For quite a long time I’ve had the feeling that if only I could organise all the information that flows my way, and make it easier to find I would become far more productive and generally get more stuff done.

Sure I’ve read David Allen’s excellent Getting Things Done, but I have never managed to move from appreciating the concepts to actually transforming the way I work.

While the book is very much arguing it’s not a particular piece of software that you need, I think I may have finally found a piece of software that enables me to get more things done – Evernote.

It basically allows you capture lots, and lots of information in different formats, text, images, pdf and then makes them all searchable. There are clients available on multiple platforms, the mac client being particularly nice, but also there is a web interface as well, so it’s accessible on most mobile platforms.

There are also multiple ways of getting information into software – cut ‘n paste text into a note, drag a pdf into it etc. There is also a screen grab clipper that can get info too, and don’t forget even images get text recognition done on them.

Evernote, is not just a great way of capturing long term information, I’m finding it really useful at handling to do items as well, including whenever blog ideas pop into my head. It also has a great feature that it will sync this information that you have captured to multiple devices, just install the client on whatever desktops/laptops you need to acces the information and they will all be kept in synch. The notes you make are also accessible over the web using a browser so you have access to all your information on the go as well via a mobile device.

I really can’t praise it highly enough, and best thing of all – the basic version is FREE.

The Different Dataguard Modes of Protection for Managers

I was recently asked to provide a description of the various modes of operation possible with a standby configuration. This was needed to explain the setup to a group of non-technical managers.

The original setup of this companies dataguard configuration was Maximum Performance but the company was concerned about the potential data loss associated with running in an asynchronous mode.

Below is what I produced for them, nothing particularly revelatory here, but perhaps it might be useful for you if you are having this debate with your management team.

Introduction

When using an Oracle standby database for Business Continuity purposes there are 3 possible modes of operation for determining how the data is sent from the primary (the database currently being used to support the business queries) database to the standby (failover database to be used upon invocation of business continuity) database.

There are 2 synchronous modes and 1 asynchronous mode.

The various modes of operation of a standby are a real trade-off between data protection and impact on the primary systems.

Modes of Operation

The following table shows the options giving the benefits and drawbacks.

Mode of Operation	SYNC/ ASYNC	Benefits	Drawbacks
Maximum Protection	SYNC	Never any data loss, no data ever accepted on primary without being on standby.	Slower response times on primary. Network disruption between primary and secondary causes downtime of primary database.
Maximum Availability	SYNC	Updates must be on standby before accepted on primary, network disruption means the primary continues running with updates sent to standby when it is available again.	Slower response times on primary. Data loss possible if network fails and then site fails.
Maximum Performance	ASYNC	No performance slowdown for applications on primary.	Data loss will happen

This customer is currently running in Maximum Performance mode, and thus the standby solution has minimal impact on the response times of the various applications.

There are 2 factors that would impact the application response if they moved to a higher level of data protection, the network latency and the network bandwidth. During normal operation, the response time of applications that make data changes would be impacted by the round trip time (rtt) to their standby

During periods of high volume activity, the bandwidth can become congested and this could lead to significant delay in response time.

Conclusion

If a higher level of data protection was deemed a business requirement I would advise a significantly larger network bandwidth to the Business Continuity site.

During normal running we typically see a transport lag (of data between the sites) of around 10 seconds and this would be the typical amount of data we would lose in a disaster situation. During high peaks of activity this can increase.

In determining to increase the protection mode of the configuration there is a real trade off between how much you value your data (for this customer it is a reputational issue as much as a financial consideration) and the impact on your normal day-to-day performance.

Stock Photos

I got a question after my “What ASM Can Do For You” presentation yesterday regarding where I got the images that are in the presentation and why there was no attribution.

An example below is one I have used in the past when talking about myths:

The images I use are all micro stock photos. These are low cost royalty free images from amateur as well as professional photographers. They are particularly useful for promotional literature and presentations, but you must not use the images for sale on things like t-shirts or mugs.

The site I tend to use is iStockPhoto which charges $3 a for a suitably sized image for use in presentations.

One option you could use, if you balked at paying around £1.50 per image, would be to use Flickr and look for images with a Creative Commons. However, these do need attribution though this could just be a slide at the end listing the copyright holders.

I find that images with the creative commons license tend not to be quite as useful as real stock photos – the stock photos tend to have a lot of empty space that you can use to actually put some text on!

The theory is of course, that it is easier for the audience to listen to the presenter when they are not being distracted by reading bullet point filled slides. Also having an appropriate visual image to go with what the presenter is saying should make it easier for the audience to remember the points.

It certainly feels more interesting creating a presentation where you have to think a bit more creatively than slapping those bullets down onto your powerpoint template.

Interviewing Oracle DBAs

You may have seen in an earlier article I was trying to drum up interest in a recruitment drive we were having. Well, we whittled down the applicants and recently held the interviews. There was one candidate that stood out quite away above the other hopefuls. I thought it might be of interest to go through some of the interview questions that we used. I know Howard J. Rogers (currently and perhaps permanently off air) has in the past written a fantastic series of pieces on his interview questions, but I feel our experience is worth sharing. Though it seems the lack of good quality Oracle DBAs is a universal theme.

First off, I think it is fair to say the interview was split roughly in two, with the first half covering more attitudinal questions and the second half covering more technical Oracle questions. We certainly place as much weight on finding the correct sort of candidate, one with a good attitude to learning and customer service, as much as what particular level of Oracle skills they have. Also it’s worth pointing out this is for a job in a small company, where there is no strong demarcation of responsibilities, so the DBAs are expected to do a goodly amount of system administration work, and have ownership of configuring the SANs. The candidate we chose I would say displayed excellent answers in both the attitudinal and technical questions. Here are the Oracle questions we asked, I think these are pretty straightforward and gentle questions, there are not even that many of them!

What approach would you take if asked to investigate a query that was running slowly?

A couple of candidates answered this pretty well. I was looking for things like checking the explain plan, and tracing the session. I was not looking for someone to mention the buffer cache hit ratio – though someone did!

What SQL commands will cause a sort to take place?

Unfortunately a few candidates did not get past order by (which all got), some said distinct and group by. The candidate we hired mentioned these and UNION. No one mentioned create index.

How does Oracle implement read-consistency?

Quite a few candidates stumbled at this question. It is such a fundamental part of the Oracle RDBMS that you would hope a candidate would at least have go at answering. A couple of candidates said they were stumped. I was looking for them to mention that a query only sees data committed before the start of the query and uses undo to reconstruct any data if it finds it with a commited time that was after the start of the query. No one mentioned SCN.

Do you know what ORA-01555 is?

This is somewhat related to the previous question. Practically every candidate did not know this. I do not expect a candidate to know every single Oracle error message, I certainly don’t. However 1555 is such a classic error that I would expect good dbas to know what it is. The one candidate who did know that it’s snapshot too old, got the job.

What resources do you use to keep up-to-date with Oracle?

Well I probably was a little disappointed with the responses to this one, only one candidate mentioned a good spread of bloggers, or indeed blogs at all. Everyone mentioned OTN, but no one mentioned actually downloading a new release and trying it out for themselves. Oh and this question lead on to which authors the candidates liked, Tom Kyte was the definite winner with Steven Feuerstein in second place. I was astounded not one candidate mentioned Jonathan Lewis.

Explain the architecture of RAC?

So, I realise this could potentially lead to a wide ranging answer, but most of the candidates had RAC on their CV so you’d expect them to able to explain it at least to a level that they new it was different from single instance Oracle. Minimum requirement was multiple instances accessing the same database datafiles, communicating through a private interconnect, which most met.

Finally, here is one of the non-technical questions we asked:

What kind of working environment do you need to be most effective?

I’ll leave it to the reader to work out what we expected from this one!

Advertisement: Nominet are recruiting

This is a bit of an advertisement for my current employer, Nominet, who are one of the largest internet registries in the world, and run the .uk ccTLD. We are currently looking for an additional DBA. You will be able to get your hands dirty on RAC and Dataguard, we currently use 10gR2 but will be looking to upgrade to 11g some time this year. Nominet actively encourages you to blog about your work and there is plenty of opportunity to become involved with the UKOUG.

You will need a good amount of experience with Oracle but an ability to learn is probably just as important as number of years clocked up. A working knowledge of Unix/Linux would be beneficial as we have to support the full stack from carving up the storage to configuring the applications. The technical department has an excellent can-do attitude and you probably will have too. Nominet is a comfortable working environment and there is also the bonus that you would be able to work alongside some excellent people 😉

I have been involved in several recruitment rounds with Nominet and I have been really astounded at the lack of applicants that we have received. Is there really a dearth of DBAs out there? I got round to concluding it was the location, as the rates of pay on offer are competitive compared to other non-city of London positions. We also seemed to attract large number of applicants with dubious work permits. I had to laugh when I saw a candidate who claimed to be OCP certified and had this stamped onto his cv, but in the text had misspelled Oracle!

I have also been surprised when an interview candidate mentioned the buffer cache hit ratio in response to a question regarding a good way of investigating users complaining about performance problems. They did not get the job. I was amazed, I think partly, due to the advent of blogging. Compared to even a few years ago, there is such a wealth of excellent information out there in the blogosphere, that I think any person claiming to be a DBA who is not keeping up-to-date with current Oracle best practice just has no excuse as it’s never been easier.

So if you or someone you know is interested, then the full details are on the Nominet website. Also there are other vacancies for a System Administrator and a couple of developers into the bargain, all based in Oxford. The closing date on these is 27/01/08, so if you are reading this after then, sorry you are too late.

Prevention is better than cure

I loved this posting. I think it applies just as much to DBA’s and System Administrators. I have to confess I have had the occasional thrill of playing the hero at the scene of a car crash of a situation. It can feel good. In fact I think you can easily get addicted to being in this type of situation, where you hold the key piece of knowledge to fix a particular problem. I think I have at times been on the edge of the precipice of enjoying it too much and almost hoping for another opportunity to be a troubleshooter. But I see the danger of enjoying being a troubleshooter and I can see the benefit to the organisation of being a “troublepreventer”.

Recently, on the Oracle-L mailing list someone was asking for a list of tasks that a good dba should do on a regular basis. The person was given short-shrift in that they were told if there are tasks that your dba is performing on a daily basis, why are they doing them and why have they not automated them?

I also like the idea of “exception based reporting”, that is if everything about a system/database etc. is running within specified criteria, I don’t want to know about it, and only when a monitored criteria becomes outwith an acceptable criteria do you want to hear about it. However, this does require you monitor the system closely. As an example, I would suggest monitoring your I/O response time on your database, and/or critical query response time, and then if these change to be outwith acceptable times you flag it up and get someone to look at it.

Of course, I should eat my own dog food a bit more here.

How to be a failure

I was at a couple of London Business Forum events today, both featuring Steve Mcdermott. The first session was on Steve’s book, which is written as a series of steps NOT to take if you want to become a complete failure. I’m a sceptic by nature, I think most people with scientific training are. Though I hope I’m not a cynic, I suspect I do tend towards this. I started out by thinking steve was a bit of twat, but I really started to warm to him. He has had decades of presenting and really, really knows how to work a crowd; straight away he got on first name terms with a couple of members of the audience, and a few laughs really breaks the ice.

Steve highlighted a handful of his 44.5 steps to being a failure then expounded on them with copious examples and anecdotes. One of the first things he mentioned, which I took exception to (a little), was claiming that to fail you should never experiment and as an example of experimenting he gave 4 members of the audience various bells, rattles, and whistles with instructions to ring/blow/shake these after a set period of time. He claimed never to have done this before. I immediately thought about IT and that (obviously) you should always, always TEST any change before it is rolled into production. I think in the Oracle world there have been (well I guess probably for most of this decade anyway) a move away from magic bullets and changing/experimenting with parameters that claim to have magical properties in terms of tuning, to a far more scientific way of fixing your problems. Everyone knows Anjo Kolk, right?

His stuff on setting goals and trying to find someone to pay you for doing something you consider a hobby did get me thinking. I currently work for Nominet and I’ve been there 6 years, which in IT is a long time to be in the same place. I guess if you don’t have any goals you really are never going to achieve them.

I thoroughly recommend reading How to be a complete failure. It is very readable, filled with some great quotes and anecdotes, and hey, if you do some things he says don’t do you might not be so much of failure after all.

In the afternoon Steve was talking about “Presentation Mastery”. I must say straight away the guy knows how to communicate with an audience. I think what is not surprising, but is a fabulous technique, is how he does it really without any slides or notes to speak of. It is most definitely NOT death by Powerpoint. I took away a few key points, perhaps fairly obvious ones, but you have to know what you are talking about and have a story to tell. Have a dress rehearsal, oh and don’t just read what is on your slide! His idea of mindmapping your talk seems like a good one to me!

All in all a good day in the big smoke and only 2 hours travelling time Oxford to London door-to-door. I’d recommend checking future LBF events.