As you may know, many of the ancient religions have such doctrines as “the 5 pillars of wisdom” or the “4 noble truths” that lead humble pilgrims to true enlightenment. Although I am not suggesting we start a new Performance Test religion that perhaps worships the god “Mercury”, I have noticed that there are the 7 levels of denial that developers/system architects/managers (aka pilgrims) seem to have to go through before they realise or admit that they have performance problems (i.e. true enlightenment).
Like following a religion, this is a personal journey and a unique path is followed by each pilgrim, with no two making the same realisations or decisions at the same point in the project lifecycle, some having to repeat parts of the journey several times (as I am writing this, Mike is explaining again to another set of pilgrims how LoadRunner works).
My Experience suggests there are 7 levels of denial, as follows:
1) The load test tool must be wrong – you may be using the industrial standard performance test tool costing £100K, but the quick test the pilgrims did with the free tool downloaded from the internet was better. When you ask whether HTTP 500 status messages were trapped or if the data returned was validated the pilgrims look confused. So, take a deep breath, explain the benefits of a proper tool and move on.
2) The performance scripts or workload model must be wrong – you have only been a performance tester for 5 years and worked on countless projects, so its nice to be told you are stupid. Take a deep breath, walk them through the code and enjoy their look of surprise as they suddenly realise what correlation is.
3) The system is not finished so doesn’t need to be tested – pilgrims can believe that the last 5% of functionality will increase performance so that a poorly performing system will obviously get better in a minute.. Explain politely how adding new code won’t magically improve the performance of the old.
4) It’s not our system, it’s the network etc- blaming the supplier of the components is a common area of denial. To correct this misconception, feign a look of surprise and then arrange a test to show that the offending component runs super fast.
5) We JUST need to configure parameter X – the pilgrim often has the belief that the correct setting of a single magic parameter will solve any problem. What is annoying is the condescending tone often in which the pilgrim states that this is surely the problem and that you the tester must be a complete donkey for not setting this. Of course you smile politely, say lets give it a go and, when nothing changes be dutifully diplomatic. Often you will iterate on “number 5” as several pilgrims in the development team search for the “silver bullet” (or is it “Rocking horse pooh”). An obvious attraction to pilgrims of this approach is that the solution is “only a test cycle away”. Many a manager pilgrim has followed this route.
6) Throw hardware at the problem – although this does often have an effect, adding an extra processor to a DB server that is crippled by too many stored procedures doing table scans is as useful as a chocolate tea pot. Take another breath, particularly when you are told the lead time to order the components and re-install the software. Just stay alert because the pilgrim will be very happy believing they are solving the problem and can relax while they await the new hardware.
7) We JUST need to tune a small part of the system – there is often a hope that only a small store procedure or code element, once tuned, will yield the magical performance improvement or that all the performance problems can be found in one part of the architecture. At last the pilgrims are getting somewhere on their journey allowing you to progress too – some measurements at least have to be taken to identify the bad boy item. You can smile now; the journey’s nearly over..
Journey’s End – Wow! we do have a performance problem – at last your bunch of pilgrims have made the journey to true enlightenment. Like any religious journey it is full of self-doubt and distractions along the way, but at last you can finally start to solve the problem. Just hope now that a new project manager doesn’t get involved and you have to start at step 1 again.
Some projects have less potential areas for deviating from the true path, and some have more, but each pilgrim has to find his own way. Your job is to guide and educate!
1202Performance 