Neil Barton

Wednesday, September 16, 2009

Let’s Benchmark!

[Fairground Music]

Roll up, roll up, ladies and gentlemen, I have great news for you! Today is your lucky day. In these health-conscious times, I am going to offer to one of you fine ladies and gentlemen a free health-check. I’m going to tell you exactly how much you should weigh.

Yes, you may well gasp with amazement. How will I do it? Simple, sir, I’m going to do a benchmark. I’m going to compare you against a group of other people just like you, and we’ll use their weights to calculate what your ideal weight should be.

So to get us started, ladies and gentlemen, I need you all to write your weight on a scrap of paper. Ladies, you have nothing to fear. I appreciate that this is highly sensitive information I am asking for. It will be kept in the absolute strictest secrecy, except for when I need it for other benchmarks. But I assure you I will never tell anyone the weight that you've shared with me, no matter how much they pester me for it.

Now then, next I need a volunteer from the audience to receive this valuable gift. How about you, sir? What’s that? An urgent call on your Blackberry? How about your young colleague over here? He looks like a healthy specimen to me. He’ld be delighted? Or course he would. Come out here, young sir, so the audience can get a good look at you.

Now, sir, your name? Joe? And where are you from? Missouri, wonderful place. Do you work-out, Joe? No? Well, it is hard these days isn’t it, the way our bosses work us. Not to worry, once our benchmark has set your target weight it will all get a lot easier. Joe, let me see the piece of paper with your weight on it. Pardon? You would rather keep it for the moment? You’re not a very trusting person, Joe, if you don’t mind me saying so.

So what we’re going to do, ladies and gentlemen, is use your weights as a benchmark database, and we’re going to select a peer group of the ones we think are a close match to Joe. The most important thing, of course, is to make sure that we carry out a fair comparison – comparing apples with apples, as we benchmarkers like to say. I think we should have six to eight people in the group, Joe – it wouldn’t be fair to pick the skinniest guy in the room and expect you to match that, not with your build. Ideally they will all be the same sex, the same age, the same race, the same wealth, the same lifestyle, and the same love of good food and wine that your parents passed on in your genes, Joe. What do you think? Not going to happen, is it? Tell you what, Joe, after we’ve picked out the peer group, we’ll agree to make an adjustment for any important differences between you and the others. This makes it a lot harder for me to calculate, to be honest, Joe. But if it will make you feel more confident in the results, then heck, we’ll do it.

Lets get started on selecting the peer group. How about that gentleman over there at the back, Joe? No? Why on earth not? Runs marathons? What, for pleasure? How extraordinary! What sort of times do you run, sir? Three hours! Very impressive! Well I don’t know what you think, Joe, but I think this is just the kind of healthy model that we need in our peer group to help you buck up and lose weight. Yes, we’re having him in the group. Now, what about one of those gentlemen from the Tokyo office over there? What’s that Joe? You’ld rather have one of the Americans? What’s wrong with the Japanese? Ah, I see what you mean, Joe, you are rather taller than them. The trouble is, Joe, there are only the eight men in the room, and once we take you and our marathon-running friend out of the picture, we’re going to have to include at least one of the Japanese. I know, Joe, we’ll make one of these adjustments to compensate for the difference in height. Who’ll come up with the adjustment? Why, I will, Joe. Don’t worry, I’m an expert in this, I do it all the time.

Excellent, we have our peer group. Now, if I can just collect the pieces of paper with your weights on from those of you we have selected, that’s you, sir … and you … and you … and you, and you, and you sir. Thank you very much. Let’s take a look … ah, one of you has put your weight in kilograms. Umm… remind me how to convert it? A pound of jam’s a kilogram, something like that isn’t it? Or is it two pounds? I can never remember. Sometimes I think it changes every year! Shall I just divide by 2, then? 2.2, is it, miss? What ever you say.

And, Joe, just to set your mind at rest, I’ve added 10% to the weight of the shorter gentleman here. What’s that? Not enough? Why Joe, you’re not trying to fiddle the target, are you? You think it should be 20%? We’ll compromise on 15%, how about that?

Now that we've got an accurate comparison, I can present the results. But Joe, I want you to take this target seriously. It’s coming up for annual performance appraisals. Joe, I want you to accept as your objective for next year that you will achieve our target weight. Oh – he’s a contractor, is he? All right, then … let’s agree that if Joe doesn’t meet our target weight, then you can reduce his daily rate by, shall we say, 10%?

Now, ladies and gentlemen, we are approaching the moment of truth. Let me just add these weights up … divide by six … um, take away the number you first thought of (just joking, Joe), and, ta-ra, we have a result.

Joe, this is the target weight that we want you to achieve. [Thud] Joe? Joe!? Quick, catch him, someone. Joe, are you all right? Thank goodness, you had us worried there. Sit down, take a breather. What do you mean, it’s impossible? We’ve benchmarked you against a group of people whom we’ve agreed are pretty much the same as you, and we’ve made adjustments where we found any real differences. So sorry, my friend, this is the target.

How did I calculate it? Ah, well, this is the clever part. We don’t want you to be just an “average” weight, do we, Joe? We want you to be a lean, mean, fighting machine. So I didn’t use the averages of the peer group, I used the 25th percentile. That means … what’s that … oh, studied statistics at college, did you? Not enough in the group for a percentile to be statistically accurate? Well, yes, I suppose so, Joe, though I can tell you that I was doing this down at the offices of a law firm the other day, and none of them had any problem with it. What do you want to use instead? Average? But surely you want to be healthier than average? You’ld rather be mathematically correct? Well, ok, if you insist, Joe - it’s your health.

So you accept the target, right, Joe? What’s that? You want to see the individual weights that the peers wrote down? Why on earth do you want that? Outliers? You really are a terribly suspicious person, Joe. And I did kind of promise the gentlemen that no-one would see their weights. I suppose it will be all right as long as I mix them up and no-one can tell which is Mr Kawasaki and which is Pavarotti over there. There you are, Joe, are you satisfied now? Yes? Excellent! And do cheer up a bit, Joe, I’ve done you a favour here!

My Last Word On Contractual Benchmarking

It's not very often that a company advertises its services as "contentious, time-consuming and expensive", but that's how Compass describe contractual benchmarking in their August newsletter. You don't hear the same message from Gartner, Maturity, Alsbridge, or Metri, but this has been a consistent position by Compass. Back in 2007 they opened a web cast on "The Power And The Pain Of Benchmarking" with the words "Benchmarking can be a long and trying process for both client and vendor (and benchmarker)".

It may be strange marketing, but Compass are telling it like it is. Contractual benchmarking is not growing, and recent staff reductions at benchmarkers suggest it may be declining. Customers see more value in having a benchmark clause then they do in executing one. Having the clause re-assures everyone that there is mechanism for prices to track the market during the contract. Actually doing a benchmark takes time and money which many customers would rather focus on improving the business.

Yet this is an industy which is reluctant to improve itself. Negotiating a benchmark clause is still a haphazard process. Customers, outsourcers, lawyers, and consultants horse-trade over the wording of the clause, but benchmarkers are rarely consulted and clauses are still signed today which benchmarkers would advise against. A benchmark clause takes 4 pages in a 400 page outsourcing contract. On most transactions, there is a strong drive to finalise as many sections as possible in the time available. There is no incentive for the industry to improve the standard of clauses.

When a customer comes to execute a benchmark clause, benchmarkers clamber over each other to offer a fixed price, even though the effort it will take them to collect data and agree a peer group is largely out of their control. Procurement helps by negotiating the price of the benchmark down to that of the lowest bidder, so it is a constant struggle for a benchmarker to deliver their work profitably. There simply aren't the margins in this work to allow the benchmarkers to invest in improving their data and methodologies, which is probably why there has been so little innovation in benchmarking over the last 10 years.

Benchmarkers are increasingly resentful when outsourcers or customer want to inspect their calculations. Yet it is surely wrong to argue the opposite - that there should be no quality standards of any kind for a process which affects the jobs and businesses of all involved.

There are no published studies on the effectiveness of contractual benchmarking. Several academics have tried, but found it extremely difficult to reach customers who are willing to talk about their benchmarking experiences. So we simply don't know how often they have a positive outcome for a customer, and whether they are cost-effective.

No wonder then, that one benchmarker said to me "Get me out of this hell-hole" when we met in the airport bar after a particularly long and contentious benchmark meeting. These are the reasons why I don't work on contractual benchmarks any more, and why the next post will be my last word on contractual benchmarks.