Neil Barton

Monday, September 29, 2008

New Alsbridge/ProBenchmark paper

Alsbridge have posted a new white paper on benchmarking. Written by Chris Pattacini, it's very consistent with the position Chris has taken in his previous lives at Nautilus and META Group. In summary:
  • Don't use cost benchmarks to benchmark prices
  • Best practice is for customer and supplier to pay half the benchmark cost each, to ensure benchmarker independence
  • Quartile clauses are "unenforceable" because they "imply statistical rigor that does not exist".

Chris comments that cost benchmarks are more widely performed than price benchmarks. That used to be the case ten years ago, but now I'm not so sure. It would be nice to see some hard data from Gartner, Compass, or Maturity.

Wednesday, September 10, 2008

Poorly Formed Security Metrics

I read this week on Dark Reading that the Centre for Internet Security has released a set of free metrics definitions which can be used to measure how well an organisation is protected against security threats. This is an excellent, valuable, worthwhile initiative which I encourage in every way. Having said that, the way the metrics are defined (from what I have read so far) suffer from two common problems which will make them difficult to use in practice: ambiguity; and impracticality. Accurate comparison requires accurate definition, so the suggestions I offer below are intended in to be positive (even if they look negative at first sight.)

Let's pick out one or two of the ten metrics CIS have defined.

First issue: Ambiguity. Take "Percentage of systems with anti-virus protection". I imagine no-one would debate that all PCs and servers should be included. But what about PDAs and smart-phones? These are increasingly subject to virus threats, and some customers have quite expensive solutions to protect them. So do they count for this metric? It really matters, because it can change the percentage a lot. Say I have 10,000 PCs, 50% of my users have PDAs (none of which are protected by anti-virus), and I have 1 server for every 100 PCs. All my PCs and servers are protected. If I don't count PDAs, I score 100%. If I do, I score only 66%. This is an extreme example, because there can be so many PDAs, but the same problem in principle applies to thin clients, LAN switches/routers, firewalls, manufacturing or point-of-sale equipment, and perhaps even VOIP handsets. There needs to be some very clear unambiguous definitions of what counts as a "system".

Second issue: Impracticality. Consider "Percentage of application code that underwent security assessment, threat model analysis, or code review prior to production deployment". We can see at once the intention of the metric: how much of the code base bas been checked over. To collect this metric, I need to gather two sets of raw material. Firstly, I need to know how many security assessments, threat model analyses, or code reviews have been done. (Already an ambiguity arises: how far do I have to go back? Every security assessment ever? Only the last 12 months? It could make the metric look good or bad depending how you answer this.)

Worse, though, is the practicality of such a task. To calculate a percentage of application code, I need to count two things: how much source code in total, and how much source code has been checked. How do you count source code? You can count Non Comment Source Code lines, but is it really meaningful if you do this across all the applications in an organisation's portfolio, in a set of languages probably ranging from Assembler via COBOL and PL/1 to Javascript? Many businesses have 500 business applications or more, and find it hard enough to count the applications themselves, let alone the lines of source code in them. And anyone who has bought a packaged application (which means everyone) won't have access to the source code so can't count it anyway.) The goal behind this metric is worthy and admirable, but I can't envisage that it will ever be collected. And if someone does try, I would certainly enjoy the discussion they have when they put the code-counting request to application maintenance.

Some practical suggestions for getting to the same result a different way: Start by counting security reviews in the last 12 months only. We probably don't value older reviews so much anyway, and it costs much less to collect this metric if we only have to go back 12 months. Then divide this into the number of applications (not the percentage of code). This is still hard to collect, but is at least practical. Since not all applications are equal here, divide them up into some categories: perhaps Small/Medium/Large. Concentrate on the Medium to Large categories, since you probably have fewer applications of this size but care about them more. There can be a large "tail" of small, often old, legacy applications which are much harder to inventory but probably don't matter so much if they are a threat.