Big government, big IT

The main problem with big governmental IT projects is also that succesful big governmental IT projects are not very distinguishable from failed big governmental IT projects. You’ll always gets cost overruns and kludges.

What makes the British big governmental IT projects so bad even when compared with dismal failures elsewhere is that New Labour fundamentally doesn’t seem to get IT at all.

It’s not just an equilibrium outcome in big goverment IT – I remember being told by a software testing tools vendor that something like 80% of IT projects (that involve writing new code, having to be compatible with old data etc.) in the private sector ‘fail’ too (or at least are hilariously over budget and late by the time they work) but we just don’t read about it in the papers. The same used to be true of the construction industry – everythimg was built late and 3x over cost. As I undertsand it, the construction industry is getting better. We have discussed this before here . The thing that strikes me about this, is that fewer projects would be classified as failures if both parties acknowledged projects were going to cost twice as much and take twice as long in the first place. The equilibrium outcome we see is one where the criterea for success (the expectation) is repeatedly unrealistic. Why don’t both principal and agent set more realistic expectations? It’s contract design. Something to do with the inherently ex-ante uncontractable nature of the job (because you don’t really know what’s needed until you start, and the client will keep changing its mind) so the equilibrium is that an unrealistic cost and schedule is agreed (which might even be loss-making for the contractor/agent – I know in construction, contractors put in loss making bids and only made a profit from adding work on later) and then cost is added as they go along.

This way of doing things, with its inevitiably high failure rate, may even keep the overall costs down. If realistic expecatations were set in the first place, the agent/consultants would be able to fleece the principal/client even more. It’s a bit like when you set yourself a deadline for writing something – if you give yourself a month, you know you’ll take two. But if you start by giving yourself two months, to be realistic and avoid ‘failures’ you’d just take three. Or perhaps somebody will come up with a cleverer way of writing the contracts, and these failures will go away. If I’m right that construction is getting better at this, then perhaps the nature of the contracts have changed there already.

If contract theory didn’t make by brain hurt, I’d love to write a paper on this.

“can actually work quite well”, even. Obviously there are plenty of disastrous private sector projects.

And John B, you are either living in a fantasty world or I would really love to know which consulting firm you are talking about that sounds so decisive.

I don’t mean the consultants – I mean that corporations have senior managers who’re able to make compromises up front rather than having to deal with endless stakeholder feedback. Not least, using something off-the-shelf and adjusting the organisation/product to fit with the software (and yes, I’ve been involved on the corporate client side with IT projects which have worked like this).

The point about government failures being public and corporation failures being private is entirely true and fair. Nonetheless, the fact that EDS’s public sector practice makes double the margins of its private sector practice, and that nearly all of the big IT firms’ margins come from contract variations, suggests the government side of things might be even less effective.

The population of Britain is under 4 million then?

The fate is the same in the US as well, and as many have mentioned in the corporate arena (and you just tend to not hear about those failures). There are too many factors to point to, but there are some very well known published studies that call out the main culprits. Requirements tends to be #1. They are either poorly understood from the outset, poorly defined, and/or change too late in the process. It is clearly well know that the later in the project lifecycle you change a requirement, the exponentially higher cost it will take to implement the change.

The difference between corporate and government IT projects is the acquisitions process. In the US, we look for the lowest bid. Is it ever a shock that the company that comes in $4m under all other competitors for a project that “should” cost $30m fails and goes over budget? Behind closed doors (and sometimes out of them) many government consulting companies/firms will tell you they low bid with the intent of “mod’ing” the project to the ends of the world to drive up the price. Often times as a consultant you can read the RFP and SOW and point out dozens (if not more) mistakes, omissions, duplications, conflicts and flat out idiocy.

Saw a presentation by an Accenture manager who admitted that “Accenture Assured Delivery” process was rubbish.

And public sector projects usually have a built-in problem, they are often written to meet brand new legislative requirements and so often can not draw on an existing knowledge base.

What you want to do is split up the massive IT project into little, and hopefully redundant, bits. Fund them all, and let God sort them out.

There was a good article in the New Statesman (trust me on this) about why UK public sector IT projects fail so badly, so often. If I recall correctly the author pinned most of the blame on two things – the lack of dedicated/experienced project managers in the civil service and the tendency to go for the biggest contract possible rather than breaking a project up into smaller contracts. He contrasted the UK, which does particularly poorly even compared to other countries’ public sectors, with (I think) New Zealand, although it might have been Denmark and it might have been both of them. Of course, the NHS IT programme is the classic example, where gargantuan contracts were awarded to the handful of companies that could (in theory) cope with them, leaving the government over a barrel when they screwed up or wanted changes to the specification. Those were hardly the only problems, of course.

I don’t mean the consultants – I mean that corporations have senior managers who’re able to make compromises up front rather than having to deal with endless stakeholder feedback.

Bitter, hollow laughter.

Large companies are the worst kind of environment for making compromises up front. There’s no worse bureaucracy than huge corporation bureaucracy.

I don’t think experience in project management is all it’s cracked up to be. None of the Bank of England CREST team were experienced project managers and very few had any direct experience in IT, but they delivered the goods. The problem is a massive failure of mojo on the part of HM Civil Service, who appear to have got into an institutional funk which leaves them unable to use the judgement for which they were hired and altogether wildly too credulous of their hired experts. And also revolving-door careerism, let’s not underestimate that.

> There are too many factors to point to, but there are some very well
> known published studies that call out the main culprits. Requirements
> tends to be #1. They are either poorly understood from the outset,
> poorly defined, and/or change too late in the process. It is clearly well
> known that the later in the project lifecycle you change a requirement,
> the exponentially higher cost it will take to implement the change.

The fundamental basis of this analysis is that the waterfall model of systems development, revolving around the fully-defined, correct, and complete “spec”, is valid. I am not fully convinced by the agile/extreme camps yet but it is well known that the waterfall model hasn’t done very well over the last 20 years. The response so far has been the PMI, CMMI, and attempts to ratchet the process and the spec down tighter and tighter until nirvana is reached and I have to say that doesn’t seem to be going very well either.

Cranky

Milkroaded? Talk about a mixed metaphor. Milkround meets railroad.

My guess with CREST is that there was a degree of atomisation in the abstract conceptualisation of the system — getting stuff from A to B without fuckups — that dovetailed pretty well with the basic skillset of the people assigned to code it all up. There’s an algorithmic purity to that kind of thing. I’d agree with those upthread that taking a UNIXy approach — dividing projects into small, self-contained, discrete functions — is optimal, though that still raises the problem of interoperability at the most basic level.

(Still, that also reminds me of a conversation ten years ago with a friend at a Large Consultancy Firm who’d been flown out to San Francisco and wanted a primer on the whole dot-com thing.)

I said this last night, but it appears to be stuck in moderation hell:

Abb1, the thing about ITIL is that it was invented by the government’s IT specialisation when the government had an IT specialisation. Since then, they sacked all the geeks and outsourced to EDS.

Secondly, and more generally, Redwood doesn’t seem to realise that there is a way of having government IT requirements met in a more incremental and human-scale way whilst not using as many consultants – but it’s called “doing it in house”, and it requires that all the organisational entities who might want their own small problems solving have their own engineers. This means re-establishing a huge government-wide IT career path, a public sector IBM. Is this really what he wants? (Mind, I’m quite keen on it myself.)

I mean that corporations have senior managers who’re able to make compromises up front rather than having to deal with endless stakeholder feedback
Erm, no – I laboured for some years in the IT department of A Very Big Airline, and believe me there were failures of small, medium, big and very big IT projects a-go-go for the discerning connoisseur. The problem was not so much the irresponsible use of the word ‘metadata’ as irresponsible use of the words ‘all we have to do . . . ‘. Oh, and the fact that systems had a habit of being designed to meet the psychiatric needs of managers not the business needs of their departments . . .

There are too many competing interests for large governmental IT projects to succeed. Some want the project done, some want the project to fail, some will only “support” the project if their pet gizmo (sometimes called a widget by those MBA types) is included. As a result, most gov-IT projects suffer from severe gold bricking. For an example, I’d refer you to the FBI’s Virtual Case system, which had to replace some 40+ different computer systems AT ONCE. If they wanted it to succeed, they’d start with a system that replaced one existing system, and with each iteration roll in the functionality of another system.

Part of it is due to impedence mismatching between the “specs.”
The paperwork says that work is done one way, (mis)managers think that work is done some other way, and the workerbees have a thirdway where work actually gets done.

And private industry isn’t any better at getting IT to work. Just better at hiding the dead bodies and spinning what can’t be shoved under the carpet. For a wonderful example, I refer one to the Denver Airport baggage handling system. The engineering management said “projects of this size take 4 years to complete,” while the sales mismanagement said “airport is finished in 2 years, so we sold it to them, you have 2 years to make it work.” As everyone knows, the baggage handling system was functional (meaning: it stopped losing and shredding bags) 2 years after the airport opened. Was it late or on-time? Does it matter when the business ends up going out of business? I’m currently involved in a similar fiasco. An internal project to replace a currently selling software product (that cannot possibly run on Vista) is 3 years behind schedule. The mismanagers are now claiming that this project is only 2 years old and therefore can’t possibly be 3 years late. My estimate is that there is about 6-9 more calendar months of work left before it is sellable, but the PHBs all say it must ship next month.

I’m coming to the conclusion that the reason that outsourcing/consulting is becoming (or is already?) a form of political patronage and graft. I’m running for elected office this November, and the more I see of the contracting/outsourcing of governmental functions, the more I smell graft.

As someone who has been on the purchasing side of a few public sector IT projects (mostly small-to-medium, but one or two big ones), I’ve always found that the biggest problem is that it is just no possible to accurately define the specs in advance, for a number of reasons. And as others have said, making changes later is incredibly difficult and expensive.

The reason is that the world is just not a static place, even (especially) in the civil service. Not only are there lots of stakeholders, their interests keep changing. Plus none of them can really know what they want until they see it.

Degree of difficulty rises disproportionately with project size, so I’ve learned the hard way that IT projects should be as small and modest as possible; never try to do too much. Also treat all tenderers as filthy liars who’ll promise the world until they get you locked into them, upon which they will proceed to merrily screw you for all the Treasury is worth.