Request for Literature: How come it’s so hard to explain Open Source to institutions?

by Clay Shirky on July 10, 2008

Another question about relevant social science literature, before they revoke my posting privileges:

I spent a lot of time talking to BigCos and people in the Gummint who are evaluating social software. Evaluating Open Source tools poses a problem for them, because they are cheap, simple to set up, and easy to use. These characteristics  defeat the normal IT evaluation process, which is supposed to start with an RFP, take the Office of the CIO 9 months to review the available features, and another 6 months of deployment and training. Free, easy, and good confuses them, and ‘Just try it and see how it goes’ is actively upsetting.

Part of the problem is price signaling — how could a free weblog tool, say, be better than this Very Expensive Content Management System? Part of the problem is feature creep — how could more features not be better? Part of the problem is the imperative for control — the stupider you think your employees are, the more features you will need to constrain their ability to act. Part of the problem is trusting vendors more than your own IT staff, and so on.

And, in the way of these things, the overall effect of this is an unspoken institutional commitment to the expensive and mediocre, simply because alternatives that may be cheap but good are assumed not to exist, and because there are tight social bonds with the vendors who sold them the last round of expensive stuff.

I bring this up because one of my former students, a very smart guy who used to work for the Gummint and now consults for same, is observing this same thing in his current work, and I wonder if anyone has studied this problem — not just price signaling, but the whole constellation of organizational behavior that makes evaluating Open Source tools in comparison with closed source so problematic?

{ 39 comments }

1

tom s. 07.10.08 at 11:49 am

My guess is that accountability is one of the big issues – not so much whether support is good or not in practice as to who is responsible in case of failure. They need “one neck to throttle” and with a lot of OS software it’s not clear whose neck that is.

But then you mention this in HCE re: perl, so I guess you already know that.

As an aside, my own experience with OS software is that “simple to set up, and easy to use” is not universally true – I’d say it’s not simpler than closed source as a rule.

2

tom s. 07.10.08 at 11:50 am

Oops – you asked for literature recommendations and I gave none. Too bad you can’t manage your own comments. Sorry.

3

abb1 07.10.08 at 12:15 pm

…and with a lot of OS software it’s not clear whose neck that is.

It’s always the CIO’s neck. If the CIO picks whatever is recommended by the forrester research, his neck is more or less safe whatever the failure; and if he picks a cheap but good alternative – he’s sitting on a barrel of gunpowder.

4

Cranky Observer 07.10.08 at 12:15 pm

> My guess is that accountability is one of
> the big issues – not so much whether support is
> good or not in practice as to who is responsible
> in case of failure. They need “one neck to
> throttle” and with a lot of OS software it’s not
> clear whose neck that is.

Having worked with enterprise-class systems and software for almost 20 years now, I find that (1) that you are correct about that perception (2) that the reality is that no entity smaller than Boeing has any leverage over any systems/software supplier, and that actual instances of “choking necks” and filing lawsuits do not exist. It is almost impossible to sue a software supplier and the rational business decision is always to walk away, absorb the loss, and start over. But in the next round the “I must have a neck to choke” argument will come right back…

You might want to look at this month’s issue of “CIO Magazine”: there is a short write-up on Virgin America Airlines and its CIO. His principle from day 1 was open source first, commercial only if no other choice. He did have the luxury of starting from absolute zero with a full enterprise-class budget and backing though.

Cranky Observer

5

john b 07.10.08 at 1:01 pm

abb1’s refinement of Tom’s argument is right on the money here. It’s simply “no-one got fired for buying IBM” syndrome; the main change in the industry over the last 30 years is that Accenture, EDS and CapGemini have joined IBM on the list.

6

Slocum 07.10.08 at 1:20 pm

Risk aversion is one reason (nobody ever got fired for buying …) Path dependency is another, probably stronger reason — it’s much simpler to transition to the new version of the system you’ve been using for years, whereas the cost of switching may be prohibitive.

But the risk aversion increasingly cuts both ways — decision makers may also face consequences if they purchase expensive proprietary systems and then their bosses read trade journal stories about other companies doing the same things for much less (it may get to the point where “Why aren’t we using free, open source software?” will become the equivalent of the ubiquitous “Why aren’t we using low-cost programmers in India” from a few years back).

7

Righteous Bubba 07.10.08 at 1:32 pm

It’s simply “no-one got fired for buying IBM” syndrome

But they should because Lotus Notes sucks.

8

tps12 07.10.08 at 1:48 pm

I don’t think it’s any harder than explaining anything else sensible to corporations, who will also regularly pay thousands of dollars for advice from outside consultants rather than avail themselves of the institutional knowledge of the human resources already on payroll.

9

Seth Finkelstein 07.10.08 at 1:59 pm

Oh, come on. This post might well have been “How come those stupid old stuck-in-the-muds don’t GET IT that there’s a wonderful new era Gift Economy where all is roses and unicorns. Can anyone point me to literature explaining why they’re soooo dumb?”
(“Free, easy, and good confuses them”)

Bleh. I’m a big advocate for open source software. But if you think it’s somehow sneer-worthy that “‘Just try it and see how it goes’ is actively upsetting”, well, that’s Kool-Aid pushing, not academics.

10

roac 07.10.08 at 2:06 pm

The proprietor of “Dr. Boli’s Celebrated Magazine” — possibly the funniest site on the Internet, the Onion not excepted — purports to sell something called “DBCOS” (Dr. Boli’s Celebrated Operating System). This, he explains, is Linux with a fancy logo on the cover, and is offered for something like $1000 an installation. The outrageous price is to enable you, as an IT manager, to get your superiors to buy it.

11

James Wimberley 07.10.08 at 2:08 pm

Open source software has been produced by a lot of people who haven’t been concentrating on their work. They have either been doing it on their employer’s time, or at weekends when they should ideally have been thinking up new ideas for their employer. Choosing open source signals employees that this behaviour is acceptable, even desirable. It’s the loss of psychological control, the Panopticon complex. Clever employers like Google and Pixar see that it’s in their interests for employees to “get a life”, but they are very much a minority as yet.

12

Dan Hirschman 07.10.08 at 2:10 pm

While it’s been about a year since I last looked, there is extremely little social science literature about anything related to open source. There are a few articles by game theorists/microeconomists trying to figure out why anyway would bother, a working paper by Kieran, and exactly one good book: Steven Weber, “The Success of Open Source”. I did not see anything that sounds like what you are looking for in the social sciences anyway, though again I last looked awhile ago, and it was just for a seminar paper so my search was nowhere near exhaustive. I think there is some orgtheory type work that I did not look at. Still it remains shocking to me how little is written about F/OSS by social scientists.

13

Barry 07.10.08 at 2:37 pm

Seth, you’re a good guy, but jeez, take your meds. What clay is pointing out is that there are alternatives to the standard pay-mucho-dinero-to-get-shafted system, and that corporations don’t seem to be that intersted [whether he’s right or wrong on the second point, I don’t know].

As for experimentation, it can be done, and is done every day by corporations. And as others have pointed out, hiring BigNewSoftCorp, Inc. to install very large corporate systems is a serious gamble. How many corporations have pissed away how many billions by having very large makeovers of IT systms in the past two decades?

14

Slocum 07.10.08 at 2:45 pm

And as others have pointed out, hiring BigNewSoftCorp, Inc. to install very large corporate systems is a serious gamble. How many corporations have pissed away how many billions by having very large makeovers of IT systms in the past two decades?

Open Source is not an alternative to that–the consultants (or internal staff) implementing new IT systems are going to be spending large sums whether or not the system is built on top of open source or not.

In fact, BigNewSoftCorp may have released all its formerly proprietary code into the public domain, expecting to earn the same consulting fees for installation and customization as always (and have the same potential for fuckups, delays, and cost-overruns as always).

In this way, the line between open source and closed systems is often blurry.

15

Chris E 07.10.08 at 2:46 pm

Simple. People naturally assume value is correlated with cost.

16

tom s. 07.10.08 at 2:49 pm

Well I did apologize for potentially derailing Clay’s request (@2) but it’s obviously well off the rails now so I’ll just keep it going.

CrankyObserver @4 says the reality is that no entity smaller than Boeing has any leverage over any systems/software supplier, and that actual instances of “choking necks” and filing lawsuits do not exist

Speaking as someone who is a neck during my day job (ie now but don’t tell my boss) that’s not true. We have customers who run into problems. They call our tech support people, who usually help them. Sometimes it’s not enough. Then they call the tech support manager, and so on up to the CEO if they are a really big customer, and then we (I’m in R&D) run around trying to fix the problem. It’s not infallible, but I can think of quite a few customers who have been very glad.

We may not get choked, but we sure get squeezed from time to time.

17

Tom 07.10.08 at 2:57 pm

As a consultant who implements systems based on open source technology, allow me to suggest that you hire a consultant who implements systems based on open source technology.

Seriously, though: downloading a tarball and handing it to IT is not the same thing as finding a vendor. The people who are wary are, to some extent, right to be wary. Hiring a professional to handle the project means you may not save as much money as you’d been hoping, but provides a level of support and institutional attention that frequently can’t be attained in-house. There are still good reasons to go with an open source technology when hiring an outside firm, though — not the least of which is that you can find other vendors/contractors to support it down the line should you want to stop using the original one.

If your application really is as simple and unimportant as a WordPress site, though, then it may be appropriate to undertake configuration in-house. My suggestion in that case is to pretend that getting the site live is an urgent priority. Stand up a simple FOSS installation — then present a plan for evaluating proprietary systems and, if necessary, transitioning to them. There’s no easier way to convince people that FOSS is sufficient than showing them.

18

Righteous Bubba 07.10.08 at 3:49 pm

Open source software has been produced by a lot of people who haven’t been concentrating on their work. They have either been doing it on their employer’s time, or at weekends when they should ideally have been thinking up new ideas for their employer. Choosing open source signals employees that this behaviour is acceptable, even desirable.

Just thought I’d mention that this is nutty.

19

Peter 07.10.08 at 4:22 pm

Slocum at #6: Risk aversion and path dependence are not the only reasons for the prevalence of: “Nobody gets fired for buying IBM”. There are two other good reasons, neither of which involves the belittlement of the intelligence, capabilities or judgment of CIOs.

1. Major purchase decisions are often (perhaps almost always) made under time pressures and in conditions of uncertainty. In such circumstances it can be cost-effective, efficient and rational to simply choose whatever option most of your peers are currently choosing.

2. For many (again, perhaps most) products there are network-good effects, since fashion comes into the buying and use of nearly all products (even raw materials commodites). When these effects exist, the utility of one purchaser’s choices depends on the choices made by other purchasers. (Note that this is something different to path dependence.)
Once again, it can be rational for a decision-maker to choose what other people are currently choosing.

20

Peter 07.10.08 at 4:26 pm

chris e at #15: “Simple. People naturally assume value is correlated with cost.”

In the absence of any other information regarding value of a product or service, this assumption is perfectly rational. Question: Why do strategic management consultants charge such high fees? Answer: Because it is usually not possible to assess the quality of their advice.

21

abb1 07.10.08 at 4:35 pm

@6: …trade journal stories about other companies doing the same things for much less…

I think that’s true, the ibm is (in a sense) coincidental, it’s all about the zeitgeist created by forrester scribes, who could be writing about the splendor of opensource just the same as splendor of the ibm.

But it’s a huge hurdle to overcome without a capital, lobbying infrastructure, etc. Opensource can’t bribe them, so, I’m afraid, the only solution is to discredit them, to intimidate them, to get rid of them.

22

Slocum 07.10.08 at 5:21 pm

Opensource can’t bribe them

Not really true — major open source software projects often have the backing of industry heavyweights (IBM, Sun, HP, RedHat, Google). For example:

http://www.eclipse.org/org/
http://www.sun.com/software/star/openoffice/
http://www.redhat.com/partners/partnerspotlight/hp/
http://blogs.zdnet.com/BTL/?p=6715

The idea that major open source projects are supported mainly by armies of amateurs working away in their basements is far from reality.

23

oab 07.10.08 at 5:44 pm

It may be worth noting that the same effect can also be found between expensive and cheap comercial software as well. In a large organsation managing to convince everyone that a $200 SW is as good/better than a $5000 (or $50 000) can be very difficult. There are two reasons for this (that I know of, there are probably lots more)
Firstly the $200 SW does not come with a sales organisation that does all the stuff that is expected/wanted: Answering RFIs and RFQs in detail, coming to several on-sites visits, doing test installations etc.
Secondly, and perhaps even more important, the feeling that a $5000 SW must be better is very human and difficult to overcome, especially in large organsations where the actual decision is far removed from the users.

24

Stuart 07.10.08 at 6:01 pm

In the absence of any other information regarding value of a product or service, this assumption is perfectly rational.

Not for IT solutions it isn’t – the main factors of the cost of new software is primarily loss of productivity while users adjust, and the training costs (factoring in both time away from productive work and cost of courses). Any business that isn’t a complete IT disaster zone is well aware of this, and the software price tag is usually a frankly minor point compared to estimating the other costs (for the more common applications anyway, some specialist relatively low volume products can be quite expensive).

25

Barry 07.10.08 at 7:55 pm

Re: #23:
(from http://www.joelonsoftware.com/articles/CamelsandRubberDuckies.html, discussing how to price software, if you’re selling it)

“Software is priced three ways: free, cheap, and dear:

Free. Open source, etc. Not relevant to the current discussion. Nothing to see here. Move along.

Cheap. $10 – $1000, sold to a very large number of people at a low price without a salesforce. Most shrinkwrapped consumer and small business software falls into this category.

Dear. $75,000 – $1,000,000, sold to a handful of rich big companies using a team of slick salespeople that do six months of intense PowerPoint just to get one goddamn sale. The Oracle model.”

26

Clay Shirky 07.10.08 at 9:03 pm

Re Tom #17 _If your application really is as simple and unimportant as a WordPress site, though, then it may be appropriate to undertake configuration in-house._

The class of dilemma I am interested in is when your application is as simple and _important_ as a WordPress site.

The question is one of leverage — software is important for its effects, so if the goal is social effects (more talking, more shared awareness, more co-editing, whatever) then the simpler apps may work better. If we were just in a pair of boxes labled ‘Complex and Important’ and ‘Simple and Unimportant’, there wouldn’t be any issue.

James #11: Open Source is often produced by people paid to produce Open Source (http://www.ibm.com/developerworks/linux/ltc/), because people of sound technical judgment recommend it (http://www.ameinfo.com/69674.html)

And Seth #9, I never use the words Gift Economy, because they are not merely hollow, they actively obscure the real economics at work.

And the point of ‘Just try it and see how it goes’ is a telegraphic version of the observation from Here Comes Everybody that the great advantage of many of these systems is that the cost of failure is so low. I am specifically *not* saying “Just adopt simple solutions because its the Right Thing To Do”; sometimes it is an sometimes it isn’t.

Instead, I am saying “One way to decide whether to adopt a piece of software is to study it; another is to try it.” I have observed, among people adopting software, that trying something that may fail is seen as inferior to studying something that may fail, even when trying it is both a lower-cost and higher-value test, precisely because studying something that may fail is only a proxy for trying it, so you can still get the green light for something that may not work.

I think that the observation about Boeing is right on — people put faith in contractual relations, even when they provide zero leverage, as with Windows. what I am wondering is why that it. It may be that failure aversion is so built in, to either minds or social systems, that any cost-benefit analysis will overestimate the willingness to try things as a way of telling whether they work or not.

27

Peter 07.10.08 at 9:49 pm

Clay –

I think you are conflating lots of different types of software in your most recent comment. What you say applies to stand-alone, single-user applications without any consequences for the organization’s mission or processes, and without major interactions with legacy systems and applications. But these are not properties of most major commercial systems. Exactly how does Boeing “try” a new enterprise-wide logistics system before deciding to use it? Exactly how does AT&T “try” new network switching management software without actually deploying it? Even the mere process of installing a complex, multi-user, multi-purpose, multi-site, mission-critical application in order to adequately evalute it may take a team of a score or more people to plan, let alone try.

This behaviour is not risk-aversion or stupidity on the part of CIOs, it is rational commonsense. Corporations have good reason to be cautious about messing with their mission-critical systems and operations.

28

abb1 07.10.08 at 10:09 pm

I don’t think there is an opensource enterprise-wide logistics system for Boeing, but it can be a relatively complex multi-user system and he gave an example: content management system. Also, quite often subdivisions of big companies have their own ERP systems and it doesn’t have to be SAP or Oracle.

29

Seth Finkelstein 07.10.08 at 10:14 pm

Clay, let me semi-apologize because this kind of stuff sets my teeth on edge, and that’s particularly relevant here because I’ve been a hardcore Linux advocate as a programmer. So I’ve run into this situation personally. The sort of la-la land implicit above is the easiest way to lose. You didn’t say “all is roses and unicorns” either. The point was your post implicitly assumes Open Source systems are so obvious better in a broad sense that any organization not rushing to immediately convert at all costs must be staffed by fearful timid corrupt dullards.

I know that sells on the conference-club. But it’s really irritating the next time I have to deal with a manager and explain no, it’s not about some sort of anti-corporate ivory-tower ideology.

To amplify what peter just said at #27, running a large project in corporate management is very different from what’s valued by a perpetual-beta early-adopter mentality.

30

Stephen Judd 07.10.08 at 10:39 pm

The literature you are looking for is Veblen.

The purchase of expensive, non-performant software is a form of conspicuous consumption. Status accrues to people in control of large budgets who spend it on visibly expensive things.

You can view the expensive lunches, fact-finding tours and other freebies that typically accompany very large enterprise software purchase as a form of potlatch.

31

Tom 07.11.08 at 1:46 am

The class of dilemma I am interested in is when your application is as simple and important as a WordPress site.

The question is one of leverage—software is important for its effects, so if the goal is social effects (more talking, more shared awareness, more co-editing, whatever) then the simpler apps may work better. If we were just in a pair of boxes labled ‘Complex and Important’ and ‘Simple and Unimportant’, there wouldn’t be any issue.

I may have spoken imprecisely. When I say simple and unimportant I’m referring to the choice of software specifically — of course its utilization may serve a very important role for a given organization. But how hard would it be to replace if it broke, or was found lacking, or you discovered you couldn’t support it within the organization? If it’s a blog the answer is “not very”. There isn’t a big downside to rolling the dice and seeing how it works.

Contrast this with, say, the selection of a VoIP system for your office. The phone system may be much less important to an enterprise than its website, but switching phone technologies is likely to be a much bigger pain than switching from WordPress to Movable Type.

I say all this from experience, incidentally. We could be forced to migrate a dozen WordPress sites to Drupal, MT or whatever before it’d be half the hassle that dealing with our FOSS VoIP system (Trixbox) has been, although on a day to day basis the phones are far less important.

32

vader 07.11.08 at 2:59 am

I was teaching a class on open software today. A piece of FOSS software failed with a lot of strange messages. In recovering, you had to do a bit technical stuff that was really obscure to do but clears the problem. On the third reboot the student hit escape because a message that was supposed to be ignored told her to hit escape. The student left the class because it was really too much for her.

The promise of open software that is not controlled by IT far exceeds the reality. It works in 90-95% of the cases, but in business where zealots and techies are not about, the 5-10% failure rate kills the reputation of the software. It often needs obtuse and obscure commands to operate.

When I inquired a local Linux user group on what to teach newbies, there were 2 responses. One was to teach them how to research problems on the web using wiki and google and the second referred me to a 10,500 word essay teaching newbies how to ask linux folks questions without pissing the linux folks off.

This will not work in the business community. If you look at OSes, Linux has a 3% market. You cannot get more market if ignore the customer is your business model.

Open source supported by consultants to handhold customers can wonderful. But bring it without the hand holding and it fails, then its reputation is ruined forever. Folks want to get the job done. Less expense is nice, but not at the cost of getting work done.

After all you are competing against something that does work. You are bring in change and change has few allies. When it fails to meet expectations, you are lost.

33

Matt Lungerhausen 07.11.08 at 3:07 am

Hmm. I am not sure I can add to the thread, but I am grateful for the post. My university has been moving from D2L to Blackboard for the past two years. Its painful. Both software packages under perform and are difficult to use. Needless to say my school is part of ‘large state colleges and university system’ with a central bureaucracy in the state capital that micromanges IT for all the branch campuses. (When ever power gets cut to the central servers all the schools go down…)

My buddy who teaches at a small liberal arts college down the road is using moodle and loves it. Its amazing, flexible and cheap. We can’t have moodle, or even our own wiki server, because the central office has forbidden it. We are all using D2L and thats that.

One last thing, the central office’s IT budget has grown steadily while tuition has gone up and salaries have staid flat, system wide for the past decade. But we have D2L… which just got sued by Blackboard.

Phooey.

34

Matt Lungerhausen 07.11.08 at 3:09 am

Whoops, I meant to say that we were forced to migrate from Blackboard to D2L. That would make more sense.

35

Ginger Yellow 07.11.08 at 3:30 am

“Having worked with enterprise-class systems and software for almost 20 years now, I find that (1) that you are correct about that perception (2) that the reality is that no entity smaller than Boeing has any leverage over any systems/software supplier, and that actual instances of “choking necks” and filing lawsuits do not exist.”

The perfect example of this is the disastrous NHS IT overhaul. Admittedly, it wasn’t helped by the most ill thought out procurement process in IT history, but it was made even worse by the fact that the IT providers have control over the code, so they can’t be turfed off for underperformance without having to start all over again.

36

Chris Bertram 07.11.08 at 7:04 am

I’m a pretty determined non-Microsoft user: write my stuff using LaTeX etc. But for most end-users, I have to admit that the Microsoft product works better than the open-source alternative which is clunky and unreliable.

Most of my colleagues write using Microsoft Word and the university also uses Excel quite a bit. When you compare these products to the OpenOffice alternatives, then there’s no contest (ditto NeoOffice for Mac users).

If my University, in the interest of cost savings, tried to push everyone into using OpenOffice, I imagine there’d be uproar.

37

CKD 07.11.08 at 3:10 pm

As someone who previously worked in government procurement, I can tell you that the answer is really quite simple. There is no identifiable supplier who can be held liable if there is a serious problem. The U.S. Government is actually prohibited from accepting volunteer services, and it is the same sort of mentality that keeps it from using open source.

38

A-ro 07.13.08 at 2:23 pm

The cost of failure is not actually that low for some gov’t entities. Unlike, say, a meetup or flickr group, they have to keep records and be permanently accountable for what happens. Isn’t the fact that internet groups don’t have to be durable part of why the cost of failure is so low?

39

Ray Davis 07.14.08 at 5:29 pm

Not exactly scholarly literature, but here are some notes from a talk on the subject by Ira Fuchs, whose work at the Mellon Foundation helps fund some higher-education-centric open source projects.

Comments on this entry are closed.