Bad models or bad modellers

“It was the presence (or, for those who predicted a that the bubble would burst, absence) of the assumption “house prices always go up”. ” I’m skeptical of this claim. IBs test their positions with all kinds of scenarios : surely they had at least several scenarios where housing prices go down. I imagine the problem is that they thought that, once the prices of their products started to go down, they could still unload all of them at the then-current price; instead, the market turned illiquid (the spreads widened and the bid prices were just too low or non-existent, as everyone was running for the exit at the same time).

In any case, I think it’s a fine line between (1) saying the models were wrong; and (2) saying the models were right but wrong assumptions were made about one of the parameters, housing prices. In any model involving many parameters – which includes pretty much any model of complex behaviour – some simplifying assumptions need to be made about the parameters, or else the number of possible scenarios become intractable. Is that the fault of the model? the parameters? Does it matter, when the end result is that complex processes are modeled incorrectly?

One should remain skeptical of models and the assumptions of a model, until they tested in both good conditions and bad. Holds for finance, holds for economics, holds for climatology. Be skeptical.

You can’t possibly model the correlations between 1 million mortgages or other financial transactions, so you have to assume they’re independent of each other, even though they’re not.

It’s not easy to model the correlations well, but that doesn’t mean you’re required to assume independence, and I’m not sure that that was really the problem here. Whether or not a model should reflect the correlations depends on what the model is used for. If you’re trying to come up with a fair price for a certain package of mortgage securities, the correlation between the individual mortgages may not be especially critical; if you managing the downside risk of your investments, then it’s absolutely essential. Risk managers have been aware of correlation for a very long time.

@Barbar

Wasn’t that presumed independence exactly the point? Mortgages payments were pooled together and then the rights to the first x dollars from the pool were sold as high level securities and the rights to the last y dollars from the pool were sold as junk.

That model only works if the mortgage payments have a payout probability independent of each other. If not – if something causes all those mortgages to default at once – you’re screwed (as was the case).

IBs model lots of things and interpret those models lots of different ways for lots of different audiences. Many are produced as sales aids. The model that prevails is the model that gets deals done. It is unlikely to be the one that tells everyone to go home. Investment banks were also new to lending — even a decade ago they didn’t really have full access to wholesale money markets and their balance sheets ballooned in the last few years before they burst. The right choice of model would not have been obvious and would have had to overcome models bringing in a lot of business.

Magistra, Adam — What Barbar said. Correlation has been a routine part of C(D/M/L)O pricing for some time. There are potential flaws with, for example, using Gaussian copulae and the models might not have fully reflected the risk of house price movement so correlation estimates may be flawed but these are much more subtle issues than assuming idependence and it is not sure that these problems are more serious than “garbage in garbage out” and inability to interpret the results.

Human factors have been included for ages. Pricing mortgages has involved making assumptions about the extent to which investors will not take full financial advantage of the options included in most US mortgages and refinancing in general. Indeed it is unlikely that an investor assuming full financial rationality on the part of borrowers would ever find themselves investing. Any appeal to behavioural factors will have to be more specific and also do more than calim to have said down when something else was saying up.

You don’t need to model the interactions between the mortgages, you need to model the system that the mortgages are a part of.

Insurers consider the probability of claims being dependent on each other all the time – Lloyds remained solvent after 9/11 because they had always considered two jet airliners colliding over NYC as their liquidity event they had to be able to meet.

In this case the insurers models demonstrated that the issuers of the ‘collateral default swaps’ were insufficiently capitalized from the start. That is why they are called default swaps, not insurance.

My business involves being an insurer, we have a pretty unique risk in that we have issued over a million policies with a maximum loss of a million dollars each and never seen a single claim in thirteen years. But we know that if we do see claims they are going to be highly correlated.

Once again J Thomas zeros in on the institutional psychology of the decision-makers
to good effect. Witness GM going bankrupt because no one is buying their cars because they cannot get financing (among other reasons) even as the credit arm of the company, (GM Acceptance) driven by attorney’s worried more about regulators than the health of the company that employs them, tightens credit requirements (credit score of 700+) to effectively prohibit anyone from buying their products except those capable of buying a Rolls-Royce–in which case they would not be in the market for GM products anyway.

Personally, I think that the problem all came down to the guys up at the top reaping 2-digit ($ millions) in annual bonuses for their ‘profitability’, and telling the middle and lowers what to assume. Those bonuses wouldn’t (and won’t) be clawed back in later years, if those ‘profits’ were all moonshine.

Follow the incentives.

Models, correlation, probability, behavioral responses– all this seems to be very sophisticated B.S.

If someone had simply looked at the loans–no down payments, interest only, no qualifications, etc. Simply applying occam’s razor (All other things being equal, the simplest solution is the best) would have predicted the current catastrophe. We were all in collusion and denial.

Could it be we owe much of our current predicament to the following, uh, “algorithm”?

(1) When market prices soar above model predictions, use mark-t0-market.
(2) When model prices soar above market prices, use mark-to-model.

(Substitute “sink below” for “soar above”, depending on the kind of financial instrument and how it yields. I’d like to say I’m sort of joking here, but with an automated feed for market prices, it would probably be hardly more than a half-page of code to automate this “algorithm” as a wrapper around existing modeling code, so that the trader at his workstation doesn’t even see a difference on the screen.)

Maybe this is how they did it at Fannie Mae? See WaPo story excerpted below. Of course, Fannie and Freddie have been fingered by conservative ideologues trying to establish some sweeping soshulist-gummint-did-it rationale for this crisis. How convenient. Perhaps a better case can be made for the proposition that we can learn something from the record of Fannie Mae (more regulated and thus more transparent) as it tried to play catch-up-game in the casino. One thing we might learn is that many of the big financial corporations were probably optimistically second-guessing their own models when those models indicated more prudent trades.

Susan Woodward, in “Rescued by Fannie Mae?” (WaPo Oct 14, 2008), suggested that we should actually use Fannie Mae’s existing models as a way to get home pricing on a better footing.

Fannie has an underwriting and valuation shop with models for valuing mortgages that are up and running. Key inputs to these models are Fannie’s own indexes of property values at the Zip code level — others make do with prices in entire urban areas. The data needed are not difficult to assemble: current loan balances, the date loans were originated, original property value (or appraisal for refinances), loan type (fixed, adjustable rate, ARM features, loan length), borrower’s credit score, Zip code and current loan status. These models project payments and proceeds from foreclosures and calculate the property’s present value. Similar but less detailed models in recent academic work show that they do quite a good job of projecting defaults for prime and subprime loans, given changes in property values.

Some argue that Fannie is discredited for this work because it, too, has losses on riskier mortgages. But Fannie’s losses arose from a failure to reserve adequately for losses that were anticipated by its models. Fannie’s business people overrode the risk managers when making the decision to keep reserves too low. The models were right.

It’s disturbing that modeling (and complex financial engineering in general) is being targeted so indiscriminately by the punditocracy. No doubt there were some bad models and some financial instruments whose complexities obscured their risks and flaws, but there’s still a lot of baby that would go out with the bath.

I’ve even read about how the problem is “derivatives”. Not even “complex derivatives.” Just “derivatives.” Jeez. So we should get rid of commodity futures markets, too, eh? Compared to this, Ned Ludd was positively nuanced.

The models used for valuation and risk (and ratings) were parameterised against market inputs (other prices) not by economic scenarios. The latter would determine how parameters would be perturbed, but could not calibrate the extent of the perturbation, which went well out of range of where the models had been (or could have been) tested.

The lack of due diligence on underlying loans and the feedback bubbling of housing prices, attributable to way easy credit and strong demand for securitisation, were exogenous to models whether for securities or for economies. But plenty of human economists picked up on it. (Some argued the circle was virtuous, others that viciousness was just around the next turn.)

Quite a lot of them certainly did use an assumption of independence; the whole point about overcollateralisation is that if the definition of a credit rating of X is a probability of default of Y per cent, and the average default probability of the loans in your bucket is Z (where Z<Y), to achieve a rating of X you toss in (Y-Z) per cent more loans above the face value of the security.

This has an implied assumption of independence whatever you do, because it’s only in the scenario where each default event is independent of the others that putting more of the same collateral in the pot will fix it. Otherwise, the extra loans (Y-Z) will just default at the same higher rate as the main pot, leaving you – or more importantly the buyer – screwed.

Actually, CT mangled that post pretty badly. Try again.

The whole point about overcollateralisation is that if the definition of a credit rating of X is a probability of default of Y per cent, and the average default probability of the loans in your bucket is Z (where Z < Y), you add (Y-Z) per cent more loans to the pool over and above the face value of the security.

Overcollateralisation is just having more nominal collateral than you need to meet your obligations. Overcollateralisation is, as suggested, used to boost the credit rating of portions of CDOs but even the rating agencies would use more elaborate maths than suggested above. The typical approach would be to come up loss distributions for each asset and then glue them together with a copula to give a combined loss distribution, that is explicitly not assuming independence. There are many inadequacies in this approach, some similar to the shortcomings of assuming independence, but it is not assuming independence which would be something quite specific. This approach is almost as well established as Black-Scholes and its limitations, smiles and what have you as well known. Nevertheless you might do quite a lot of hard work and still not have a firm grip on the role of house prices in your risk.

Also models tend to be tuned to particular risks. When the market is terribly competitive and things are good, focussing on tail risk might well lose you money in the short term. In the past that would have been fine for investment banks because they would have been long gone by the time there was trouble but that changed in the last few years and it is possible that the situation changed faster than the use of models. Pricing and risk management aren’t exactly the same.

Fannie’s business people overrode the risk managers when making the decision to keep reserves too low. The models were right.

This sounds oddly reminiscent of the Challenger (yes, the space shuttle). The engineers were concerned about cold-weather performance of the solid-rocket-booster seals, but were overruled by the managers, who decided to launch anyway. Boom (literally). And the whole thing (well, not the disaster itself, but the decisionmaking that led up to it) might have been covered up if not for one oddball physicist.

Imagine you are working for an investment bank and you put together a new model in, say, 2005, predicting that within a few years, declining real estate prices will prevent the continuous refinancing of these complex loan instruments, leading to higher default rates and an industry-wide death spiral of securities devaluation. Your bank is currently exposed enough to become insolvent if this happens. Do you take that model to your boss? How do you expect he would react if you did? Fire you and look for someone with a positive attitude who works well with others, would be my guess.

Maybe we need to give a little more respect to negativity.

P.S. HTML blockquote doesn’t show up in the instant preview, so I added the quote tag too. We’ll see how it comes out when posted.

Christian, funny that you mention aircraft design. Long-range flight, and space flight even more, does in fact have a significant multiplier effect: if you diminish weight (or something that can be trade-off against weight) by a kilo, you need less fuel, which in turn leads to a lighter structure, etc.

The result is that aircraft design indeed has much smaller safety margins than most engineering disciplines, and a much higher reliance on modelling to get every bit of performance out of a design. Still, aircraft have become extremely safe, while the financial world has completely lost control. It might be interesting to think why this difference exists.

My suspicion is the lack of (non-oligopolic) competition in aircraft industry. If there were a thousand aircraft manufacturers, including “hedge manufacturers” that rebuild existing planes to other specs, some more safe than others, a race to the safety bottom would be much more likely. As it is, companies gladly oblige the strongest safety regulations, because they know their competitors and know they adhere to the same rules, so they can pass on the cost to customers. The downside is massive overspending on safety, on the order of dozens of millions of dollars per life saved, much more than in most sectors.

Another aspect is PR: the aircraft industry knows that if flying was as safe as driving a car,
there would be massive crashes every few days, hurting the image of flying beyond repair. The financial world seems to fear such bad PR a lot less, perhaps because there are no human lives directly at stake, or perhaps it is less clear where to put the blame for financial crises.

Chris@27: One of the better analyses of the Challenger disaster was Diane Vaughan‘s, in her The Challenger Launch Decision: Risky Technology, Culture and Deviance at NASA. She described the basic social mechanism as “normalization of deviance” — the tendency, as time goes on without a disaster, to ignore warning signs that might have been initially troubling, even alarming.

Normalization of deviance is a real problem with anything bubble-related, because, even though it’s really not that hard to tell when you’re in a bubble if you’re sensitized to the signs, it’s virtually impossible to predict when bubbles will end. It’s almost a disadvantage to be sharply attuned to bubble behavior. You’re just a Chicken Little act to everyone around you, after a while. You’re not wrong. Catastrophe is coming. But you’re just a drag, and everybody comes to hate seeing your sorry depressive ass looming in the hallway.

Interestingly, Vaughan’s earlier work on how couples break up (Decoupling) provided her an invaluable clue for understanding the Challenger launch decision (and the Columbia disaster later — it was the same damned social process, just eating away at a different part of the vehicle hardware.) Couple breakups can take a long time, but isn’t it funny how you can sometimes tell with a particular couple that it’s happening, even if it seems they don’t? It’s because you’re getting a snapshot of what, for them, is a gradual process of “normalizing” increasing differences in their relationship.

I live in Japan, visiting the U.S. now and again. I would come back one year and see, “Wow, 0% financing for cars, something’s wrong here.” Another year, another visit, “Wow, 0% financing for mortgages, this is nuts!”

I had two advantages, however: (1) I’d seen a couple of housing bubbles already, and I’d had friends and family lose their shirts in them, and (2) I was getting time-lapsed snapshots. Without the “benefit” (?) of both, it might be arrogant of me to assume I could have seen what was happening.

We would do well to remember the wealth of economic interactions pursued involving alternative medical products, that clearly have no demonstrated benefit, and, to the contrary, may often have an elevated risks due to interactions with prescribed medications etc. Americans still spend billions a year on this quackery, and it is marketed to them in the guise of “freedom of choice” rather than “Studies demonstrating efficacy.” (When studies are provided, they generally don’t involve The New England Journal of Medicine)

In other words, they may be pursuing this care for reasons other than health maintenance, and if the claims are great enough, lack of evidence is minimalized. Also, a thick envelope of “alternative” spirituality usually accompanies the product; its therapeutic value cannot be determined by mere rationality.

Uninsured credit-default swaps, how can this work? (not to mention the side bets placed on them) Who cares, when the promises are nearly infinite; don’t be so negative and bourgoise in your demands for evidence…