Increasing Marginal Utility

A blog so good it violates the law of diminishing marginal utility.

A Hierarchy of Empirical Evidence

Before I started grad school, I had gotten really into epistemology and philosophy of science. I’ve paid less attention to it as of late because my studies made it obvious that being an economist pretending to be a philosopher was not a good career path (even though some faculty members refer to my own work as being economic philosophy, which annoys me). But it was on my mind tonight, so here’s the way I like to think about how conclusive any piece of empirical evidence I come across is.

This is probably one of the most uncontroversial (it’s perhaps even banal) things I’ve ever written on this blog, but whatever. Note, however, that any of these types of evidence absolutely can change my mind, as what matters is Bayesian updating.

10. Anecdotal evidence. Storytelling. Narratives.

9. History that historians do, assuming they don’t quantify anything. Yes, this is where I would place nearly all libertarian histories.

8. Simple correlation.

7. Correlation with statistical significance.

6. The last one, plus controlling for rather obvious independent variables.

5. The last one, while also doing basic checks for things like the presence of heteroskedasticity (and then using robust standard errors, etc) or autocorrelation.

4. The last one, plus doing semi-plausible things about endogeniety, like most IV. This is how far majoring in economics will get you.

3. The last one, but actually dealing with endogeneity. This rarely happens. Sometimes IV or fixed effects can do it. Usually, it takes a natural experiment or random assignment to actually seem real. This is where I would place the work of Heckman.

2. The last one, but robust against Black Swans and crazy distributions. It’s very difficult to convincingly argue that the distributions in question behave in an orderly manner without assuming asymptotics. Non-parametric methods pretend to do this, but you can’t do real hypothesis testing with it, which puts you back at square one. Laboratory evidence and behavioral economics often does not have this problem, although it’s debatable whether they’ve always dealt with omitted variable bias correctly, which would push them back to #4.

1. Falsifiability. I know it’s a concept that isn’t entirely actionable, but the way I think about it is “make novel empirical predictions that go against nearly everyone’s intuitions but turn out to be true.” You have to be careful about the broken clock being right twice a day, e.g. Austrians “predicting” the Great Recession, but besides that it’s good intuition. This was how the theory of relativity gained much of its evidence. If you want an example in economics, I would point to Milton Friedman’s prediction that both high unemployment and high inflation were possible at the same time. Typically, economists won’t stick their necks out in such a way that this broadly conceived concept actually applies to them; every blog post Krugman publishes is very carefully worded so he can never be wrong. His antecedent, Paul Samuelson, stuck his neck out on issues like the growth of the Soviet Union and lost credibility.


I would call the last two “science.” If you want to be an economist and a scientist, your empirical work must fulfill either of the conditions (or both). If you want to provide evidence that changes people’s minds, you can still do any of the others since people should Bayesian update against it. The data that are typically posted on economics blogs fall into many of these categories, and they have changed they way we think about things. Scott Sumner is very good at this. Others, including some economists I otherwise respect, post things I would label an anecdote and don’t surprise me at all. When you find empirical evidence that falls under anything 2-8, I would not call it science. I would call it applied philosophy.

5 responses to “A Hierarchy of Empirical Evidence

  1. Daniel Kuehn October 19, 2011 at 8:04 am

    Anything under 2-8 is not science? That seems like a very high bar. So you are saying only falsifiability? Do any sciences other than physics do this? Do all physicists do this?

    I like the way you start out #3 :)

    • rhmurphy October 19, 2011 at 12:11 pm

      If you have realistically dealt with endogeneity and fat tails, I would say econometrics becomes scientific, which is number 2. Physics and chemistry certainly clear these bars because endogeneity doesn’t matter usually, and they have tools for dealing with the complexity associated with fat tails. The primary economists who use those tools are connected with the Sante Fe Institute. You can make the argument that Hayek was scientific; he connects his thinking directly with Popper’s and describes falsifiable pattern prediction, which is essentially what I described in falsifiability. Economics can sometimes be truly scientific, but when formalistic economics is, it’s because they got a natural experiment going for them or something. When you aren’t formalistic, you can still clear the bar set in number 1 with pattern prediction, or as your BFF would say, being vaguely right instead of precisely wrong.

  2. Gene Callahan October 19, 2011 at 12:51 pm

    9) Historical conclusions are (or can be) every bit as certain as anything in physics. Quantifying things has nothing to do with it.

  3. Gene Callahan October 19, 2011 at 12:58 pm

    Oh, and to really get your goat: the conclusions of historians (can be) much, much more certain than anything economists arrive at using econometrics. Instead of taking Taleb’s word for what historians do, it is much better to actually study what they really do!

    • rhmurphy October 19, 2011 at 2:18 pm

      They can’t establish causality without assuming a theory applies. How do you determine which of many competing explanations were the important ones without statistical testing? Without it, you just read in whatever explanation you want into the “facts,” the selection of which also came from your theory. It’s not that I believe that you can actually do this in real life on a regular basis, but unless you do, you have no idea what is causality and what is coincidence.

      I get what I’m saying from Popper more than Taleb.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


Get every new post delivered to your Inbox.

Join 810 other followers

%d bloggers like this: