Before I started grad school, I had gotten really into epistemology and philosophy of science. I’ve paid less attention to it as of late because my studies made it obvious that being an economist pretending to be a philosopher was not a good career path (even though some faculty members refer to my own work as being economic philosophy, which annoys me). But it was on my mind tonight, so here’s the way I like to think about how conclusive any piece of empirical evidence I come across is.
This is probably one of the most uncontroversial (it’s perhaps even banal) things I’ve ever written on this blog, but whatever. Note, however, that any of these types of evidence absolutely can change my mind, as what matters is Bayesian updating.
10. Anecdotal evidence. Storytelling. Narratives.
9. History that historians do, assuming they don’t quantify anything. Yes, this is where I would place nearly all libertarian histories.
8. Simple correlation.
7. Correlation with statistical significance.
6. The last one, plus controlling for rather obvious independent variables.
5. The last one, while also doing basic checks for things like the presence of heteroskedasticity (and then using robust standard errors, etc) or autocorrelation.
4. The last one, plus doing semi-plausible things about endogeniety, like most IV. This is how far majoring in economics will get you.
3. The last one, but actually dealing with endogeneity. This rarely happens. Sometimes IV or fixed effects can do it. Usually, it takes a natural experiment or random assignment to actually seem real. This is where I would place the work of Heckman.
2. The last one, but robust against Black Swans and crazy distributions. It’s very difficult to convincingly argue that the distributions in question behave in an orderly manner without assuming asymptotics. Non-parametric methods pretend to do this, but you can’t do real hypothesis testing with it, which puts you back at square one. Laboratory evidence and behavioral economics often does not have this problem, although it’s debatable whether they’ve always dealt with omitted variable bias correctly, which would push them back to #4.
1. Falsifiability. I know it’s a concept that isn’t entirely actionable, but the way I think about it is “make novel empirical predictions that go against nearly everyone’s intuitions but turn out to be true.” You have to be careful about the broken clock being right twice a day, e.g. Austrians “predicting” the Great Recession, but besides that it’s good intuition. This was how the theory of relativity gained much of its evidence. If you want an example in economics, I would point to Milton Friedman’s prediction that both high unemployment and high inflation were possible at the same time. Typically, economists won’t stick their necks out in such a way that this broadly conceived concept actually applies to them; every blog post Krugman publishes is very carefully worded so he can never be wrong. His antecedent, Paul Samuelson, stuck his neck out on issues like the growth of the Soviet Union and lost credibility.
I would call the last two “science.” If you want to be an economist and a scientist, your empirical work must fulfill either of the conditions (or both). If you want to provide evidence that changes people’s minds, you can still do any of the others since people should Bayesian update against it. The data that are typically posted on economics blogs fall into many of these categories, and they have changed they way we think about things. Scott Sumner is very good at this. Others, including some economists I otherwise respect, post things I would label an anecdote and don’t surprise me at all. When you find empirical evidence that falls under anything 2-8, I would not call it science. I would call it applied philosophy.