FiveThirtyEight’s Data Problem

Nate Silver’s new FiveThirtyEight has been catching a lot of flak since it launched last week. Perhaps the harshest has been directed at the site’s retention of the often-contrarian climate analyst Roger Pielke Jr., with everyone from Paul Krugman to the Center for American Progresspiling on. The onslaught is disturbing. I’ve disagreed with Roger often, but he is genuinely well intentioned. People who care about getting good policy should want more thoughtful voices, not fewer, proposing options – and organized campaigns to run heterodox thinkers out of town are awfully ugly.

But that doesn’t mean I’m impressed with the new FiveThirtyEight. Indeed it’s another energy post – “U.S. and Chinese Current Accounts Converging” – that’s troubling.

The post starts out with an arresting chart (reprinted at the top of this post) that shows an impressive improvement in the U.S. current account balance. The author then explains what’s happened:

“In the U.S., a natural gas boom is cooling demand for imported petroleum, and oil represents a huge share of American imports. The dollar, meanwhile, has depreciated, boosting American exports.”

This is certainly the conventional wisdom. But what one expects from FiveThirtyEight is a data-driven interrogation of that conventional wisdom. Alas that would have produced a different result.

Here’s another chart that combines the data presented by FiveThirtyEight (the blue line – it looks different because I’m showing absolute numbers rather than percentage of GDP) with three other series. Start by comparing the blue and red lines. These seem to move together beginning in 2009. In particular, what you’ll see is that the big shift in the current account comes in 2008-9. Indeed the decline in the oil trade balance (the red line) accounts for a little more than half the decline in the current account balance.

So far so good: changes in oil trade seem to explain a lot of what’s happening with the current account.

But now look at the dashed lines.

The green dashed line shows that net petroleum and product imports have been declining steadily since about 2007. This raises a first problem with FiveThirtyEight’s analysis: if falling oil imports explain so much of the lower current account deficit, why hasn’t the current account deficit continued to plunge alongside oil imports? The answer is pretty straightforward: oil prices have gone up. The United States imported 64 percent as much oil in 2013 as in 2009, but the average cost of an imported barrel was 66 percent higher.

It’s the purple dashed line, though, that’s the most damning. What that line shows is that in the span that the U.S. current account balance really shifted – 2008-9 – U.S. oil production had barely begun to pick up. It was only later, beginning in 2011-12, that output really took off. But that’s not the period when the big change in the current account balance appeared. (Had I added in natural gas liquids, which started rising earlier, the comparison would look a little better, but the basic problems would remain.) Had FiveThirtyEight actually juxtaposed the current account balance with oil trade data, it wouldn’t have ratified the conventional wisdom.

Indeed one shouldn’t be surprised with this result. As Robert Lawrence showed in a paper that my program published in January, there are strong theoretical reasons to be skeptical of claims that falling U.S. oil imports and rising U.S. oil production will substantially reduce the U.S. current account deficit.

Which gets to the heart of FiveThirtyEight’s challenge. They want to do theory-free data analysis. But without at least some sort of theory, you at best need to investigate a much larger volume of data in order to get useful results. (That’s what Nate Silver did so well with election predictions and baseball statistics.) At worst, if your data isn’t good enough, theory-free analysis leaves you with nothing. Here’s hoping that FiveThirtyEight will be disciplined enough to stick to analyses where its unusual approach works.