Hidden Margins of Error OR the Fallible Uncertainty of Nate Silver
A lot of very smart people are trying to get polling right. Here's why they might still be wrong. Plus: I offer my foolhardy prediction for the winner of the US presidential election.
Thank you for reading The Garden of Forking Paths. Much of this edition—including my foolhardy forecast for the US presidential election—is for paid subscribers only. My writing is fully reader-supported. So, to support my work and unlock this article, consider upgrading to a paid subscription for $4/month. Or, you could support my work by buying a copy of my new book, FLUKE, recently named an Amazon “best book of 2024.”
It’s that time again: doomscrolling the news while clicking refresh on FiveThirtyEight or Nate Silver’s new outfit—Silver Bulletin—rocking slowly back and forth in a fetal position while reassuring yourself that there’s a new good poll from Wisconsin.
Soon, it will worsen. You will find yourself on Election Night, slowly descending into Goblin Mode during the early hours of the following morning, glaring with trepidation at that scariest of digital widgets—The New York Times Election Needle. (It could make a good last minute Halloween costume)!
And boy, the polls are close! But here’s the thing: we simply don’t know if the race is close, or how close it is, or whether Harris or Trump is ahead. The reason for that is simple: the polls are fallible, and not just for the reasons you often hear about in the three minute soundbites about polling on cable TV news.
If you look “under the hood” of polling, it’s ugly—there’s a lot of stuff there that many pollsters are keen to skim over and that they hope nobody notices. That doesn’t mean they’re wrong about the overall state of the race, but it does mean polling is based on an ever-wider array of potentially incorrect subjective assumptions.
The upshot is this: Nobody knows what’s going to happen on November 5th. Not you, not me, not the betting markets, not Nate Silver.1 It’s impossible to know.
This isn’t because pollsters or forecasters are stupid; quite the opposite—they’re intelligent, thoughtful people trying to conduct nearly impossible wizardry. But despite their best efforts, the polls only tell us one thing with near certainty: it’s not likely to be a double-digit blowout, so either candidate could win. The rest is clouded in unresolvable uncertainty. But there are smarter and stupider ways to think about polling. It’s better to be on the smart end of the spectrum.
I’ve studied this stuff—and lived it. For about a decade, I’ve taught a postgraduate-level course about social research methods. I’ve also written a book laced with themes of chance, probability, and uncertainty, so I’ve thought deeply about the fallibility of presidential polling, the realm where citizens most routinely encounter attempts to forecast the future using unavoidably flawed statistics. And before I became an academic, I worked at a high level in US campaign politics, which included a fair bit of internal polling.
In this edition, I’m going to:
Walk you through how modern polls work (it’s not how you think and it has changed, even since 2020);
Give you the tools to think smarter about polling errors and explain why official margins of error are absurdly and unjustifiably overconfident;
Explain where polls end up going wrong with various forms of bias;
Argue that—believe it or not—we still don’t know how accurate past polls have been (and may not for some time);
Emphasize what’s unique about this presidential election that introduces new forms of polling uncertainty.
Finally, for reasons I explain in Fluke, I’m an evangelist for making firm predictions in social research, even when dealing with deeply uncertain forecasting. And so, dear reader, despite many essential caveats, I will conclude this edition with my personal forecast for who will likely win the 2024 presidential election—and how I’ve reached that conclusion based on the available data combined with my subjective assumptions about the race.
Keep reading with a 7-day free trial
Subscribe to The Garden of Forking Paths to keep reading this post and get 7 days of free access to the full post archives.