The Perils of Moneyballing Everything

Data analytics are increasingly defining our world. In some systems, that's wise and wonderful. In others, it laces catastrophic risk into a complex world that's more Calvinball than Moneyball.

Thank you for reading The Garden of Forking Paths. This article is for everyone, but if you’d like to support my work—please consider upgrading to a paid subscription for just $4/month. Or, you could support my work by buying a copy of my new book, FLUKE: Chance, Chaos, and Why Everything We Do Matters.

---

This is a story about the hubris of data analytics; about the perils of treating Calvinball problems as though they are Moneyball problems; and how modern attempts to control our world have fundamentally misunderstood the nature of the complex systems that govern our lives.

It’s also a story of cultural sludge, convergence toward boring averages, and the catastrophic consequences that occur when you treat mysteries as though they are puzzles. In short, this is the story of how and why we misunderstand modern society in the age of Big Data—and how we can learn to be smarter.

I: Moneyball and the Wizardry of Analytics

When I was a kid, I learned to read partly by perusing the Minnesota Twins’ baseball box scores—the original mechanism for quantifying sports and translating on-field performances into numbers that could be analyzed and compared. It was a crude method, capturing hits, runs, and averages for individual ballplayers.

But crucially, box scores focused on individuals, not systems. You could read a box score and figure out how a player was doing, but not understand what made a winning team. For more than a century, baseball—like most sports—remained stuck in this primitive form of atomized analysis.

With only crude stats as a guide, players were drafted based on hunches. Coaches followed folk superstitions. Front offices poured most of their budget into big name stars to please fans, even though the result often was just a highly paid All Star on a losing team. For the baseball gurus, numbers could never replace fundamentals and intangibles, the good or bad feeling a scout got when they watched a prospective player up close.

In the 1960s and 1970s, Earnshaw Cook, a professor at Johns Hopkins, and Bill James, a security guard at a bean canning plant, began to evangelize about the value of more robust data analytics for baseball. Nobody listened. Bill James initially found just 75 buyers for his statistical analysis, Baseball Abstract. They persisted in obscurity.

Then, as Michael Lewis documented in his hit book-turned-Brad-Pitt-film Moneyball, the Oakland Athletics under Billy Beane decided to harness the power of data—to spend less money but win more.

The thesis of Moneyball is that Beane outsmarted the competition because he realized that wins are made by teams, not by individual stars. Using sophisticated data analytics, the Oakland A’s were able to spot value where others overlooked it, realizing that understanding a team, like understanding any system, often requires more than just analyzing constituent parts. This systems-based, data-driven approach—aided by so-called sabermetrics—revolutionized the game, making statistics central to modern sports. And it worked: new data tools were really effective at making teams win. The analytics boom transformed sports.

Moneyball was the book and film that launched a thousand insufferable LinkedIn posts. “Everything should be moneyballed,” proclaimed the hordes of business bros. Trying to drive your lagging Q3 profits through the roof? No success on the dating apps? Well, why don’t you step away from your typewriter, join the modern world, and take a page out of the Billy Beane playbook, you antediluvian fool!

The world, the LinkedIn bros claimed, was only broken because antiquated idiots were ignoring the obvious lesson that Michael Lewis had spotted: if you just deploy data analytics on any problem, you need only wait for the Poof! sound as it’s magically fixed before your eyes. Modern wizards should no longer wrap their fingers around wands, but extend them to dance across keyboards, unleashing a powerful form of 21st century digital magic: Excel formulas.

Now, I don’t want to alarm you, but if you were to take a little look around you, it may not escape your notice that, even two decades after Moneyball, the world has not been magically fixed.

What went wrong? Are all of our problems simply derived from the sad fact that the powerful evangelism of the business bros has been callously ignored?

II: Treating mysteries as puzzles will destroy us

Alas, our problems run deeper and cannot be solved with LinkedIn wisdom. Data analytics are revolutionary in some contexts, solving problems that were previously impossible to tame.¹ But in other situations, worshipping analytics as a digital gospel creates avoidable catastrophic risks.

How can we spot the difference?

In their book, Radical Uncertainty, Mervyn King² and John Kay draw a distinction between two kinds of problems: puzzles and mysteries.

Puzzles are easy.

They’re well-defined problems, often with clear-cut rules, and a straightforward set of possible outcomes. This is the Land of Nate Silver, where data analytics offer the best guide and better number crunching is always the answer.

Kay and King highlight that puzzles are easy because the underlying process is more or less understood; the nature of cause and effect remains constant over time; and the process isn’t altered by our beliefs or behavior. NASA missions offer a puzzle. The calculations have to be perfect, but if they are, a spacecraft can land on a distant asteroid and there will be no mystery as to why it worked. Puzzles can be solved in a relatively straightforward way—with more data, better models, smarter analytics. Some puzzles are harder than others, but they follow those general principles.

Baseball is a classic puzzle. The rules are basically unchanging. There are two teams competing. The same teams battle it out season after season. And there are only two outcomes: Team A or Team B will win. A third team can’t win. Rules can’t change midway through. There’s no conceivable way that the game will last four days or that one team will score 1,000 runs. We know, with certainty, that the 9th inning won’t spark a political revolution. It’s contained, constrained, with a small set of possible, repeatable outcomes. It’s more akin to a series of coin flips—we understand the causal dynamics and the more you do it, the better you understand what’s going on.

In the realm of puzzles, past patterns offer an accurate guide to future outcomes. Moneyball works in those situations, so the LinkedIn bros are proven right.

Mysteries, by contrast, are hard.

These are loosely defined problems with radically uncertain outcomes.³ “What will the world look like in 2050?” is a mystery, not a puzzle. You can’t just feed data into an Excel spreadsheet and find out the answer. The limitations of our understanding within the realm of mysteries becomes particularly clear when you consider that every forecast for 2020 and beyond—even those made months earlier—was obliterated by the covid-19 pandemic.

In the realm of mysteries, the past offers no credible guide to future outcomes. In some areas of life, we don’t—and can’t—know what’s going to happen, no matter how adept we become with spreadsheets and statistics. (Trying to determine, right now, who will win between Trump and Biden is a mystery, not a puzzle. It’s literally impossible to know).

King and Kay, in Radical Uncertainty, accurately diagnose a key problem: we treat too many mysteries as though they are puzzles. They use the analogy of a man stumbling home drunk, having lost his keys. He begins his search on a dark road, gravitating toward the area of pavement bathed in the glow of a lone streetlamp. “Is that where you lost your keys?” his friend asks. “No,” the drunk replies, “but the light’s better here.” Just because a strategy is possible doesn’t mean it’s wise.

That analogy helps explain why perils lurk when we confuse a mystery for a puzzle, deploying data analytics as the sole tool for decision-making in a radically uncertain environment. When that happens, it often blows up in our faces, a human-engineered catastrophe borne from the hubris of misapplied Big Data. Powerful tools used incorrectly are dangerous.⁴ They can make us behave with arrogant certainty in the face of radical uncertainty.

The smart strategy, then, isn’t to Moneyball everything, but rather to recognize when we are in a world that’s more akin to Calvinball—and then to tread carefully.

III: Closed systems, open systems, and Calvinball

In the much-beloved comic “Calvin and Hobbes,” the young boy and his pet tiger play a game known as Calvinball. The game has precisely two rules:

1. The rules must be different every time.

2. All players must wear a mask at all times.

While both rules applied best to the unsettling masking period during the pandemic, the central notion of Calvinball is the opposite of Moneyball: if the rules are constantly changing, then data analytics will be useless. After all, past performance is no guide for future success. You could win Tuesday’s Calvinball game by being a fast runner, only to have that same speed turn into a hindrance under the Wednesday rules. Past isn’t prologue.

A completely unique game of Calvinball.

Calvinball therefore presents a world of non-stationarity, where the patterns of cause and effect are constantly shifting. You might think that you’ve cracked the code by inferring patterns from previous games, but you’d only be fooling yourself. The patterns would be illusory, subject to constant flux, and if you bet on Calvinball based on those analytics, you’d eventually lose a lot of money. Forecasting Calvinball is impossible; the game will always be a mystery, never a puzzle—no matter the computing power.

In many realms of the world, we pretend that we are playing by the rules of Moneyball when we’re actually playing Calvinball. Our world is defined by non-stationarity. So, how do we grapple with this problem? Well, I regret to inform you that most of the time, we just pretend it doesn’t exist. That’s why many of you have never heard the phrase “non-stationarity” before this instant.

Most models of our insanely complex world also pretend that the world is a stationary closed system, a make-believe fantasyland with stable cause and effect relationships, a finite set of inputs, and a constrained set of outputs.

By contrast, the real world is an open system, in which cause and effect relationships are ever-changing, reality is far more chaotic than any model can process, and (so long as events follow the laws of physics), just about anything can happen. As with Calvinball, the same event can play out with totally different rules and outcomes, across time and space.

What might have happened if covid-19 had struck in 1990 instead of 2020? Imagine the exact same virus, infecting someone in Wuhan, but thirty years earlier. Everything would have unfolded differently. For starters, in 1990, just three million people had access to the internet. Working from home would have been impossible. The economic and social effects of the same virus would have diverged immensely.

But that’s just the beginning. Who knows what would have happened with the collapse of the Soviet Union if the pandemic hit right before the 1990 mass protests began? It’s impossible to say. In a chaotic world, changing anything can change everything.

And it’s not just decades that need to pass for our world to lurch into Calvinball territory. If September 11th, 2001 had been September 10th, 2001, some of the hijackers might have never taken off, stuck on delayed flights. There were storms on the 10th, but blindingly blue skies on the 11th. Moreover, as I wrote in Fluke:

The passengers on United Airlines Flight 93 took down their hijacked plane before it could reach its intended target, but it’s perfectly plausible that a different set of passengers on September 10th or 12th might have acted differently—and the White House or U.S. Capitol might’ve been destroyed.

Consider this question:

Would the attack on January 6th, 2021 have happened if the US Capitol had previously been blown up by a hijacked airplane two decades earlier?

Who knows! Our world would be different if the US Capitol had become rubble. But it’s absurd to believe that data analytics could help answer that question. It’s an area of radical uncertainty, not one in which better spreadsheets or regressions can help you find the right answer. A mystery—not a puzzle.

And yet, modern life is governed, to an alarming extent, by those who are using Moneyball tactics to solve Calvinball problems. It laces catastrophic risk into our societies. Why? Because we think we know the answer to questions that are fundamentally unknowable. And that produces hubristic decision-making.

IV: The optimization / resilience trade-off

In Moneyball systems—the stable world of puzzles—there is limited downside to ever-greater analytics. NASA, for example, wants to converge toward perfect precision within the stable systems of space objects. If the rocket nerds can get even 0.0001 percent more accurate in their measurements, that could be decisive in their ability to land a human-made object on a distant rock hurtling through space while following the predictable laws of physics. For NASA, there’s no downside to better data, fed through improved analytics, exploited more effectively.

But in most complex systems—the social systems we inhabit and use to govern ourselves—there is a tradeoff between optimization and resilience. Puzzles can be optimized. Mysteries can’t—or at least not with significant risk of disaster.

In the world of mysteries—like whether a pandemic lurks around the corner, or what might happen if global temperatures continue to surge—focusing on resilience is a wiser strategy. When you know exactly how something works and those dynamics don’t change over time, then, by all means—optimize to the limit using past data. But when you are unsure of exactly what’s going on in a system, or the underlying dynamics might shift, then optimization to the limit courts disaster. Never Moneyball a mystery.

Beyond resilience, it’s also wise to approach mysteries through experimentation. I previously explained the virtues of experimentation, but in the face of uncertainty, experimenting helps you trial possible solutions. This yields the following lessons, which are rarely followed in modern, data-driven life:

1. Use data analytics only to optimize when solving puzzles—and when you know exactly what you’re trying to solve.

2. If facing a mystery that you don’t, or can’t, fully understand, dial down efforts to optimize, rely less on flawed data models, and focus instead on resilience and experimentation.

Even then, the misapplication of data analytics can backfire if it turns out you’re optimizing for the wrong metric, getting ever-better at solving the wrong puzzle.

V: The McNamara Fallacy, the Surefire Mediocre, and the emptiness of a Lifehack Lifestyle

Robert McNamara was the original King of Big Data. He used metrics for everything, from his work at Ford Motor Company to overseeing the disastrous Vietnam War. But he was seduced by the flawed notion that only that which could be measured is important—and in the process forgot what actually mattered. In one instance at Ford, he insisted that all spare parts be used up before production could continue. Eventually, his subordinates met their data target by dumping all the spares in the nearby river.

In Vietnam, he fetishized the “body count,” a tally—often exaggerated to placate McNamara—of American versus Viet Cong casualties. Not only did McNamara underestimate how much more Americans cared about US deaths than those of the Viet Cong, he also completely ignored difficult-to-measure aspects that decide wars, such as the hearts and minds of the local people.

One general, Edward Lansdale, told McNamara that he should include the feelings of rural Vietnamese people in his analysis of the war. McNamara apparently scribbled this suggestion down in pencil, then erased it, telling Lansdale that it must not be important since it couldn’t be measured.

The Vietnam data boondoggle gave rise to the McNamara Fallacy, the false notion that every problem can be analyzed—then solved—exclusively with quantitative data and better metrics. McNamara thought he should optimize for body count, when in fact, the only thing that mattered was whether the US could win the war. They were not remotely the same.

With much lower stakes, Moneyballing fell into a similar trap of mismatching the optimization metric with the broader goal. Again, from Fluke:

“The analytics were so effective that the game became boring…Baseball became more like two spreadsheets of convergent probabilities battling it out on a diamond. The sport was optimizing for the wrong thing.”

We watch sports for excitement, not statistical optimization. As teams followed the data, fans stopped watching. In response, Major League Baseball was forced to change several rules to “de-moneyball” the game, creating more uncertainty—and excitement. They had fixated on the analytics, but solved the wrong puzzle.

In creative realms, Moneyballing can create an explosion of cultural sludge, as companies chase the “surefire mediocre,” converging toward banal blockbusters, McMansion architecture, and simply replicating that which worked previously. If applied incorrectly, data analytics can lead to convergence toward bleak averages, with dreary consequences for human culture.

The same lesson, alas, applies to us.

We have been sold the lie that the good life is one that is fully optimized, where life hacks obliterate any lingering demons of inefficiency, with successful days measured solely by the numbers of items crossed out on to-do lists. When trying to get to the airport on time, Google Maps is a godsend. But when we choose to take an optimized route to shave off a minute—even it diverts us from a slower but better road toward awe, or wonder, or exploration—we strip out and murder part of our humanity, sacrificing it on the altar of hyper-efficient, data-driven control.

13.8 billion years of the Universe’s tinkering has culminated in us, creatures with the most extraordinary and seemingly unique consciousness. It would be a waste to devote that cognitive magic and the overwhelming improbability of our existence to a Lifehack Lifestyle, with an unquenchable thirst for optimization.

Modern data wizardry allows us unprecedented opportunities to tame truly destructive scourges: crippling poverty, devastating disease, premature death. It will help us solve important puzzles. But we would be wise to avoid taming the wildness that makes us human—all while ensuring that we don’t make the catastrophic, Sisyphean error of imagining that we can and should try to use data analytics to tame the world’s enduring, unknowable mysteries.

---

Thank you for reading The Garden of Forking Paths. If you value my writing and want to show your support—so I can keep at it—please consider upgrading to a paid subscription. My writing is fully reader supported, so you make this possible.

1. For example, there’s no doubt in my mind that we will live longer, healthier lives with the help of these tools. Big Data combined with machine learning will be transformative for earlier diagnoses and better treatment in health care.

2. Mervyn King is the former Governor of the Bank of England. He is also clearly a gentlemanly genius, since he kindly blurbed my book. (Seriously, though, Radical Uncertainty is an excellent book—and an important corrective).

3. We recognize, instantly, that questions such as “what is the meaning of life?” or “Is there a God?” cannot be answered with data analytics. But many more concrete questions are mysteries, even if we too often mistake them for puzzles.

4. Don’t misunderstand me here: I am a huge believer in the power of data to solve puzzles. This is about which realms are beyond the scope of analytics. It’s possible to believe in the power of Big Data while suggesting that it’s being over-used, inappropriately, in some realms where it is useless or dangerous. That’s my argument.