Superforecasting
The Art and Science of Prediction
Book Summaries
Hosts: Ethan
Timeline
Summary Preview
In 1956, Archie Cochrane—the pioneer of evidence-based medicine who revolutionized how doctors test treatments—was told by a renowned surgeon that he had terminal cancer. Cochrane accepted the diagnosis without question. He prepared to die. But when a pathologist finally examined the biopsy, the result was clear: no cancer at all.
Here's the irony. Cochrane had spent his career proving that doctors were dangerously overconfident. He showed that physicians relied on personal experience rather than randomized trials, that they treated patients based on what "felt right" rather than what the data said. He called it the "God complex" in medicine. Yet when a specialist pointed a finger at him, he folded instantly. He didn't ask for evidence. He didn't seek a second opinion. He just believed.
What happened to Cochrane happens to all of us. It happens to experts. It happens to analysts. It happens to CEOs. And it happens to forecasters making predictions that shape the decisions of governments and corporations.
The problem is not that we don't know enough. The problem is how we think.
**The dart-throwing chimpanzee**
For twenty years, Philip Tetlock studied the predictions of 284 expert political scientists, economists, and intelligence analysts. He asked them to forecast outcomes like whether a country would go to war, whether an economy would collapse, whether a leader would fall. The results were devastating. The average expert performed no better than a dart-throwing chimpanzee—meaning their accuracy was essentially random.
But here's what made it worse. When these experts were wrong, they rarely admitted it. They explained away their failures. They shifted goalposts. They claimed they were "almost right." And the public kept listening, because the experts spoke with confidence, told compelling stories, and never seemed to doubt themselves.
Tetlock wasn't a nihilist. He didn't conclude that forecasting was impossible. Instead, he asked a different question: If most experts fail, what separates the few who succeed?
**Two systems at war**
Cochrane's mistake reveals the root cause of poor forecasting. Psychologist Daniel Kahneman describes two systems in our brains. System 1 is automatic, fast, intuitive. It's the part that recognizes a friend's face, that catches a ball, that jumps to conclusions. System 2 is slow, deliberate, analytical. It's what you use to solve a math problem or plan a vacation.
System 1 is essential for survival. A hunter-gatherer who stops to calculate probabilities when a shadow moves in the grass won't live long enough to pass on their genes. But System 1 is terrible for forecasting. It grabs the first plausible explanation and runs with it. It doesn't check for alternatives. It doesn't seek contradictory evidence.
When the surgeon told Cochrane he had cancer, Cochrane's System 1 kicked in. The surgeon was an authority. The diagnosis fit the symptoms. The story was coherent. Cochrane didn't engage System 2 to ask: "What's the evidence? What's the base rate? Could I be wrong?"
This is not a failure of intelligence. Cochrane was brilliant. It's a failure of process.
**Confirmation bias and the bait-and-switch**
Two specific cognitive errors trap most forecasters.
The first is confirmation bias. Once we form a belief, we naturally seek evidence that confirms it and ignore evidence that contradicts it. Think of the 2011 Oslo bombing. When news broke, experts immediately blamed Islamic terrorist groups. The narrative fit. Similar attacks had happened in London, Madrid, Bali. But the actual perpetrator was Anders Breivik, an anti-Islamist Norwegian. The experts had jumped to a conclusion and stopped looking.
The second error is what psychologists call the bait-and-switch. When faced with a hard question, our brains substitute an easier one. Cochrane didn't ask "Do I actually have cancer?" He asked "Is this surgeon the kind of person who would know whether I have cancer?" The second question is easier to answer. But it's not the right question.
The same error happens in intelligence analysis. During the lead-up to the Iraq War, analysts didn't ask "Does Saddam Hussein actually have weapons of mass destruction?" They asked "Is Saddam Hussein acting like someone who has weapons of mass destruction?" The switch from the hard question to the easy one led to a war that cost thousands of lives.
**The Good Judgment Project**
If the dart-throwing chimpanzee represents the baseline, then the Good Judgment Project represents the proof that something better is possible. In 2011, Tetlock launched a forecasting tournament funded by the Intelligence Advanced Research Projects Activity. He recruited 2,800 volunteers—ordinary people, not experts. He asked them to make numerical predictions about world events: Will Russia annex Ukrainian territory in the next three months? Will Greece default on its debt by year-end?
The results were remarkable. The volunteers outperformed professional intelligence analysts who had access to classified information. And within this group, a small fraction—about 2%—consistently outperformed everyone. Tetlock called them superforecasters.
These weren't geniuses. They weren't subject-matter experts. They were people who had learned to think differently. And Tetlock discovered that their approach could be taught.
**The core insight**
Most expert predictions are unreliable because experts are human, and humans are prone to cognitive biases. But accurate forecasting is not about being smarter or knowing more. It's about following a process that counteracts those biases.
The first step is recognizing the problem. Cochrane's mistake shows that even the most evidence-minded among us can fall for overconfidence. The second step is understanding the mechanisms: System 1 thinking, confirmation bias, and the bait-and-switch error. These are not flaws in other people. They are flaws in all of us.
The third step is the most important. Once you see the problem, you can build a system to fix it. That's what the Good Judgment Project did. And that's what this book will show you how to do.
But before we get to the solutions, we need to ask a harder question. How do you measure whether a forecast is actually accurate? How do you separate skill from luck? How do you know if you're getting better or just fooling yourself?
That's the question we'll tackle next. Because without measurement, improvement is impossible. And without improvement, we're all just throwing darts in the dark.
About the Book
Superforecasting reveals how a small group of ordinary volunteers consistently outperformed intelligence analysts and defied statistical luck. Drawing on decades of research from the Good Judgment Project, Philip Tetlock and Dan Gardner show that accurate forecasting isn't about genius or secret data—it's about thinking like a fox: embracing doubt, updating beliefs in tiny steps, and measuring every prediction. This book offers a practical, evidence-based system for making better decisions under uncertainty.
Key Takeaways
Replace vague language with precise numerical probabilities to improve forecast accuracy.
Stop using words like 'likely' or 'probably' and instead assign specific percentages (e.g., 70% chance). This forces clearer thinking, enables precise measurement of accuracy using Brier scores, and holds you accountable for your predictions.
Think like a fox, not a hedgehog: use multiple perspectives instead of one big idea.
Hedgehogs rely on a single grand theory and consistently underperform. Foxes draw on diverse frameworks, seek contradictory evidence, and start with base rates (the outside view) before adjusting for specifics, leading to significantly more accurate forecasts.
Update your beliefs frequently but in tiny, calibrated increments.
Superforecasters make small adjustments (around 3-5%) as new evidence arrives, avoiding both overreaction to noise and underreaction to genuine signals. When you realize a forecast is fundamentally wrong, recalculate from scratch rather than making small corrections to a broken foundation.
Adopt a 'perpetual beta' growth mindset: treat every forecast as a learning experiment.
Commit to continuous improvement by conducting postmortems on every prediction—analyzing both successes and failures. Write down your reasoning before outcomes are known to avoid hindsight bias, and actively search for reasons you might be wrong.
Build superteams by enforcing independent judgment before group discussion.
Teams outperform individuals by 23% when members make their own forecasts privately first, then share reasoning, and then update independently again. Weight contributions by demonstrated track record (not seniority or confidence) and create psychologically safe environments for constructive disagreement.
Balance decisive leadership with intellectual humility by calibrating your confidence.
Effective leaders distinguish between what they know and don't know, stating confidence levels (e.g., 'I'm 70% sure') and listing factors that would change their mind. Decentralize decision-making so team members can act independently within clear objectives, just as Prussian Field Marshal von Moltke did.
Break complex problems into tractable sub-problems using Fermi decomposition.
When faced with a hard question, don't guess—break it into smaller, answerable pieces. For example, to estimate piano tuners in Chicago, calculate pianos, tuning frequency, time per tuning, and work hours per tuner. This transforms impossible questions into solvable ones.
Triage your forecasting efforts: focus on questions where careful thinking makes a difference.
Don't waste time on questions that are too easy or impossible to predict. Concentrate on the middle ground where systematic analysis can improve accuracy. Pair 'superquestioners' who identify critical questions with 'superforecasters' who provide precise, calibrated answers.
Who Should Listen?
Executives and business leaders who need to make high-stakes strategic decisions under uncertainty and want a systematic method to reduce costly forecasting errors.
Intelligence analysts, policy advisors, and risk managers whose professional credibility depends on accurate geopolitical or economic predictions.
Investors and traders who rely on probabilistic thinking to allocate capital and want to distinguish genuine forecasting skill from lucky streaks.
Curious general readers who are tired of confident but wrong expert predictions on cable news and want to learn how to think more clearly about the future.
