Superforecasting

Philip E. Tetlock & Dan Gardner

19 annotations • Jan 2023 – May 2023 • data

Chapter 1

Machines may get better at "mimicking human meaning," and thereby better at predicting human behavior, but "there's a difference between mimicking and reflecting meaning and originating meaning," Ferrucci said. That's a space human judgment will always occupy.

Chapter 2

It's human nature. We have all been too quick to make up our minds and too slow to change them
Galen's writings were the indisputable source of medical authority for more than a thousand years. "It is I, and I alone, who has revealed the true path of medicine," Galen wrote with his usual modesty. And yet Galen never conducted anything resembling a modern experiment. Why should he? Experiments are what people do when they aren't sure what the truth is. And Galen was untroubled by doubt. Each outcome confirmed he was right, no matter how equivocal the evidence might look to someone less wise than the master. "All who drink of this treatment recover in a short time, except those whom it does not help, who all die," he wrote. "It is obvious, therefore, that it fails only in incurable cases." Galen is an extreme example but he is the sort of figure who pops up repeatedly in the history of medicine. They are men (always men) of strong conviction and a profound trust in their own judgment. They embrace treatments, develop bold theories for why they work, denounce rivals as quacks and charlatans, and spread their insights with evangelical passion
It was cargo cult science, a term of mockery coined much later by the physicist Richard Feynman to describe what happened after American airbases from World War II were removed from remote South Pacific islands, ending the islanders' only contact with the outside world.
The rate of the development of science is not the rate at which you make observations alone but, much more important, the rate at which you create new things to test
What people didn't grasp is that the only alternative to a controlled experiment that delivers real insight is an uncontrolled experiment that produces merely the illusion of insight.
In celebrated research, Michael Gazzaniga designed a bizarre situation in which sane people did indeed have no idea why they were doing what they were doing. His test subjects were "split-brain" patients, meaning that the left and right hemispheres of their brains could not communicate with each other because the connection between them, the corpus callosum, had been surgically severed (traditionally as a treatment for severe epilepsy). These people are remarkably normal, but their condition allows researchers to communicate directly with only one hemisphere of their brain—by showing an image to only the left or right field of vision—without sharing the communication with the other hemisphere. It's like talking to two different people. In this case, the left field of vision (which reports to the right hemisphere) was shown a picture of a snowstorm and the person was asked to point to the picture that related to it. So he quite reasonably pointed at the shovel. The right field of vision (which reports to the left hemisphere) was shown an image of a chicken claw—and the person was then asked why his hand was pointed at a shovel. The left hemisphere had no idea why. But the person didn't say "I don't know." Instead, he made up a story: "Oh, that's simple," one patient said. "The chicken claw goes with the chicken, and you need a shovel to clean out the chicken shed."
Scientists must be able to answer the question "What would convince me I am wrong?" If they can't, it's a sign they have grown too attached to their beliefs.
The key is doubt. Scientists can feel just as strongly as anyone else that they know The Truth. But they know they must set that feeling aside and replace it with finely measured degrees of doubt—doubt that can be reduced (although never to zero) by better evidence from better studies.
Daniel Kahneman noted, "but declarations of high confidence mainly tell you that an individual has constructed a coherent story in his mind, not necessarily that the story is true."
Formally, it's called attribute substitution, but I call it bait and switch: when faced with a hard question, we often surreptitiously replace it with an easy one. "Should I worry about the shadow in the long grass?" is a hard question. Without more data, it may be unanswerable. So we substitute an easier question: "Can I easily recall a lion attacking someone from the long grass?" That question becomes a proxy for the original question and if the answer is yes to the second question, the answer to the first also becomes yes.

Chapter 3

The truth is, the truth is elusive
Foxes don't fare so well in the media. They're less confident, less likely to say something is "certain" or "impossible," and are likelier to settle on shades of "maybe." And their stories are complex, full of "howevers" and "on the other hands," because they look at problems one way, then another, and another. This aggregation of many perspectives is bad TV. But it's good forecasting. Indeed, it's essential.
But remember the old reflexivity-paradox joke. There are two types of people in the world: those who think there are two types and those who don't. I'm of the second type.

Chapter 6

As the legendary investor Charlie Munger sagely observed, "If you don't get this elementary, but mildly unnatural, mathematics of elementary probability into your repertoire, then you go through a long life like a one-legged man in an ass-kicking contest."
Vonnegut drums this theme relentlessly. "Why me?" moans Billy Pilgrim when he is abducted by aliens. "That is a very Earthling question to ask, Mr. Pilgrim," the aliens respond. "Why you? Why us for that matter? Why anything?"25 Only the naive ask "Why?" Those who see reality more clearly don't bother. It's a trenchant insight. When something unlikely and important happens it's deeply human to ask "Why?"
Meaning is a basic human need. As much research shows, the ability to find it is a marker of a healthy, resilient mind. Among survivors of the 9/11 attacks, for example, those who saw meaning in the atrocity were less likely to suffer post-traumatic stress responses. But as psychologically beneficial as this thinking may be, it sits uneasily with a scientific worldview. Science doesn't tackle "why" questions about the purpose of life. It sticks to "how" questions that focus on causation and probabilities.
probabilistic thinker will be less distracted by "why" questions and focus on "how." This is no semantic quibble. "Why?" directs us to metaphysics; "How?" sticks with physics. The probabilistic thinker would say, "Yes, it was extremely improbable that I would meet my partner that night, but I had to be somewhere and she had to be somewhere and happily for us our somewheres coincided."
Robert Shiller would not have been born. But rather than see fate in his improbable existence, Shiller repeats the story as an illustration of how radically indeterminate the future is. "You tend to believe that history played out in a logical sort of sense, that people ought to have foreseen, but it's not like that," he told me. "It's an illusion of hindsight."