Michael Crichton (famous novelist and never-practicing MD), describing the cognitive bias of “Gell-Mann Amnesia” in a 2002 speech (as included in How Not to Invest by Barry Ritholtz):
You open the newspaper to an article on some subject you know well. You read the article and see the journalist has absolutely no understanding of either the facts or the issues. Often, the article is so wrong it actually presents the story backward—reversing cause and effect. I call these the wet streets cause rain stories. Papers are full of them.
In any case, you read with exasperation or amusement the multiple errors in a story—and then turn the page to national or international affairs, and read with renewed interest as if the rest of the newspaper was somehow more accurate about far-off Palestine than it was about the story you just read. You turn the page, and forget what you know.
One of the points Crichton is making, particularly with regard to the media, is that we are more forgiving of certain institutions than we are with people in real life. When someone opines confidently but incorrectly about your area of expertise, you know they’re a blowhard. But we are generally forgiving of institutions (until we become so blinded by bullshit that we discount everything, adrift on a sea of conspiracy theories).
We do discount what are considered credible and trustworthy sources when we know they’re wrong but otherwise assume they’re generally right. It’s almost an institutional halo effect; we give more credit than due unless we are forced by our own knowledge to confront it.
In reality, the lesson is: truth is hard. People are wrong all the time, and institutions are full of people. That doesn’t mean they don’t matter—they do!—or that everyone is consistently wrong or that you should never trust anyone, but it does mean that our beloved institutions require maintenance and care.
Ritholtz:
There are tremendous advantages in recognizing what you do not know. Acknowledging shortcomings in your informational intelligence is a form of situational awareness that prevents you from being blindsided. There are other benefits as well. It shifts your focus to process over outcome; you can better understand what results come from skill versus dumb luck. It prevents you from being fooled by randomness.
We should be a little more thoughtful and a little less credulous. A little less precious with our own knowledge, acknowledging the shortcomings in what we know, where we seek information, and how we incorporate it into our latticework of mental models.
In other words: Less Certainty, More Inquiry.