Two great longish weekend reads:
Vox’s The 7 biggest problems facing science, according to 270 scientists:
Scientists need more carrots to entice them to pursue replication in the first place. As it stands, researchers are encouraged to publish new and positive results and to allow negative results to linger in their laptops or file drawers.
This has plagued science with a problem called “publication bias” — not all studies that are conducted actually get published in journals, and the ones that do tend to have positive and dramatic conclusions.
And David McRaney’s fun discussion of survivorship bias.
The military looked at the bombers that had returned from enemy territory. They recorded where those planes had taken the most damage. Over and over again, they saw that the bullet holes tended to accumulate along the wings, around the tail gunner, and down the center of the body. Wings. Body. Tail gunner. Considering this information, where would you put the extra armor? Naturally, the commanders wanted to put the thicker protection where they could clearly see the most damage, where the holes clustered. But Wald said no, that would be precisely the wrong decision. Putting the armor there wouldn’t improve their chances at all.
Embedded in both:
You can’t just look at success (or successful people) to achieve meaningful success or become successful (corollary: data is not the plural of anecdote). Arguably more important is looking at the failures to see why they fail. And if both groups do things the same, then it’s not the method.1
Also, this applies to every self-help/self-improvement personal story you hear (entertaining, potentially inspiring, and scientifically without merit).