Publication bias: the studies you never get to read

Imagine twenty teams each test the same useless pill. By chance alone, one or two will get a fluky positive result. Those are the ones that get written up, submitted, and published. The eighteen that found nothing quietly go into a drawer. A reader who later searches the literature finds only the exciting studies and concludes the pill works. That is publication bias, and it is one of the most important things to understand about science news.

It happens because journals, authors, and even readers prefer a positive, surprising result to a boring null one. A study showing a new treatment works is exciting; a study showing it does nothing feels like a non-event, so it is less likely to be submitted and less likely to be accepted. The nickname for the pile of unpublished null results is the file-drawer problem.

The consequence is that the published record can be a skewed sample of all the research that was actually done. Effects look larger and more certain than they really are, because the failures are invisible. This is a particular danger for meta-analyses, which pool the published studies: if the negative ones were never published, even a careful pooling of everything available can land on the wrong answer.

You cannot see the missing studies directly, which is what makes this so slippery. But you can stay alert to it. Be extra cautious about a single dramatic result, especially in a small study. Give more weight to large trials that were registered in advance, because a pre-registered trial has to report its result whether it is exciting or not. And when a body of evidence rests on lots of small positive studies, ask where the null ones went.