Published on 2026-07-25
What actually makes a scientific study solid?
Peer review, sample size, a control group, declared conflicts of interest, replication: five checks that tell a solid study from a shaky one.
A headline lands in your feed: 'New study proves coffee adds years to your life.' Your thumb hovers. Is this true, or is it one of those claims that dissolves the moment anyone opens the actual paper? Here's the useful bit: you don't need a science degree to tell the difference. You need to know where to look, because the words 'a study found' cover everything from a rock solid trial to a survey of forty students filling in a questionnaire for course credit.
In 2014, two researchers at Cardiff University, Petroc Sumner and Christopher Chambers, decided to find out where health claims actually go wrong. They compared 462 press releases issued by 20 leading UK universities with the peer reviewed papers behind them, then checked what the news stories built on those releases said. Their study, published in The BMJ, found that around 40 percent of the press releases exaggerated the advice given, and about a third stretched a correlation into a cause. The exaggeration hadn't started with lazy journalists rewriting a careful paper into a wild headline. It was already sitting in the release, and once it was there, most news coverage repeated it faithfully.
The paper is not the press release
That gap between the study and the story told about it is exactly why a claim needs checking at its source, not at whatever hand it passed through last. A press release is written to get picked up. A study is written to survive scrutiny from people whose job is to find its holes. Those are different goals, and only one of them owes you the whole picture.
So what tells you a study can bear some weight? Five things, and none of them require reading a statistics textbook.
- Peer review: has it been checked by independent specialists before publication, or is it a lab's own announcement with nobody else's eyes on it yet?
- Sample size: was it tested on 20 people or 20,000? Small numbers can hint at something worth exploring, they rarely settle anything on their own.
- A control group: was there a comparison group that didn't get the treatment, so the effect can be told apart from coincidence or time simply passing?
- Declared conflicts of interest: who paid for this, and does the funder have a stake in the result coming out a certain way?
- Replication: has anyone else, in another lab, with other participants, found the same thing?
None of these checks demand that you distrust science. They ask you to treat one paper the way a good editor treats one source: useful, worth citing, but not the end of the conversation.
A brick, not the whole wall
Amanda Kay Montoya, a psychologist at UCLA, wrote about this for The Conversation in 2025, describing what happens when nobody rechecks a result. Picture someone flipping a coin ten times, getting seven heads, and announcing that coins land on heads 70 percent of the time. Nothing was faked. The coin is fine. The sample was just too small for chance to average itself out. Replication, retesting the same question with a fresh set of people, is the only way anyone finds out whether the first result was the coin or the flipper's luck.
This is where 'a study found' stops meaning much on its own. One study is a brick. A field of knowledge is the wall built from many bricks, tested by many hands, some of which will crack and get replaced. A single paper that hasn't been through peer review, run on a handful of people, funded by whoever stood to gain from its conclusion, and never checked again by anyone else, is not a wall. It might not even be a brick yet.
None of this means waiting for absolute certainty before believing anything, because certainty rarely arrives in science and waiting for it would leave you paralysed. It means noticing the difference between 'researchers are exploring whether X might help' and 'X is proven to work', and giving your trust in proportion to how many of those five checks a claim can actually pass.
A study earns your trust one check at a time. It doesn't come with it built in.
Next time a headline announces what a study 'proves', try the short version of the exercise: who ran it, how many people, compared against what, paid for by whom, and has anyone else found the same thing since. Most claims survive two or three of those questions. The interesting ones are the claims that don't, and you'll only notice them once you've started asking.
Sources
- Exaggeration in health related science news and press releases (PMC, National Library of Medicine) (pmc.ncbi.nlm.nih.gov)
- Research replication can determine how well science is working (The Conversation) (theconversation.com)