Published on 2026-07-25
Can you trust an 'AI or not' image detector's verdict?
Five detector tools tested on the same 45 photos disagreed on 35 of them. A percentage from one tool is a guess, not a verdict.
You upload a photo to one of those 'is this AI?' sites and it hands you back a number: 87 percent likely AI generated. It feels like an answer. It has a decimal point and everything. What it actually is, most of the time, is one model's best guess, trained on a specific batch of generators, tested against images that may look nothing like the one you just fed it.
NewsGuard, the organisation that tracks online misinformation, ran exactly this experiment in May 2026. It fed the same 45 images (15 authentic photos, 15 lightly edited ones, 15 heavily altered ones) into five widely used detection tools: Hive, AI or Not, ZeroGPT, Sightengine and ScamAI. On 35 of those 45 images, at least one tool reached a different verdict from the rest. Not a close call on a handful of tricky edge cases. Disagreement on the large majority of the set.
Wrong in both directions
The errors didn't all point the same way, which is the part worth sitting with. Some tools were too trigger-happy: ScamAI flagged 40 percent of genuinely authentic photos as AI generated, and ZeroGPT flagged 20 percent. Other tools missed real fakes instead: Sightengine caught only 5 of the 15 heavily altered images, letting two out of every three slip through as if nothing had been touched. A tool that cries wolf on real photos and a tool that waves through fakes are both wrong, just in opposite directions, and you have no way of knowing in advance which failure mode you've landed on.
This isn't only a social media problem. A study published in the journal PeerJ in February 2025 tested three detectors, Is It AI?, Hive Moderation and Illuminarty, on a set of 48 AI-generated scientific images and 48 authentic ones drawn from published research. Accuracy ranged from 53 to 75 percent depending on the tool: better than a coin flip, but not by nearly enough to build a verdict on. One detector correctly caught 96 percent of the fakes but wrongly flagged nearly half of the real images as fake too. Push a tool to catch more fakes and it usually starts accusing more innocent photos in the process.
Why a single score can't hold that much weight
Detectors work by spotting statistical fingerprints: patterns in pixels, compression artefacts, textures that tend to show up when a particular generator built the image. That works reasonably well against the generators a tool was trained on. It works far less well against a newer model it has never seen, against an image that's been resized, screenshotted or run through a filter, or against a photo that just happens to have unusual lighting or noise for innocent reasons. None of that shows up in the neat percentage you get back. The number reads like a measurement. It's closer to an opinion, delivered with more confidence than the tool has actually earned.
It also matters who trained the fingerprint in the first place. A detector built and tested mostly on one popular generator will naturally be sharper at catching that generator's output, and comparatively blind to a smaller or newer tool it barely saw during training. New image generators show up every few months. The detector you're using today was very likely finished before the generator that made the photo in front of you even existed, which is a strange position for something claiming a precise percentage.
A confidence score is the tool's opinion of itself. It isn't a fact about your image.
There's also a quieter problem sitting underneath the accuracy numbers: what happens when a detector is wrong depends on which way it's wrong. Wave a fake through as real, and a fabricated photo spreads a little further with an undeserved stamp of approval. Flag a real photo as fake, and you've just told someone their genuine picture, evidence, or memory can't be trusted. Neither error is harmless, and no detector tells you, in the moment, which kind of mistake it's more likely making on your specific image.
What actually holds up
None of this means detectors are worthless. A tool flagging an image is a reason to look closer, the same way one witness is a reason to keep asking questions. It just isn't the closing argument. The sturdier approach stacks several checks instead of leaning on one: what a detector says, whether the image has a traceable origin somewhere else online, whether independent sources are showing the same scene, and whether the visual details (hands, text, shadows) hold up under a plain look. Each check alone is fallible. Together, they're much harder to fool than any single percentage.
So treat the number for what it is: one opinion, from one tool, trained on yesterday's generators, about an image that might have been made by tomorrow's. Useful information. Not a verdict you can stop at.