The Prover, The Skeptic, and The Judge
Point an autonomous LLM-powered security agent at a target and tell it to find vulnerabilities, and it will find vulnerabilities. It finds them every time. Point it at a hardened, freshly patched box with nothing actually wrong on it, and you still get a report back, formatted nicely, seemingly suitably alarmed, impact ratings and everything. Unfortunately, current models are biased towards generating plausible analyses rather than confidently concluding that no vulnerabilities were found, particularly when prompts and agent workflows implicitly assume there is something to discover. The result can be nice-looking reports containing convincing but nonexistent vulnerabilities.
So you end up with a pile of findings and no quick way to know which ones are real because the actual bugs and the completely invented ones show up looking identical.
When confidence tells you nothing
An example of a finding an agent produced not long ago:
SQL Injection In /login Endpoint
Impact: Critical
It even provided a PoC python script that, according to the transcript, ran and came back with a Success message.
Maybe it's genuine? Or maybe the endpoint returned a 500 for some unrelated reason and the model read "error in the response" as "injection confirmed". Maybe the script printed Success but nothing in it ever actually checked whether it actually succeeded. Maybe the whole thing's a convincing hallucination, stitched together from the shape of a few thousand security writeups the model saw during training. You can't tell which of these happened by looking at the report, because the report reads exactly the same either way.
It gets worse when you look at where the finding came from. The first pass is usually run on a fast, cheap model, because maybe you want to cover as much surface as possible without spending a fortune, but the fast cheap models are the ones that over-report and over-dramatize, and from what I have seen, have little idea how to correctly rate a finding with respect to the overall context or business purpose of the application. They pattern-match to the shape of a vulnerability and file it. So the stage that produces the most findings is also the stage you should trust the least, which is a bad combination when the report from that stage is the thing that could land on someone's desk.
The obvious response is to write a firmer prompt:
Only report vulnerabilities you've verified.
Don't report false positives
(which never sticks anyway).
You're asking one agent, working in one train of thought, to be the eager hunter who turns up bugs and at the same time, be the hard-nosed critic who throws most of them out. Those two want different things but they're living in the same head, and the hunter tends to win because turning something up feels like getting somewhere, and admitting you were wrong feels like walking backwards.
One solution is to stop asking a single agent to do both.
So we split the work across roles that don't share a train of thought and, more to the point, don't share a motive.
The Prover
The Prover does the thing the agent was always going to do. It pokes at the target, comes back with a bug candidate, and writes it up as a claim with evidence behind it. SQL injection at /login, critical impact, here's the poc.py that shows it. The Prover gets to be eager. That's what it's wired for. Nobody's trying to talk it into caution. The goal is to let it be productive and then send whatever it produces somewhere that isn't.
The Skeptic
That somewhere is the Skeptic. The Prover can run cheap, because turning up candidates is the easy part. The Skeptic runs on a stronger model with a bigger reasoning budget. It takes the Prover's claim and the evidence and goes after them hard in an attempt to knock the thing down.
The Judge
The Judge comes last. It doesn't run anything itself. It reads the Prover's case and the Skeptic's objections side by side, weighs them against a fixed set of questions about evidence, reproducibility, impact, and confidence, and rules. It's the only role whose whole job is to say no and mean it, and because it never had the fun of finding the bug, it has nothing invested in the bug being real.
Written out, the instruction that drives this is blunt about the three rounds and about not letting the first one stand:
MODE: panel. Treat each finding as an adversarial match played in three
rounds, and play all three in full:
(1) PROVER Build the strongest possible live reproduction and state the
case that the finding is REAL, with captured evidence.
(2) SKEPTIC Argue as hard as you can that it's a FALSE POSITIVE. Reproduce
the unauthenticated / baseline behavior, disable redirects,
check that the "impact" content is truly route-specific and
absent from the baseline, and surface every alternative
explanation.
(3) JUDGE Weigh both rounds against the evidence and rule.
Do not let the Prover round go unchallenged.Run it lighter and the same session plays all three parts in sequence. Run it heavier and each round is a separate model with its own context, which is the version I trust, for reasons the last section gets into.
What breaking a claim actually looks like
The Skeptic's work is mostly about the baseline, and the baseline is the part a single eager agent almost always skips.
Take that SQL injection at /login for example. The Skeptic reproduces the exact request the Prover recorded, using the real wire values rather than a tidied-up version, and watches what actually comes back. Then it goes hunting for the boring explanation. It pulls the same page while logged out, to see whether the "leaked" content was ever protected in the first place. It turns off automatic redirect following, because a lot of authentication-bypass findings are really a 302 to a login page that the tooling quietly chased all the way to a 200.
For a claim that rests on something like a vulnerable dependency, it checks that the installed version is genuinely in the affected range instead of trusting a banner. For blind and out-of-band bugs, the kind where nothing visible comes back, it waits for a real callback to a server it controls rather than reading success into a delay or a hunch. The rule underneath all of it is easy to state and hard to live by. Actively try to make the finding fail. Only once you've honestly tried and failed to break it are you allowed to call it real.
Underneath the theatrics, the Skeptic's prompt is an unglamorous checklist it has to run before it's allowed to agree with anything:
Argue the other side. Actively try to make the finding FAIL.
Does the "impact" content also appear in the logged-out baseline?
Was a redirect being followed to a page that only looks protected?
For a version or CVE claim, is the installed version actually in range?
For a blind finding, did a real out-of-band callback fire, or are you
inferring success from a delay?
What is the most boring explanation that fits the same evidence?
Only after you've genuinely tried to refute it may you confirm it.
Evidence must be output you captured this session, never a paraphrase and
never a reconstruction. If you can't capture it, the verdict is
INCONCLUSIVE or NEEDS-MANUAL, never CONFIRMED on inference.Four ways to rule
A finding is confirmed when the reproduction ran live and the captured evidence shows the stated impact. An actual record that should have been out of reach, not just a 200 that could literally mean anything. A finding is refuted when the conditions reproduce and the claim still falls over. The sensitive content turns out to sit in the logged-out baseline too, or the protected page was only reachable because a redirect was being followed, or the vulnerable version isn't in range, or the proof of concept just doesn't fire a second time. Those two are the clean outcomes.
A finding is inconclusive when the target has moved, the endpoint is gone, a WAF started blocking, or the evidence stays genuinely ambiguous after a real effort, and the Judge says exactly that instead of guessing. A finding is needs-manual when confirming it would take something the system can't or shouldn't do on its own like click a link in an email, tap a hardware key, or run the one destructive step that'd actually prove it. The verifier is read-only by design, so a bug that can only be confirmed by breaking something gets handed off with a written description of the safe manual check rather than a made-up result.
Two smaller disciplines fall out of this. Evidence has to be something the Skeptic actually captured this session, the real request and the real response, not a paraphrase and not a plausible reconstruction. If it can't be captured, the finding can't be confirmed, and it drops to inconclusive or needs-manual. And surviving the Skeptic doesn't lock in the original severity. A confirmed bug still gets re-rated, and plenty get marked down. An "admin data exposure" that turns out to require admin access to reach in the first place is real, technically, and also close to worthless, and the rating should say so.
The ruling itself is a structured object, not a paragraph of vibes. Each finding the Skeptic touches gets one written back onto it, which is what lets the report carry a per-finding badge and what forces the reasoning to be spelled out instead of implied:
{
"verdict": "CONFIRMED", // CONFIRMED | REFUTED | INCONCLUSIVE | NEEDS-MANUAL
"confidence": "high", // high | medium | low
"reproduced": true,
"evidence": "",
"reasoning": "why this verdict, including the refutation that was tried",
"adjusted_severity": "medium", // re-rated; may sit below the original
"artifacts": ["poc_verify_3_sqli_login.sh", "poc_verify_3_sqli_login.log"]
} The adjusted_severity sitting a notch under a CONFIRMED verdict is the re-rating in action. The bug is real and reproduced, and it still isn't the critical impact finding the first pass claimed.
When the ruling is done it rides back onto the original finding, so the report that started as a wall of undifferentiated claims now carries a verdict on each one. You can see at a glance which survived and which got struck down.
Why keeping them apart matters
You could write all three of these behaviors into a single model as instructions. Discover first, then criticize, then judge. It slides right back into the same tangle of competing motives that beat the firmer-prompt idea. The steps were never where the value sat. It sat in the Skeptic starting fresh, with a clean context and one plain instruction, which is to tear the claim down. It hasn't spent forty turns talking itself into believing the bug it just found. It reads a claim from someone it has no stake in, and its sole purpose in life is to find where that claim breaks.
The cleanest version of this gives each role its own process and its own context, so the Skeptic literally cannot see the Prover's private reasoning, only the claim and the evidence it chose to stand behind. A lighter version plays all three rounds inside one session, one after another, with the model told in plain terms to switch sides each round and to leave the Prover's case unchallenged at its peril. That still helps, because the adversarial framing does most of the work. The separate-context version is stronger for the obvious reason. A model that never watched you talk yourself into the bug is harder to talk into it.
Operational realities
This isn't free. It costs tokens, and lots of them, because the expensive stages are the ones you run on the stronger model and the reject loop can go around more than once before it settles on a verdict. For a lot of work, that still beats paying a person to waste their time sorting real findings from garbage by hand, but it's a genuine cost and you should weigh it against how much a wrong finding actually costs you (or your business).
The Skeptic is only as good as its prompt. Give it a thin one and it will reproduce the finding, wave a hand at the baseline, and pass everything through, and now you have a chain of agents hopelessly nodding along to a hallucination inside a process that makes the nodding look like rigor. The Skeptic's prompt is the wall this whole thing leans on. Every concrete refutation step, pull the logged-out baseline, kill the redirects, check the version is in range, wait for the real callback, has to be spelled out, because "be skeptical" on its own buys you nothing.
And the Judge is only as calibrated as the model underneath it, so you should still keep an eye on it. A Judge that waves everything through is just a slower path to the same bad output. A Judge that rejects everything teaches the Prover to stop trying.
Build the thing so that being wrong has to survive a run-in with something whose only job is to prove you wrong. Telling a model "don't report false positives" was never going to hold. Giving a second, stronger model a reason to go catch them, a clean slate, and a verdict it has to sign, might have a chance. - insertchaos