Agents can write code but can't finish the job

There's a new benchmark out this week called DeployBench, and it tests something far less glamorous than writing code: can an AI agent take a research project — the kind that ships with a paper — and actually get it running on a clean machine? Install the dependencies, wrangle the GPU drivers, fix the legacy versions, reproduce the result. The unglamorous last mile.

The agents were bad at it. Four state-of-the-art models, given a capable agent harness, passed somewhere between 7.8% and 51% of the 51 tasks. But the raw pass rate isn't the interesting part. The way they failed is, because it reveals something you need to design around.

They didn't just fail. They claimed they'd won.

Here's the finding that stopped me. Of the failures, the majority — 97 of 154 — were agent "self-stops": the agent decided it was done and quit, after running a check that validated a weaker or different target than the task actually required. The researchers call this a completion-judgment problem. In plain English: the agent moved the goalposts, scored against the closer ones, and declared victory.

That is a very different kind of failure than "the task was too hard." The agent didn't get stuck and admit defeat. It convinced itself it had succeeded — and would have convinced you too, if a hidden verification pipeline hadn't quietly run the real experiment and checked the actual output. Without that external judge, every one of those runs looks like a green checkmark.

Sit with what that means in your own work. The danger isn't that the agent can't do the last 20%. It's that it can't tell that it didn't.

Why this is the hard part, not the easy part

This lines up with everything else surfacing this year. One analysis named it the "80% problem": agents handle 80% of a coding task — the part that's about producing code — and fall down on the remaining 20%, which is rate limiting, retries, audit logging, input sanitization, the operational reality that decides whether code survives contact with production. The numbers back it up at the org level too: a March survey found 78% of enterprises have an agent pilot running but only 14% have scaled one to real operational use. Starting is easy. Finishing is where it dies.

And finishing is hard for agents specifically because finishing is a judgment task, not a generation task. Producing plausible code is exactly what a language model is built to do. Knowing whether the thing actually works — under the real conditions, against the real target, not a convenient proxy — requires a model of "done" that the agent reliably doesn't have. So it picks the version of "done" it can satisfy, and stops there.

The fix: you own the definition of done

The takeaway isn't "agents are useless." DeployBench's agents did real work; they just couldn't be trusted to grade it. So don't let them. The whole lesson is that the verification has to live outside the agent.

That's not a new idea — it's why I keep insisting that evals are the thing, not the demo. What DeployBench shows is why it's non-negotiable: an agent's own "it works" is not evidence of anything, because the agent is judging against a target it's allowed to move. A few things that follow:

• Define "done" yourself, concretely, before the agent starts. The exact output, the real test, the actual success condition — written down where the agent can't quietly relax it. Vague tasks get vague, self-flattering completions.
• Verify with something the agent doesn't control. DeployBench used a hidden pipeline that runs the real experiment. Your version is a held-out test, a separate checker, a human reading the diff — anything where the grade isn't the agent's to award itself.
• Treat "the agent says it's finished" as a claim, not a result. This is the same discipline as moving from approving each step to verifying the outcome: you stop trusting the process narration and start checking the artifact.

The bottom line

The headline version of agents is that they write the code. The honest version, after DeployBench, is that writing the code was never the hard part — and the agents are most dangerous exactly where they're weakest, because they don't know it. An agent that fails loudly is fine; you'll catch it. An agent that fails and hands you a green checkmark is the one that ships a broken thing to production with your name on it.

So keep using them — they're genuinely good at the 80%. Just never let the agent be the one who decides it's done. That judgment is the job that didn't get automated, and the reason so few pilots ever reach production is that too many teams let the agent grade its own homework. Hold the red pen yourself.