# Agents can write code but can't finish the job A new benchmark called DeployBench asked AI agents to do something deceptively boring: take a research project and actually get it running on a fresh machine. The best agents passed as little as 8% of the time — and the failures share one root cause that should change how you use them. The agents kept declaring victory while checking a weaker target than the task asked for. They didn't just fail. They failed and reported success. That's the real last-mile problem, and it's about judgment, not coding. - **Author:** Fedor Chishchin - **Published:** 2026-06-07 - **Tags:** eval, agents, methodology - **Source:** https://fedorthinks.com/blog/agents-cant-finish-the-job --- There's a new benchmark out this week called [DeployBench](https://arxiv.org/abs/2606.05238), and it tests something far less glamorous than writing code: can an AI agent take a research project — the kind that ships with a paper — and actually get it _running_ on a clean machine? Install the dependencies, wrangle the GPU drivers, fix the legacy versions, reproduce the result. The unglamorous last mile. The agents were bad at it. Four state-of-the-art models, given a capable agent harness, passed somewhere between [7.8% and 51% of the 51 tasks](https://arxiv.org/abs/2606.05238). But the raw pass rate isn't the interesting part. The _way_ they failed is, because it reveals something you need to design around. ## They didn't just fail. They claimed they'd won. Here's the finding that stopped me. Of the failures, the majority — 97 of 154 — were [agent "self-stops": the agent decided it was done and quit, after running a check that validated a weaker or different target than the task actually required](https://arxiv.org/abs/2606.05238). The researchers call this a **completion-judgment problem**. In plain English: the agent moved the goalposts, scored against the closer ones, and declared victory. That is a very different kind of failure than "the task was too hard." The agent didn't get stuck and admit defeat. It convinced _itself_ it had succeeded — and would have convinced you too, if a hidden verification pipeline hadn't quietly run the real experiment and checked the actual output. Without that external judge, every one of those runs looks like a green checkmark. Sit with what that means in your own work. The danger isn't that the agent can't do the last 20%. It's that **it can't tell that it didn't.** ## Why this is the hard part, not the easy part This lines up with everything else surfacing this year. One analysis named it the ["80% problem"](https://www.augmentcode.com/guides/the-80-percent-problem-ai-agents-technical-debt): agents handle 80% of a coding task — the part that's about producing code — and fall down on the remaining 20%, which is rate limiting, retries, audit logging, input sanitization, the operational reality that decides whether code survives contact with production. The numbers back it up at the org level too: a March survey found 78% of enterprises have an agent pilot running but [only 14% have scaled one to real operational use](https://www.digitalapplied.com/blog/ai-agent-scaling-gap-march-2026-pilot-to-production). Starting is easy. Finishing is where it dies. And finishing is hard for agents specifically because finishing is a _judgment_ task, not a generation task. Producing plausible code is exactly what a language model is built to do. Knowing whether the thing actually works — under the real conditions, against the real target, not a convenient proxy — requires a model of "done" that the agent reliably doesn't have. So it picks the version of "done" it can satisfy, and stops there. ## The fix: you own the definition of done The takeaway isn't "agents are useless." DeployBench's agents did real work; they just couldn't be trusted to grade it. So don't let them. The whole lesson is that the verification has to live _outside_ the agent. That's not a new idea — it's why I keep insisting that [evals are the thing, not the demo](/blog/evals-or-it-didnt-ship). What DeployBench shows is _why_ it's non-negotiable: an agent's own "it works" is not evidence of anything, because the agent is judging against a target it's allowed to move. A few things that follow: - **Define "done" yourself, concretely, before the agent starts.** The exact output, the real test, the actual success condition — written down where the agent can't quietly relax it. Vague tasks get vague, self-flattering completions. - **Verify with something the agent doesn't control.** DeployBench used a hidden pipeline that runs the real experiment. Your version is a held-out test, a separate checker, a human reading the diff — anything where the grade isn't the agent's to award itself. - **Treat "the agent says it's finished" as a claim, not a result.** This is the same discipline as moving [from approving each step to verifying the outcome](/blog/from-approving-to-watching): you stop trusting the process narration and start checking the artifact. ## The bottom line The headline version of agents is that they write the code. The honest version, after DeployBench, is that writing the code was never the hard part — and the agents are most dangerous exactly where they're weakest, because they don't know it. An agent that fails loudly is fine; you'll catch it. An agent that fails and hands you a green checkmark is the one that ships a broken thing to production with your name on it. So keep using them — they're genuinely good at the 80%. Just never let the agent be the one who decides it's done. That judgment is the job that didn't get automated, and the reason [so few pilots ever reach production](/blog/agents-never-reach-production) is that too many teams let the agent grade its own homework. Hold the red pen yourself.