# For long-running agents, cost-per-task is the only benchmark NVIDIA's new Nemotron 3 Ultra isn't pitched on being the smartest model. It's pitched on being cheap to run for hours — built for agents that plan, call tools, and reason across hundreds of turns. That framing is the real story. When an agent runs long, the number that matters stops being the benchmark score or the per-token price and becomes dollars-per-finished-task. Two models at the same token price can differ 2x on a real job. Here's why the leaderboard is the wrong thing to shop on once your agent runs for more than a moment. - **Author:** Fedor Chishchin - **Published:** 2026-06-07 - **Tags:** ai-native, business, eval - **Source:** https://fedorthinks.com/blog/cost-per-task-is-the-benchmark --- NVIDIA shipped a new open model on June 4, and what's interesting isn't a leaderboard score — it's the pitch. Nemotron 3 Ultra is sold on being [faster and cheaper to run for long-running agents](https://developer.nvidia.com/blog/nvidia-nemotron-3-ultra-powers-faster-more-efficient-reasoning-for-long-running-agents/): agents that plan, call tools, and reason across many turns. NVIDIA claims about 5× higher throughput than comparable open models and up to 30% lower cost on agentic tasks, and even ships a "medium-effort" reasoning mode that uses [roughly 2.5× fewer tokens](https://www.marktechpost.com/2026/06/04/nvidia-ai-releases-nemotron-3-ultra-an-open-550b-mixture-of-experts-hybrid-mamba-transformer-for-long-running-agents/) than full reasoning. A model whose headline feature is "cheap to run for a long time" tells you where the market actually is. The interesting competition for agents isn't who tops the intelligence benchmark anymore. It's who finishes the job for the fewest dollars — and that's a completely different number than the one on the leaderboard. ## Why "long-running" changes the whole equation A one-shot model call is cheap and the per-token price barely matters. But an agent that runs for hours is a different beast: it plans, reads, calls a tool, reads the result, reasons, calls another, over and over, sometimes for hundreds of steps. Every one of those steps spends tokens, and they _accumulate_. The cost of a long agent run isn't a rounding error — it's the dominant cost, and it grows with every turn. That changes what you should optimize for. For a chatbot, "which model is smartest per answer" is a fine question. For a long-running agent, the question becomes "which model gets to a correct finish for the lowest total spend" — and those two questions have different winners. A model that's slightly less impressive on a benchmark but uses half the tokens to complete the actual task is the better choice, and the leaderboard will never tell you that. ## The per-token price is a trap too Here's the part that catches people. You'd think the cheapest model is the one with the lowest price per token. It isn't, necessarily. What you pay is price-per-token times tokens-used, and models vary wildly on the second factor. The data is striking: in one analysis, two models at broadly similar token prices finished the same benchmark for [about \$817 versus \$1,888 — a \$1,000+ gap](https://zylos.ai/research/2026-04-13-inference-economics-ai-agent-compute-markets) — because one was far more token-efficient at actually getting the work done. Same sticker price, double the bill. That's why the serious framing in agent economics has shifted to [dollars per successful workflow step, not dollars per raw token](https://zylos.ai/research/2026-04-13-inference-economics-ai-agent-compute-markets). A chatty model that needs three times the steps to finish is expensive even at a bargain per-token rate. Nemotron's whole design — fewer active parameters, a lighter architecture for long sequences, an effort dial — is a bet on winning that real number, not the sticker one. ## What to actually measure This is the practical heart of it, and it connects to things I've argued before. The benchmark score was [never the job](/blog/benchmark-is-not-the-job), and the right way to spend on models is to [match the model to the work](/blog/cheap-model-most-of-the-work). Long-running agents make both of those concrete and urgent. So: - **Measure dollars-per-finished-task, not per token and not the leaderboard.** Run your actual workflow end to end on each candidate model and compare the total cost to a _correct_ completion. That single number quietly decides your margin. - **Count tokens-to-completion, not just price.** A cheaper-per-token model that rambles can cost more than a pricier one that's terse and decisive. Efficiency of getting there is the hidden variable. - **Use the efficiency levers.** Effort modes, prompt caching, reusing stable context across turns — these can cut realized cost dramatically on a repeated- call agent. The expensive part is the fresh, uncached work; design to reuse the rest. - **Right-size per step.** A long run doesn't need your most expensive model on every turn. Cheap, fast models for the routine steps and the strong one only where it earns its keep is still the move — just now measured over a whole run, not a single call. ## The bottom line When NVIDIA's flagship agent model competes on being cheap to run for hours instead of topping the intelligence chart, that's the market telling you what matters now. For anything that runs longer than a single answer, the leaderboard is the wrong thing to shop on. The benchmark measures how smart a model is on one question. Your bill measures how efficiently it finishes a hundred of them in a row — and only one of those numbers shows up on your invoice. So before you pick a model for an agent, stop asking "which is smartest" and start asking "which finishes my actual task for the least money." Run it, count the dollars to a correct result, and choose on that. The smartest model that burns twice the tokens to get there isn't the better agent. It's just the more expensive way to arrive at the same place.