AI Tech Inspire

A Fair Benchmark for AI Agents: Separating Model Skill from Harness Design

When a coding agent nails a pull request or faceplants on a bug, what actually caused it? The core model, the orchestration, the tools, or a fragile acceptance gate? At AI Tech Inspire, we spotted a thoughtful benchmark proposal that aims to cleanly separate model capability from agent architecture — and it raises the kind of questions developers and researchers should be asking before trusting any leaderboard.

Key facts from the proposal

  • Most coding-agent benchmarks combine model and harness into a single score, obscuring root causes of failure.
  • Harness effects (e.g., output truncation, weak acceptance checks) can make a model look artificially worse or better.
  • Design: a 2×2 experiment crossing workflow (monolith vs. decomposed slices with explicit contracts) and model policy (frontier-only vs. cheapest-capable with escalation).
  • Four cells: frontier monolith, routed monolith, frontier decomposed, routed decomposed.
  • Frontier-decomposed is especially important for isolating architectural impact while holding model tier constant.
  • Controls to be frozen: tasks, source revisions, tools, total retry budget, acceptance criteria, validator versions, and the verifier.
  • All cells judged against the same final delivered outcome, not the persuasiveness of the agent’s report.
  • Primary measures: cost per independently accepted change, false acceptance, false rejection, first-pass accepted yield, verification time, reproducibility across three fresh runs.
  • Secondary measures: token use, latency, escalation count, and context volume.
  • Main confound: budget normalization. Decomposition may create more calls; per-slice full budgets would subsidize decomposition, but a shared budget may hide which slices needed capacity.
  • No results claimed; the author invites feedback on preregistration and whether decomposition should be treated as part of the evaluated system or isolated from model quality.

Why monolithic scores hide the truth

Most “agent benchmarks” compress a complex pipeline into a single number. That sounds convenient — until a run fails and it’s unclear whether the blame falls on the model’s reasoning, poorly assembled context, an overzealous truncation policy, a brittle tool, or a lenient acceptance gate. A simple harness tweak (e.g., a tighter output token cap) can tank performance; a naive success check (e.g., “did the test log contain ‘passed’?”) can inflate it.

In practice, this ambiguity makes it hard to answer the question that matters for engineering: “If this fails in production, what lever should we pull?” Better model? Better decomposition? Better retry logic? Better gating? A fair benchmark should point to the lever, not just the scoreboard.

A clean 2×2 that many teams can run

The proposal introduces a crisp, reproducible 2×2:

  • Workflow: monolithic task vs. decomposition into bounded slices with explicit contracts and acceptance criteria.
  • Model policy: frontier-only vs. cheapest-capable with escalation after a capability-graded failure.

That yields four cells: frontier monolith, routed monolith, frontier decomposed, and routed decomposed. The frontier-decomposed cell is particularly revealing because it changes system architecture while holding model tier constant. If the same top-tier model performs differently across monolith vs. decomposition, you’ve isolated architectural lift (or drag).

Strong controls, single outcome

To minimize noise, the benchmark freezes as much as possible: the task set and source revisions, toolchain versions, total retry budget, acceptance criteria, validators, and the final verifier. Every cell is judged against the same definition of a correct deliverable — no points for eloquent agent reports or screenshots that “look right.”

Key takeaway: benchmark the delivered change, not the narrative the agent tells about it.

Metrics that matter in production

The proposed primary measures map closely to what engineering teams care about:

  • Cost per independently accepted change: direct signal on unit economics.
  • False acceptance (FA): cases that pass the gate but are actually wrong — the most dangerous failure mode.
  • False rejection (FR): good changes that the gate incorrectly rejects — a drag on throughput and morale.
  • First-pass accepted yield (FPY): how often the very first attempt sticks.
  • Verification time: time the verifier needs to confirm/deny, capturing operational friction.
  • Reproducibility across three fresh runs: stability under repeated trials.

Secondary measures — token use, latency, escalation count, and context volume — give architectural fingerprints. For example, decomposition often increases context volume while lowering escalation count if simpler slices are routed to smaller models first.

The budget problem: fairness without subsidies

The thorniest confound is how to normalize the retry/context budget. Decomposition naturally creates more calls. If each slice inherits the monolith’s full budget, you’ve implicitly subsidized the decomposed condition. A shared system-level budget is cleaner but hides which slices were starved or actually needed more capacity.

Options worth preregistering:

  • Shared cap + per-slice ceilings: a global token/time cap with per-slice maximums (e.g., N attempts, M tokens) to prevent pathological over-allocation.
  • Yoked budgets: pair each monolith run with a decomposed run using the exact same total token/latency budget; attribute any differences to architecture and routing.
  • Two-budget report: report both per-slice and total budget consumption so readers can see if a condition “won” by burning more budget on long-tail slices.
  • Escalation tariffs: treat escalation to frontier as a priced action (e.g., a fixed cost unit), making routing policies comparable across cells.

What to preregister (before numbers appear)

To keep the benchmark falsifiable and robust to hindsight bias, it helps to lock down details up front. Consider preregistering:

  • Task list and sources: commit SHAs, dataset versions, environmental constraints.
  • Harness parameters: temperature, top_p, truncation rules, stop sequences, tool invocation policies, timeouts.
  • Acceptance gate vs. validator vs. verifier roles: define each, pin their versions, and publish the verification script.
  • Retry logic: backoff rules, maximum attempts, and when to escalate.
  • Escalation criteria: capability thresholds that trigger model upgrade under the routed policy.
  • Primary/secondary metrics with exact formulas (e.g., how FA/FR is computed, FPY definition, cost accounting).
  • Randomness controls: seeds for tools and sandboxes; fixed versions of dependencies to minimize drift.
  • Stopping conditions: when a run is declared irrecoverable.
  • Statistical plan: number of runs per cell, reproducibility threshold, and what constitutes a “meaningful” difference.

Is decomposition part of the system — or a factor to isolate?

There are two legitimate evaluation philosophies:

  • System view: treat decomposition as a first-class part of the solution. This matches real deployments, where architecture is a lever teams can optimize.
  • Isolation view: hold the model constant and vary only the decomposition to estimate its marginal value. This helps attribute improvements without conflating them with model-tier effects.

The 2×2 already helps, but you can go further by adding a yoked frontier-only comparison: same frontier model, same overall budget, with and without decomposition. That provides a strong test of architectural value. If practical, add an oracle ablation where the acceptance gate is replaced by a high-quality offline verifier, reducing feedback noise during retries.

Reproducibility: tame the non-determinism

Three fresh runs per cell is a good start. Also consider:

  • Pinning provider versions and documenting any model updates (especially for OpenAI GPT models).
  • Constraining randomness (temperature, sampling) and tool nondeterminism.
  • Containerizing tools and freezing dependencies via PyTorch or TensorFlow images where relevant.
  • Archiving prompts, tool I/O, and decisions as signed artifacts; use Hugging Face repos or similar for versioned logs.

From a practitioner’s perspective, having a run.jsonl that lets you Ctrl+C a flaky run and still reconstruct what happened is gold.

Why this matters for your team

If you’re shipping agents that write or modify code, this benchmark style surfaces the trade-offs you wrestle with daily:

  • Throughput vs. reliability: Does decomposition increase FPY enough to offset orchestration overhead?
  • Cost control: Does routing to smaller models first actually save money when you count extra calls and verification time?
  • Safety: Are false acceptances concentrated in certain slices? If yes, redesign the slice contracts or harden the acceptance gate there.
  • Operational clarity: Which lever (model tier, context assembly, retry policy) moves your cost per accepted change the most?

Run this 2×2 on a small but representative internal workload — even 20–30 tasks with pinned repos and deterministic tests — and you’ll learn more about your agent stack than a dozen generic leaderboards can tell you.

Closing thought

A fair agent benchmark should tell you not only who won but why, and at what price. By separating workflow architecture from model policy, freezing the environment, and committing to outcome-based verification, this proposal points in the right direction. The final piece is radical clarity: preregister the plan, publish the harness, and make it easy for others to rerun the exact cells. That’s how the field moves from leaderboard theater to engineering signal.

Recommended Resources

As an Amazon Associate, I earn from qualifying purchases.