When a coding agent’s change fails validation, the response becomes part of the context for its next attempt. How much the agent can learn from that response depends on what the validator reports.
Feedback changes the repair problem
A bare failure message leaves the agent to diagnose the problem from scratch:
architecture check failed
Before it can repair the change, the agent has to inspect the repository, infer what failed and locate the cause.
Compare that with a validator response that includes the relevant context:
New dependency cycle: payments -> orders -> inventory -> payments Introduced by: payments/checkout.go:84 Rule: orders must not depend on payments
The agent still has to write the fix. It no longer has to rediscover the failure first.
What the experiment found
In Structured Feedback Improves Repair in an LLM Agent Loop, Jaideep Ray and Ankit Goyal tested how different validator responses affected repair performance. Their harness generated a candidate, passed it to an external validator, returned the result to the model and allowed another attempt.
They compared raw diagnostics with feedback containing three pieces of information: where the failure occurred, what value was observed and which alternatives were valid. Across 50 paired TextWorld tasks with a four-call limit, Qwen2.5-Coder-14B went from 14 successful tasks with raw diagnostics to 36 with the full feedback. Llama-3.1-8B went from 8 to 29.
The ablation provides more detail than the aggregate results. Location and observed value alone produced results close to the raw diagnostic baseline. Most of the improvement appeared when the validator also supplied the admissible alternatives: the model benefited from knowing what would be accepted instead.
The information matters more than the format
The researchers also tested whether the format itself mattered. The same repair information presented as prose performed almost the same as the keyed JSON representation. There was no evidence that JSON syntax itself improved repair.
For tool authors, the relevant distinction is therefore the information in the response rather than whether it is represented as JSON or prose.
Consider a dependency rule:
orders may depend on inventory orders may not depend on payments
A validator could report:
forbidden dependency
It could also report:
orders/service.go:61 introduced orders -> payments Allowed outgoing dependencies from orders: - inventory - shipping - shared
The second response gives the next model call a much smaller search space. The agent knows where the problem came from and which outcomes satisfy the rule.
From diagnostics to agent input
This resembles what good developer tooling has done for years. A compiler that points to the offending expression is more useful than one that only says the build failed. A test showing the expected and actual values is easier to debug than a bare non-zero exit code.
Coding agents make the quality of these interfaces more visible because validator output becomes input to another program.
The tool is no longer reporting a failure only for a developer reading a terminal; its output may immediately become part of the next model invocation. A useful response identifies the location and observed state, includes constraints on valid repairs when they are known, and omits unrelated output.
The limit of the evidence
The experiment used 50 TextWorld games rather than production repositories. The models operated under a fixed four-call budget, and the tasks were designed around externally verifiable states. The reported improvement should not be treated as an expected success-rate increase for software engineering.
The narrower result is still relevant: when a model is asked to repair a rejected attempt, the contents of the validator response affect its ability to do so.
A second recent study, An Empirical Study of Harness Design for Coding Agents, approaches the problem from a wider angle. Across 176 matched settings on SWE-Bench Verified and Terminal-Bench 2.1, the authors found that the effects of context management, planning and available actions depended on the model, context budget and task. These choices changed coding-agent performance, cost and trajectory length, but no single configuration was best in every setting.
Both papers examine the part of harness design between one attempt and the next: what context is retained, which actions are available and what feedback follows a rejected result.
Where this applies
A test runner can return the failing assertion and relevant diff instead of the full test log. A type checker can identify the symbol and incompatible types. A security scanner can point to the data flow or rule that triggered the finding. An architectural check can return the new edge, cycle or boundary violation that appeared after the edit.
The check should still be deterministic where possible. The agent can reason about the repair; it should not have to guess whether the violation exists.
enola applies this approach to architectural changes. It builds a deterministic structural model of the repository and can return the edge, cycle or boundary violation behind a failed check. The harness can feed that result back into the coding loop while the task is still running.
The validator is a product surface
For teams building coding-agent workflows, the validator interface is a product surface of its own. A binary pass or fail may be enough for CI, but an agent expected to repair its work needs enough information to make a better next attempt.
More from the harness series
- What is an agent harness? defines the runtime around the model: tools, context, permissions, execution and feedback.
- The harness decides what the model sees explains how context selection shapes every model invocation.
- Instructions are not controls separates guidance before a change from deterministic checks and enforcement afterward.
- enola is an architecture fitness sensor places architectural checks in the guides-and-sensors model.