Answer
How should a coding agent handle a failing test?
Work out what the test is asserting before changing anything, and never change the test to match the code without saying so.
Read what the test asserts and decide whether the code or the assertion is wrong, then fix that. Changing a test so it passes is occasionally correct and must always be reported, because silently doing it removes the guard.
A failing test is a disagreement between two statements about what the system should do. Resolving it means deciding which is wrong, and that decision requires knowing what the test is asserting and why it exists. The failure mode is skipping that step: the fastest route to a green suite is to adjust whatever is easiest to adjust, and the test is frequently easier than the code.
So the required sequence is read, diagnose, then act. Read the assertion and the failure output. Determine whether the behaviour changed or the expectation was wrong. Only then change something. This is ordinary practice and it is worth stating explicitly in a brief, because the alternative sequence produces a passing suite that has lost a guarantee, and the loss is invisible afterwards.
Changing a test is legitimate in specific cases: the requirement genuinely changed, the test asserted an implementation detail rather than a behaviour, or the test was wrong from the start. All three are real and all three should be stated in the report, with the reason. A change to a test buried inside a change to the code is the single most consequential thing to miss in a review, because the guard against a regression is removed at the same moment the regression becomes possible.
The pre-existing failure needs its own handling. If the suite was already failing before the change, the agent should say so and stop rather than fix it as part of unrelated work, for the same reason unrelated changes are unwelcome generally. It is also worth verifying rather than assuming: an agent's claim that a failure is pre-existing is checkable by running the suite on the unchanged code, and it is sometimes wrong.
Flaky tests are the case that corrupts everything else. A suite with intermittent failures teaches every participant, human and otherwise, that a red result means try again, and that lesson generalises to real failures. If a test is flaky, the honest options are to fix it or to remove it; leaving it in place while telling people to ignore it is how a suite stops functioning as a check at all.
Finally, the failure output should reach the reader. A report that says a test was failing and is now fixed omits the one piece of information that would let anyone assess the fix. The assertion, the expected value, the actual value: three lines that convert a claim into something checkable.
A test changed to match the code is not a passing test, it is a deleted requirement with a green tick on it.
Siddharth Sharma, Context Theory
Related questions
Should an agent be allowed to modify tests at all?
Yes, with the change surfaced. Forbidding it produces contorted code written to satisfy an assertion that was wrong, which is worse than an honest test change. The control is visibility rather than prohibition: test modifications called out in the report, and reviewed as the substantive changes they are.
What about a test that fails only in a particular environment?
That is information about the environment and should be reported as such rather than resolved by making the test tolerant. Adding a condition so a test passes locally is the same category of change as adjusting an assertion, and it removes coverage in exactly the environment where the difference was detected.
METHOD
Every figure below carries its source and the date it was verified. Nothing on this page is asserted.
The numbers on this page.
| What | Value | Specific to |
|---|---|---|
| Close rate — response under 5 minutes vs over 24 hours | 32% vs 12% | Category-wide |
| Sub-15-minute compliance — automated routing vs manual only | 62.5% vs 39.1% | Category-wide |
Optifai speed-to-lead benchmark · n=939 companies · Q2 2025–Q1 2026 · verified
2026 speed-to-lead benchmark · verified
What is specific to this page.
| Kind | Claim | Check it against |
|---|---|---|
| Workflow | A failing test is a disagreement between two statements of required behaviour, so resolution requires determining which is wrong, and the shortest route to a green suite is adjusting whichever is easier to adjust, which is frequently the test. | Reviewing changes that turned a failing suite green for whether the test or the code was altered. |
| Constraint | A test modification inside a code change removes the guard against a regression at the same moment the regression becomes possible, which makes it the most consequential thing for a reviewer to miss. | Filtering a change's diff for modifications to test files and checking whether they were mentioned in the description. |
| Software | A claim that a failure is pre-existing is checkable by running the suite against the unchanged code and is sometimes incorrect, so verification is available and cheap rather than requiring trust. | Running the suite on the base revision before accepting a pre-existing-failure claim. |
| Response | An intermittently failing test teaches every participant that a red result means retry, and that response generalises to genuine failures, so tolerating flakiness degrades the suite's function as a check rather than only wasting time. | Recording how many red results in the last month were resolved by re-running rather than by investigation. |
Each row would be wrong on another industry's page. Where a sourced figure exists it is in the table above instead; these are the constraints that shape the work and do not happen to be numbers.
Start with the measurement.
Reading about a benchmark is not the same as knowing your own number. The audit produces yours, measured rather than estimated.
$497 · delivered in 5 business days · credited against month one