Answer
What should an AI regression test look like?
A fixed input, a property the output must have, and a threshold across many runs rather than a pass on one.
A fixed input, an assertion about a property the output must hold rather than its exact text, and a threshold measured across repeated runs. A single pass proves little when the same input can produce different output.
Conventional regression tests compare an output against a recorded one. That fails immediately here, because the same input legitimately produces different text, so an exact comparison flags every run as a regression and gets disabled within a week. What survives is an assertion about a property: the answer contains this value, selects this category, cites this document, omits this class of content, parses as this shape.
Choosing the property is the whole craft. It has to be strict enough that a genuine regression breaks it and loose enough that acceptable variation does not. A test asserting that a summary mentions the refund policy is durable; one asserting the summary's first sentence is durable only until someone changes the phrasing requirement. The good properties are usually about content presence, structural conformance, or an explicit exclusion.
The second structural difference is that a result is a rate rather than a verdict. Because output varies, a test that passes once may fail on the next identical run, so the meaningful measurement is over repetitions: this property holds in a stated proportion of runs. That is more work than a binary test and it is the only honest form, and it also makes the threshold explicit rather than implicit in whichever run happened to execute.
The third is what a failure means. In ordinary software a failing test indicates a defect introduced by a change. Here it may indicate that, or that the underlying platform moved, or that the property was too strict, or that the input distribution shifted. That ambiguity means the response to a failure starts with locating the cause rather than with fixing the code, and a test suite that does not distinguish these produces a habit of re-running until green.
Exclusion tests deserve their own mention because they are the most valuable and least written. Assertions that the output does not contain a category of content — a price it was not given, a commitment it cannot make, personal data, a claim about a named party — catch the failures that matter most and are usually expressible as a straightforward check. These are also the tests that stay relevant longest, because the prohibitions change more slowly than the requirements.
Finally, every production failure should become one of these. The case that broke, the property that should have held, added to the suite at the moment it is understood. This is the same discipline as any regression suite and it is what makes the collection represent the real failure surface rather than the imagined one.
Assert what must be true about the answer, not what the answer was, because the answer was one sample and the property is the requirement.
Siddharth Sharma, Context Theory
Related questions
How many repetitions are needed?
Enough that the measured rate is stable between runs of the suite, which for most properties is a handful rather than dozens. The purpose is to distinguish a genuine change from ordinary variation, and that is achievable well before statistical precision. Where a property is borderline, the borderline itself is the finding.
Should these run on every change?
The fast deterministic ones, yes. The full repeated-run suite is usually too slow and expensive for every change and belongs on a schedule and before any release. Splitting into a quick set and a full set is the same arrangement ordinary test suites arrive at, for the same reasons.
METHOD
Every figure below carries its source and the date it was verified. Nothing on this page is asserted.
The numbers on this page.
| What | Value | Specific to |
|---|---|---|
| Close rate — response under 5 minutes vs over 24 hours | 32% vs 12% | Category-wide |
| Sub-15-minute compliance — automated routing vs manual only | 62.5% vs 39.1% | Category-wide |
Optifai speed-to-lead benchmark · n=939 companies · Q2 2025–Q1 2026 · verified
2026 speed-to-lead benchmark · verified
What is specific to this page.
| Kind | Claim | Check it against |
|---|---|---|
| Software | Exact output comparison fails as a regression method because identical inputs legitimately produce different text, so such tests flag every run and are disabled quickly, leaving property assertions as the only durable form. | Running the same input twice and diffing the outputs. |
| Workflow | A regression result under variability is a rate across repetitions rather than a single verdict, which makes the threshold explicit instead of implicit in whichever run happened to execute. | Running the same property assertion repeatedly and recording the proportion of passes. |
| Response | A failure here is ambiguous between an introduced defect, a platform change, an overly strict property and an input shift, so investigation must precede correction or the response becomes re-running until green. | Recording the cause of each regression failure over a period and classifying them. |
| Constraint | Exclusion assertions — that the output does not contain an ungranted price, an unauthorised commitment, personal data or a claim about a named party — catch the highest-consequence failures and remain valid longer because prohibitions change more slowly than requirements. | Comparing how often exclusion assertions require revision against content assertions over the same period. |
Each row would be wrong on another industry's page. Where a sourced figure exists it is in the table above instead; these are the constraints that shape the work and do not happen to be numbers.
Start with the measurement.
Reading about a benchmark is not the same as knowing your own number. The audit produces yours, measured rather than estimated.
$497 · delivered in 5 business days · credited against month one