Answer
How do you test something that gives a different answer every time?
Test the properties that must hold rather than the text, and measure across runs. Variation is a feature, not a defect.
Assert the properties that must hold in every acceptable answer, and measure how often they hold across repeated runs. Comparing against a recorded answer tests whether the output matched one sample, which is not the requirement.
The instinct is to make the system deterministic and then test it normally. That is available in a limited way — fixing the sampling settings reduces variation — and it does not remove it, and it is usually the wrong goal anyway, because the variation is a property of the component rather than a misconfiguration. Testing has to accommodate it rather than eliminate it.
The reframe that makes this tractable is to move from the answer to the requirements on the answer. What must be true of any acceptable output? It must select one of these categories. It must contain the customer's actual order number. It must not commit to a delivery date. It must be under a certain length. It must cite a document that exists. Each of these is checkable, each is stable across acceptable variation, and together they usually pin down correctness more tightly than people expect.
The second move is from verdict to rate. A property that holds on one run and fails on the next has told you something real, and a single execution cannot distinguish a solid property from a marginal one. Running each check several times converts a coin-flip into a measurement, and the measurement is what supports a decision about whether to deploy a change.
The third is that comparison beats absolute judgement. Whether a workflow is good enough is a hard question; whether this version holds the properties more often than the previous one is straightforward, provided the input set and the properties are fixed. Most practical decisions are comparative, and building the harness for comparison is cheaper than building one for absolute assessment.
The fourth is to shrink the non-deterministic surface. Every part of the workflow expressed as ordinary code is a part that can be tested conventionally, and the residual model step is the only thing needing this treatment. Systems designed with that in mind end up with a small, well-characterised model component and a large ordinary one, which is easier to test and easier to reason about.
One thing to be clear about: none of this gives the guarantee that a deterministic test gives. A property holding in a high proportion of runs is not a proof that it holds. That is the actual situation and it should be stated in those terms rather than obscured by a green tick, because the design decisions downstream — how much to check, what to allow unattended — depend on knowing which kind of assurance you have.
Stop asking whether it gave the right answer and start asking what has to be true of any right answer.
Siddharth Sharma, Context Theory
Related questions
Does setting the randomness to zero make it deterministic?
It reduces variation substantially and does not guarantee identical output, since other factors in the serving path can differ. It is worth doing for testing, and treating the result as fully deterministic is a mistake that surfaces later as an unexplained failure. Use it to reduce noise, not to justify a single-run test.
How do you handle a property that holds most of the time?
Decide whether the failures are acceptable and design for them, rather than raising the threshold and hoping. If the property is essential, it belongs in a deterministic check that rejects the output when it does not hold. Marginal properties are exactly where a validator should be added rather than where a test should be tuned.
METHOD
Every figure below carries its source and the date it was verified. Nothing on this page is asserted.
The numbers on this page.
| What | Value | Specific to |
|---|---|---|
| Close rate — response under 5 minutes vs over 24 hours | 32% vs 12% | Category-wide |
| Sub-15-minute compliance — automated routing vs manual only | 62.5% vs 39.1% | Category-wide |
Optifai speed-to-lead benchmark · n=939 companies · Q2 2025–Q1 2026 · verified
2026 speed-to-lead benchmark · verified
What is specific to this page.
| Kind | Claim | Check it against |
|---|---|---|
| Software | Fixing sampling settings reduces variation without eliminating it, because other factors in the serving path can differ, so a single-run test justified by determinism will eventually fail unexplained. | Running an identical request repeatedly with randomness minimised and comparing the outputs. |
| Workflow | Invariant properties — category membership, presence of a specific value, absence of a commitment, length bounds, citation resolvability — are stable across acceptable variation and together constrain correctness more tightly than an exact-output comparison would. | Listing the properties every acceptable answer must hold for a real task and testing whether an incorrect answer violates one. |
| Response | Comparative measurement against a fixed input set and fixed properties is tractable where absolute assessment is not, which is why most practical deployment decisions should be framed as version against version. | Running two versions over the same fixed set and comparing property hold rates. |
| Constraint | A property holding in a high proportion of runs is not proof that it holds, so the assurance available here differs in kind from a deterministic test and downstream decisions about unattended operation depend on that distinction. | Comparing the guarantee provided by a passing deterministic check against a measured pass rate. |
Each row would be wrong on another industry's page. Where a sourced figure exists it is in the table above instead; these are the constraints that shape the work and do not happen to be numbers.
Start with the measurement.
Reading about a benchmark is not the same as knowing your own number. The audit produces yours, measured rather than estimated.
$497 · delivered in 5 business days · credited against month one