Should you use a model to grade a model?

The distinction that decides this is between discrimination and assessment. Discrimination is a comparison against something stated: does this output satisfy this requirement, does this passage support this claim, does this answer contain the value from that record. Assessment is an open judgement: is this good, is this thorough, is this well reasoned. The first is performed usefully and reproducibly. The second correlates with surface properties, and the surface is what the system producing the output is best at.

That gives a design rule. Never ask for a score without a rubric, and make the rubric a list of conditions rather than a set of qualities. Each item should be answerable with a yes, a no and a pointer to the evidence. A grader that returns per-item verdicts with references can be spot-checked; one that returns a number out of ten cannot be, and the number will drift with the phrasing of what it is grading.

Independence is the second requirement and it is the one usually missing. A grader that sees the reasoning which produced the output re-derives the same conclusions and agrees, which is why in-conversation self-grading returns approval at a high rate. Supplying only the artefact and the criteria is what makes the verdict informative, and withholding the reasoning is deliberate rather than an oversight.

Known biases are worth designing around because they are consistent. Longer answers score higher than shorter ones of equal content. Answers matching the grader's own habitual structure score higher. Position affects comparative judgements, so the same pair evaluated in both orders is worth doing where the comparison matters. None of these disqualifies the method; each of them means an unstructured score is measuring something other than what it claims.

The strongest use in practice is as a filter rather than a verdict. Run the grader across everything, use it to surface the cases that fail a stated criterion, and have a person look at those. This uses discrimination where it is strong and reserves judgement for a small set, and it produces a system whose weakest link is a human reading twenty items rather than a number nobody can check.

Finally, validate the grader itself before relying on it. Give it a set of outputs whose correctness you already know, including some that are wrong in the specific ways you care about, and see whether it catches them. A grader that has never been tested against known-bad cases is an unmeasured component being used to measure other things, which is not an improvement over having no measurement.