Validating a model-based evaluator

Define exactly what the evaluator must decide

The main task is to determine a claim's relation to the supplied evidence: supported, contradicted or insufficient information. Product identity, conditions and source version must align. This is verification against the supplied materials, not an independent establishment of truth about a real product.

Document applicability, answer completeness and publication decisions are recorded separately. The instructions support hand-washing the T500 lid. That evidence does not apply to a claim about T350; without other information, the verdict for T350 is ‘insufficient information’. A supported number may also be only part of an answer about the full protocol.

The unit must be defined. If a sentence states volume, mass and care method, split it into verifiable claims while preserving the shared context. A condition in the preceding paragraph must not disappear during this split. Otherwise, the evaluator is being tested on damaged input.

In the study FActScore the text is split into atomic facts and the proportion supported by a selected corpus is calculated. The work tests the approach on biographies and Wikipedia; facts have equal weights in the original metric. This is a useful basis for distinguishing units, but published evaluator-performance figures cannot be transferred to a product catalogue.

Our rubric additionally records material conditions, product scope and violation severity. These are applied design choices. They should not be concealed under the name of someone else's metric.

The set must require distinctions between similar cases

Each case contains the claim, available context, permitted corpus, expected verdict and annotation evidence. A case's corpus is not the entire library: if only the line about tubes is supplied, colour information from another article is not available.

These are different inputs, not an attempt to assign contradictory properties to one product. Retain the actual passages used for each corpus. The full T500 protocol includes 450 ml, an initial 95 °C, air at +20 °C, a mug kept closed for six hours and a final temperature of at least 60 °C.

Cases with a valid condition in an adjacent sentence are also needed. For example, all conditions of a specific protocol are stated first, followed by ‘The final temperature was no less than 60 °C.’ Evaluation of the whole passage must account for this scope. The isolated sentence is not the same input.

Pairs that preserve or change meaning extend the set: a number written in words, a removed condition or a changed care object. Both long and short versions must be combined with correctness in different ways so that length does not replace the criterion.

A reference starts with a rubric and independent decisions

Annotators receive the same context and permitted sources. Each first decides independently; disagreements are then resolved against the evidence. If the claim itself is ambiguous, clarify its wording or retain a disputed status instead of hiding disagreement in an average.

Retain the quotation, entity, material conditions and reason with the label. For the 120 ml calculation, record the operation and inputs. For unknown dishwasher suitability, record the boundaries of the corpus containing neither permission nor prohibition. This allows the reference decision itself to be checked, rather than merely comparing label strings.

Agreement on the distinction between ‘contradicted’ and ‘unsupported’ is particularly important. A nominal 450 ml is incompatible with the specified 500 ml for the same parameter. Dishwasher suitability is simply not established. Publishing a categorical claim may be an error in either case, but the evidence relation differs.

In FEVER evidence retrieval is separated from claim verification; support and refutation are linked to corpus passages. This provides a useful record structure for our test. The reference still requires checking that a document belongs to the correct product.

Separate evidence verification from evidence retrieval

In the first mode, the evaluator receives the claim and preselected evidence with the necessary context. This tests its decision on available information. For ‘insufficient information’, define the limits of the available material too: absence from one paragraph cannot be declared absence from every document in the world.

In the second mode, provide a claim and a fixed corpus in which evidence must be found. Retain retrieved passages and the final verdict. If the required document is not found, the error can be located before pairwise verification.

The two modes need not always yield the same verdict. If the first is deliberately limited to the line about tubes and the second includes an additional palette document, the number of colours may gain new evidence. Comparison is meaningful only when input differences and each input's reference are considered.

To investigate an error, repeat a failed second-mode case with evidence known to be applicable. If the decision is corrected, this is a reason to investigate retrieval and context transfer. The observation alone does not establish that every other case has the same cause.

Freeze the configuration before the final test

Instruction tuning and final evaluation use different cases. Families of close paraphrases should stay in the same split. If the instruction was tuned on the nominal-450-ml error, an almost identical version of that task does not provide an independent test.

Record the model, its available version identifier, task instructions, run parameters, corpus, retrieval method, amount of supplied context and output format. Retain the actual assessment too. Changes in decisions can then be compared with configuration changes.

Once revisions are made using held-out results, those cases become known material. Further quality confirmation requires an independent set, or an honest description that regression on familiar examples was tested.

When comparing two revisions, their order can be varied and author identities hidden. Work by Zheng and colleagues shows position dependence and a preference for excessive length under the studied conditions. Such a control helps detect instability; a stable choice does not itself mean a correct decision.

If behaviour on identical inputs varies between runs, the plan must define how to account for it: how many repetitions are made and which result is published. It is not acceptable to select only successful evaluator answers or silently change the rule after seeing the results.

Report misses, false signals and the limits of coverage

First compare verdicts with the reference using the chosen unit. For error detection, the positive class is the presence of a violation; the confusion matrix and denominators are discussed in article 09. For the three evidence relations, separately report confusions such as contradiction versus insufficient information, or support versus insufficient information. Abstentions are recorded outside this three-class table.

A correct ‘insufficient information’ is a complete verdict. Here, abstention means failing to produce a decision required by the protocol. Report its count, the proportion of the set assessed automatically and the result of subsequent manual review separately. Skipping every difficult case must not make the whole system appear better.

The proportion of supported facts also differs from answer completeness. FActScore measures precision relative to a corpus, not recall; its authors separately discuss small fact counts and frequent abstentions. Its denominator is the atomic facts in the assessed text. It is not the proportion of justified error signals from an evaluator. Verdict correctness is established by comparison with independent annotation. A listing can retain only the correct 500 ml and achieve a good support fraction while leaving care and testing questions unanswered. That requires a separate coverage plan.

For a catalogue, define the treatment of repeated claims and critical errors in advance. Repeating a correct volume several times must not conceal one unsupported prohibition. This policy is our addition to the rubric; the original work's equal weighting of atomic facts is a different design choice.

Accompany overall results with subsets: product, parameter type, negation, condition, calculation and retrieval of another model's data. Subset sizes are needed too. One critical miss in a small group deserves investigation, although it does not estimate the error rate precisely across an entire future catalogue.

Retain the reason for the next action

For the ‘nominal 450 ml’ error, the log should show the source claim, retrieved protocol passage, expected verdict and actual verdict. Then investigate where the quantity's role was lost: extraction, retrieval or comparison. A wrong decision does not automatically reveal its internal cause.

If the evaluator rejects 6 × 20 ml, check whether the rubric permits transparent inferences and whether both input values were supplied. If it accepts T350 using a T500 document, examine applicability. If it ignores a condition in the previous paragraph, examine the context window it received.

The analysis leads to a specific process revision or data clarification. After a change, repeat the affected checks and retain the earlier configuration for comparison. Regressions can occur in the evaluator itself too.

The protocol produces verified inputs, expected decisions and a procedure for analysing actual outputs when testing takes place. Until outputs exist, no accuracy conclusion is drawn. For an individual listing, these materials can be connected to a source dossier, question map and accepted revision, creating a traceable history of every decision.