What routine text checks fail to notice

One sentence contains several things to check

Consider “The 500 ml mug keeps drinks hot for six hours and is dishwasher-safe.” It contains at least three claims. Capacity, temperature retention and care each require a separate decision. A single “broadly correct” verdict would conceal the differences.

The fictional T500 dossier permits hand-washing the removable lid but contains no information about a dishwasher. The protocol specifies 450 ml of a drink initially at 95 °C, the mug closed throughout six hours, ambient air at +20 °C and a final temperature of at least 60 °C. The nominal 500 ml are therefore supported, the unconditional six-hour promise needs correction, and dishwasher permission is unsupported.

Store the object of review with its location in the text and the document version. Otherwise, “conditions missing from the second sentence” becomes unclear after paragraphs are rearranged. A semantic check also needs the selected product, dossier version, question and evaluation rules.

A verdict on a claim differs from a verdict on a page. A page with one correct number may still leave important compatibility questions unanswered. Checking the claims that are present will not discover all missing content. That requires a separate review of the product page's questions and priorities.

What each check actually establishes

A format check establishes whether the data can be accepted: required fields are present, types and lengths are correct, or the markup is permitted. A language check finds typos, inconsistencies and unclear relationships. An evidence check compares the claim with applicable information. Each can be useful, but the result of one cannot be attributed to another.

A literal requirement also needs an exact definition. “The text must contain the identifier T500” and “the text must contain T500 as a standalone word” specify different searches. Decide in advance whether case, word boundaries, line breaks and non-breaking spaces count. An evaluator that normalises the string on its own may violate the exact-match requirement.

A language check can notice negation, but negation supplies no fact about the product. “The lid must not be washed in a dishwasher” is grammatically correct. On the available dossier, the prohibition is just as unsupported as permission: missing information has been turned into a definite rule.

For the passage found, record separate results for its relation to the evidence, applicability to the product, preservation of conditions and sufficiency for the question. Insufficient evidence does not mean the statement must be false in reality. It may still be insufficient for publication; the next step is clarification or a narrower claim, not automatic replacement with its opposite.

Finally, check the delivered page. The attribute “mass — 280 g” and the explanation “empty mug” may end up in different sections or variants. A correct source paragraph does not guarantee a correct representation after export.

A finding must make the decision verifiable

Compare two findings. “Too bold a promise; score 7/10” does not explain which claim breached which rule. “The phrase ‘warm for six hours’ omits the test conditions; the protocol concerns 450 ml, an initial 95 °C, ambient air at +20 °C and a closed mug; the final temperature is at least 60 °C” identifies the issue.

In the second case, someone can open the evidence and check whether the qualifications are genuinely material and whether they already appear nearby. The evaluator can overlook context too. The record should therefore include the quotation, the dossier passage used, the discrepancy type and the proposed action. A link to an entire lengthy document without the relevant passage makes this harder.

Severity is determined separately. A wrong variant, unsupported compatibility or loss of a condition that changes suitability may block publication under the chosen rubric. Repeating a benefit or writing a cumbersome sentence usually calls for a different level of editing. A high style score must not override an established block.

An overall score is acceptable as a supplementary overview if its components are known. Publication decisions need the actual violations and acceptance rules. Otherwise, several successful paragraphs can average out a claim that leads a buyer to choose an unsuitable product.

The model evaluator needs its own tests

A model can be asked to find discrepancies between a text and its dossier. A detailed explanation of a decision does not establish that the decision is correct. The model may select the wrong passage, miss a restriction or justify a finding with information absent from the supplied materials.

The LLM-as-a-Judge study by Zheng and colleagues examined position-dependent preferences and a tendency to favour unnecessarily long answers under certain conditions. When comparing two revisions, their order can be swapped to see whether the decision persists. Stability after swapping is a useful check, but does not itself prove correctness.

Product-content evaluation needs its own set of cases. Include correct short answers, detailed answers containing an error, valid calculations, unknown parameters and similar numbers with different roles. If the set contains only obvious inventions, success says little about the ability to notice a lost condition.

A specific configuration is being tested: the model, instructions, available documents, retrieval method and output format. Changing any element may change the decisions. The model's general name does not describe the whole process.

Human annotation needs a rubric too. Independent editors first decide cases under the same rules, then discuss disagreements. If they disagree about what constitutes an error, an average score cannot hide the dispute. Sometimes the task must be clarified or a case marked as disputed rather than assigned an arbitrary reference label.

Two ways to make mistakes when detecting errors

Define the positive class as “the claim contains an error under the agreed rubric”. The evaluator either flags it or does not. Missed errors and false alarms can then be counted separately. The unit here is a claim, and the reference label is relative to the given dossier and rules, not the complete truth about the product.

Under this rubric, missing information in the dossier is not in itself an error in the description. An error occurs when the text turns that gap into categorical permission, a prohibition or another unsupported promise.

Define a fictional set of 20 claims: eight contain an error according to the reference labels, and twelve are correct. Suppose the evaluator detects six of the eight errors and wrongly flags three correct claims. Two errors receive no signal, while nine correct claims receive no erroneous finding. These numbers were chosen to explain the calculation; no test was conducted. There are no abstentions in these twenty fictional cases.

The proportion of valid alarms, precision, is 6 / (6 + 3) ≈ 66.7 %. Six of the nine findings are justified. Detection completeness, recall, is 6 / (6 + 2) = 75 %: six of the eight errors were found. These are standard classification metric definitions applied to detecting violations.

Overall accuracy is (6 + 9) / 20 = 75 %. It does not show which errors remain. If we specify that the set contains just two critical errors and both were missed, recall for that critical group is 0 out of 2. Overall values therefore need to be accompanied by counts for different error types and consequences.

A metric with a zero denominator is undefined. If the evaluator produces no findings at all, it cannot silently be awarded 100 % precision. A subset with no reference errors does not evaluate their detection. Publish the underlying counts alongside the proportions.

“The dossier contains insufficient information” may be a correct claim-review verdict. Here, abstention means the evaluator could not make the decision required by the rubric. Record that outcome separately. If the system sends difficult cases for manual review, show their number and share. Calculating only over the easier cases left behind would otherwise overstate the performance of the whole process.

What approval for publication means

T500's decisions can be recorded without an average score: nominal 500 ml accepted; nominal 450 ml to be corrected; test result to be checked with its conditions; dishwasher claim to be clarified. For important unanswered questions, separately record missing data or missing content.

Approval connects check results with the acceptance rubric. It concerns a specific text version, the evidence reviewed and remaining questions. “Checked by AI” conveys none of that. A human signature without recorded evidence leaves the same problem.

Every case can be examined in a small set. At scale, automated checks help select suspicious passages, while manual sampling helps find missed errors among accepted results. Reviewing rejected texts alone cannot reveal how many errors passed unnoticed.

Approval applies to the reviewed revision and the data supporting it. After shortening the text or updating a source, reconsider the affected part of the decision. Even a precise instruction to “restore the condition” does not guarantee a successful edit: the addition may create a contradiction, while a neighbouring cut may remove another restriction.